Back Original

Rebuilding my local voice assistant pipeline

I wrote an article a couple of months ago describing how I rebuilt my old voice assistant pipeline in Platypush.

And I wrote earlier articles describing my journey through self-hosted voice assistants - in 2024 (with Picovoice), in 2020 (diving into the legacy assistant.google integration, the now defunct assistant.alexa integration, the now defunct Snowboy integration, the now defunct DeepSpeech integration and Picovoice) and the original article in 2019 describing the Google Assistant integration.

For a couple of months I used the voice pipeline described in my latest article as my main driver, but after getting used to the robustness of commercial voice assistants I was unhappy with the high rates of false positives/negatives of my self-hosted pipeline, as well as its lack of robustness in detecting start/end of speech, especially in noisy environments and using cheap microphones.

After a bit of experimentation, I decided to swap and fine tune a few parts, and after a couple of days of trying the results are satisfying enough for me to write a short article about it.

The resulting pipeline is:

HotwordDetectedEvent

SpeechRecognizedEvent

follow-up

Hotword detection

Speech-to-text

Generate response

Text-to-speech

HotwordDetectedEvent

SpeechRecognizedEvent

follow-up

Hotword detection

Speech-to-text

Generate response

Text-to-speech

Wake word detection: a new assistant.microwakeword plugin

Until recently I used assistant.openwakeword, which wraps openWakeWord.

It works, but the project has been quiet for a while and all of its pre-trained models are v0.1 files from 2023, mostly trained on synthetic speech.

In practice I was getting both false positives (the hotword triggering on background noise) and false negatives (the hotword not detected unless I was close to the microphone).

I added a new plugin, assistant.microwakeword, on top of microWakeWord through the pymicro-wakeword Python bindings.

microWakeWord is the engine used by ESPHome and Home Assistant Voice, its models are actively maintained, and the built-in models (okay_nabu, hey_jarvis, hey_mycroft, alexa) ship with the package — no download required.

The current official models are provided under the ESPHome micro-wake-word-models repository.

Custom models are supported too: drop a microWakeWord JSON manifest (with its .tflite model) in the plugin's models directory and reference it by name.

The plugin exposes the same API as assistant.openwakeword (same models, detection_threshold, frame_duration, patience_frames, input_device, input_volume parameters, same list_models action, same HotwordDetectedEvent), so switching is a matter of just changing the configuration to point to the new plugin:

assistant.microwakeword:
  models:
    - alexa

A few design notes:

Speech-to-text: assistant.openai, with a much better ear

assistant.openai records audio when a conversation starts and transcribes it through Whisper.

The tricky part has always been end-of-speech detection: knowing when the user has finished talking. The previous implementation used a per-frame boolean from WebRTC VAD, and any single frame classified as speech would reset the silence timer. With even a bit of background noise, phrases would only end when the maximum conversation duration was hit.

The audio preprocessing layer (AudioPreprocessor, shared by the STT integrations) has been reworked:

assistant.openai:
  # auto = silero -> webrtc -> energy, depending on what's installed
  vad_backend: auto

On a synthetic benchmark (Piper-generated phrases mixed with babble and burst noise), the old settings kept conversations open for 6–10 seconds after the speech had ended; the new pipeline ends them 1.0–1.6 seconds after the last word, which is the configured conversation_end_timeout plus the smoothing lag.

Since the changes on Platypush have been performed directly in the AudioPreprocessor also used by other native speech-to-text engines, other speech-to-text plugins also benefit from them - e.g. if you use assistant.vosk for full on-device detection.

Responses and context: the openai plugin

The transcribed phrase is processed by the openai plugin, which handles both the API key and the conversation context. By default the assistant renders the response through the configured tts_plugin; if the response ends with a question mark, the conversation continues with a follow-up.

Custom hooks can intercept phrases before the default response kicks in — for example, to handle home automation commands locally without an LLM round-trip:

from platypush import run, when
from platypush.message.event.assistant import SpeechRecognizedEvent

@when(SpeechRecognizedEvent, phrase='play (the)? music')
def play_music():
    run('music.mpd.play')

And the hotword event is wired to the conversation start with a similar hook:

from platypush import run, when
from platypush.message.event.assistant import HotwordDetectedEvent

@when(HotwordDetectedEvent)
def on_hotword(**_):
    run('assistant.openai.start_conversation')

Text-to-speech: tts.openai

Responses are rendered by tts.openai, with volume and voice configurable. In my tests however tts.piper also yielded good results, and it runs entirely locally. In my tests on a RaspberryPi 5 however it added ~3-4s of latency to process the response to audio compared to tts.openai.

Since the TTS output goes to the speakers and the wake word detector listens on the microphone, the assistant plugins pause hotword detection while a conversation is active — otherwise the assistant would hear itself.

The full configuration

openai:
  api_key: YOUR_API_KEY
  # Or, for a custom setup using a model provider compatible with the OpenAI API,
  # e.g. a local Ollama instance:
  # base_url: http://localhost:8000/v1
  # model: gpt-5
  # Optional context that can be passed to initialize the assistant
  # context:
  #   role: system
  #   content: You are a useful assistant.

assistant.microwakeword:
  models:
    - alexa

assistant.openai:
  tts_plugin: tts.openai
  conversation_start_sound: ~/sounds/wake_word_detected.wav
  input_volume: 100

# assistant.vosk:
#   model: /path/to/vosk/model
#   conversation_start_sound: ~/sounds/wake_word_detected.wav
#   input_volume: 100

tts.openai:
  output_volume: 80.0

# Or, for a fully local setup that uses tts.piper:
# tts.piper:
#   output_volume: 80.0

What changed on the code side

Caveats

The microphone on my RPi doesn't support 16 kHz natively, so audio is captured at 44.1 kHz and software-resampled — it works, but if detection feels sluggish software resampling is usually the first suspect.

The VAD and threshold defaults were tuned on synthetic audio plus a couple of my audio recordings.

Different rooms, microphones and acoustics may yield different results. You may want to adjust detection_threshold on the wake word plugin and noise_floor_factor on the STT plugins if you don't feel satisfied with your mileage.