Wyoming Voice Match

A Wyoming protocol ASR proxy that verifies speaker identity and extracts your voice from background noise before forwarding audio to a downstream speech-to-text service. Designed for Home Assistant voice pipelines to prevent false activations from TVs, radios, and other people - and to deliver clean transcripts even in noisy environments.

The Problem

Home Assistant voice satellites listen for a wake word, then stream audio to a speech-to-text service. But the satellite microphone picks up everything - your voice, the TV in the background, other people talking. This causes two issues:

False activations: The TV says something that triggers a command
Noisy transcripts: Your voice command gets mixed with TV dialogue, producing garbage like "What time is it? People look at you like some kind of service freak"

Wyoming Voice Match solves both: it verifies that the audio contains your voice before allowing it through, and it uses voiceprint-based speaker extraction to isolate your command - removing TV dialogue and other speakers - before sending clean audio to the speech-to-text service.

Before and After

In this example, a TV is blasting in the background while the user says "Tell me the weather and then turn on the living room lights". The raw audio captured by the satellite contains both the TV dialogue and the user's command:

Before processing (raw satellite audio — 13.9s):

Original.mp4

After processing (extracted speaker audio — 5.8s):

Processed.mp4

Here's what the pipeline did behind the scenes:

--- Speaker Verification ---
Speech detected: 6.5-8.7s (2.1s segment, peak_energy=7020)
Pass 1 (speech): verifying 2.1s segment 1/3 (6.5-8.7s)
Pass 1 (speech) matched segment 1 in 237ms (0.4909)

  Speaker              Similarity   Result
  ──────────────────── ──────────   ──────────
  jx                       0.4909   MATCH

--- Speaker Extraction ---
Found 4 speech regions: 0.7-5.7s, 6.5-9.2s, 9.6-11.6s, 12.0-12.9s

  Region 0.7-5.7s  → trimmed to 2.7-5.7s, KEEP (0.40)
  Region 6.5-9.2s  → KEEP (0.48)
  Region 9.6-11.6s → discarded (0.09)
  Region 12.0-12.9s → discarded (0.13)

  Input:  13.9s → Output: 5.8s (59% discarded)
  Total pipeline: 313ms

The extraction identified 4 speech regions in the audio. Regions 1 and 2 matched the enrolled voiceprint (the user's voice) while regions 3 and 4 were TV dialogue and were discarded. Region 1 was further trimmed — its first 2 seconds were TV audio that overlapped with the start of the user's speech, and the sub-region scan narrowed it down to just the user's portion.

How It Works

Wyoming Voice Match sits between Home Assistant and your ASR (speech-to-text) service. When a wake word is detected, Home Assistant opens a connection and starts streaming audio. Here's what happens:

flowchart LR
    A["🎙️ Mic (Device)"] --> B["Wake Word Detection"]
    B --> C["Wyoming Voice Match"]
    C --> D["ASR Service (Transcribe)"]

    subgraph C["Wyoming Voice Match"]
        direction TB
        C1["1. Buffer audio"]
        C2["2. Verify speaker"]
        C3["3. Wait for stream"]
        C4["4. Extract speaker"]
        C5["5. Forward to ASR"]
        C1 --> C2 --> C3 --> C4 --> C5
    end

Step by step

Buffer audio - audio streams in from Home Assistant after the wake word is detected
Verify speaker - after 5 seconds, an energy analysis isolates the loudest segment (your voice near the mic) and compares it against your enrolled voiceprint using an ECAPA-TDNN neural network. If it doesn't match any enrolled speaker, the pipeline is silently stopped with an empty transcript
Wait for stream - once verified, the proxy continues buffering audio until the satellite's VAD signals the end of the stream, capturing your complete command regardless of length
Extract speaker - the full audio buffer is split into speech regions using energy analysis, then each region is verified against your voiceprint. Only regions matching your voice are kept; TV dialogue and other speakers are discarded
Forward to ASR - the cleaned audio (only your voice) is sent to the speech-to-text service for transcription

The result:

In a quiet room, Voice Match adds only milliseconds of overhead to your existing pipeline - verification is nearly instant
With a TV blaring, speaker extraction removes TV dialogue from the audio, delivering clean transcripts like "What time is it?" instead of "What time is it? So I've been all for it and just..."
Commands of any length are fully captured - no fixed time limits
TV audio and other speakers are rejected based on voiceprint mismatch, not energy levels

Requirements

A running Wyoming-compatible ASR service such as wyoming-faster-whisper or wyoming-onnx-asr. Wyoming Voice Match sits in front of this service as a proxy and forwards verified, cleaned audio to it.

The Quick Start guide below uses Docker and Docker Compose for deployment. An NVIDIA GPU is recommended for fast inference (~5-25ms verification) but not required — CPU inference works at ~200-500ms. The scripts can also be run directly with Python 3.10+ and the dependencies listed in requirements.txt.

Quick Start

1. Create a Project Directory

mkdir wyoming-voice-match && cd wyoming-voice-match
mkdir -p data/enrollment

2. Create `docker-compose.yml`

GPU (recommended):

services:
  wyoming-voice-match:
    image: ghcr.io/jxlarrea/wyoming-voice-match:latest
    container_name: wyoming-voice-match
    restart: unless-stopped
    ports:
      - "10350:10350"
    volumes:
      - ./data:/data
    environment:
      - UPSTREAM_URI=tcp://wyoming-faster-whisper:10300
      - LISTEN_URI=tcp://0.0.0.0:10350
      - VERIFY_THRESHOLD=0.30
      - EXTRACTION_THRESHOLD=0.25
      - HF_HOME=/data/hf_cache
      - LOG_LEVEL=DEBUG
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

CPU-only:

services:
  wyoming-voice-match:
    image: ghcr.io/jxlarrea/wyoming-voice-match:cpu
    container_name: wyoming-voice-match
    restart: unless-stopped
    ports:
      - "10350:10350"
    volumes:
      - ./data:/data
    environment:
      - UPSTREAM_URI=tcp://wyoming-faster-whisper:10300
      - LISTEN_URI=tcp://0.0.0.0:10350
      - VERIFY_THRESHOLD=0.30
      - EXTRACTION_THRESHOLD=0.25
      - HF_HOME=/data/hf_cache
      - LOG_LEVEL=DEBUG

Update UPSTREAM_URI to point to your ASR service.

3. Enroll Your Voice

Record at least 30 WAV files per speaker, 5 seconds each, speaking naturally at varied volumes and distances. Place them in data/enrollment/<speaker>/:

mkdir -p data/enrollment/john

Linux:

for i in $(seq 1 30); do
  echo "Sample $i - speak naturally for 5 seconds..."
  arecord -r 16000 -c 1 -f S16_LE -d 5 "data/enrollment/john/john_$(date +%Y%m%d_%H%M%S).wav"
  sleep 1
done

macOS:

for i in $(seq 1 30); do
  echo "Sample $i - speak naturally for 5 seconds..."
  sox -d -r 16000 -c 1 -b 16 "data/enrollment/john/john_$(date +%Y%m%d_%H%M%S).wav" trim 0 5
  sleep 1
done

Requires SoX: brew install sox

Windows (PowerShell):

Download the recording script and run it - it will list your microphones, let you pick one, and guide you through recording:

.\record_samples.ps1 -Speaker john

Requires ffmpeg: winget install ffmpeg

Alternatively, use any voice recorder app on your phone or computer and save the files as WAV. The enrollment script handles resampling automatically, so any sample rate or channel count will work.

Example phrases to record (one per sample, speak naturally):

"Hey, turn on the living room lights and set them to fifty percent"
"What's the weather going to be like tomorrow morning"
"Set a timer for ten minutes and remind me to check the oven"
"Lock the front door and turn off all the lights downstairs"
"What's the temperature inside the house right now"

Tip: The best results come from enrolling with 30 samples at varied volumes and distances. The more variety in your samples, the more robust your voiceprint will be.

Recording from a Satellite

If you're using a Wyoming satellite (like the Home Assistant Voice PE), recording samples from your PC or phone may produce a voiceprint that doesn't match well with the satellite's microphone. For best results, record enrollment samples directly through the satellite using the enroll_record script.

Stop the main service first, then run the recording script:

docker compose stop wyoming-voice-match

docker compose run --rm --service-ports --entrypoint python wyoming-voice-match \
  -m scripts.enroll_record --speaker john --samples 10

The script listens on the same Wyoming port as the main service. Say your wake word on the satellite, speak naturally for a few seconds, and wait for the satellite's done sound. Each done sound means a sample was saved. Progress is logged to the terminal. Repeat until all samples are collected. The script automatically runs enrollment and generates the voiceprint when done.

Restart the main service afterward:

docker compose start wyoming-voice-match

Tip: You can combine satellite and PC recordings. Record some samples from the satellite and place additional WAV files in data/enrollment/<speaker>/, then re-run enrollment. This produces a voiceprint that works well across different microphones.

Generate the voiceprint:

docker compose run --rm --entrypoint python wyoming-voice-match -m scripts.enroll --speaker john

Repeat for additional speakers:

docker compose run --rm --entrypoint python wyoming-voice-match -m scripts.enroll --speaker jane

Manage enrolled speakers:

# List all enrolled speakers
docker compose run --rm --entrypoint python wyoming-voice-match -m scripts.enroll --list

# Delete a speaker
docker compose run --rm --entrypoint python wyoming-voice-match -m scripts.enroll --delete john

4. Start the Service

docker compose up -d

5. Configure Home Assistant

In Home Assistant, update your voice pipeline to use this service as the STT provider:

Go to Settings → Devices & Services → Wyoming Protocol
Add a new Wyoming integration pointing to your server's IP on port 10350
In Settings → Voice Assistants, edit your pipeline and set the Speech-to-Text to the new Wyoming Voice Match service

Configuration

Environment Variables

All configuration is done in the environment section of docker-compose.yml:

Variable	Default	Description
`UPSTREAM_URI`	`tcp://localhost:10300`	Wyoming URI of your real ASR service
`LISTEN_URI`	`tcp://0.0.0.0:10350`	URI this service listens on
`VERIFY_THRESHOLD`	`0.30`	Cosine similarity threshold for speaker verification (0.0-1.0)
`EXTRACTION_THRESHOLD`	`0.25`	Cosine similarity threshold for speaker extraction — regions below this are discarded
`REQUIRE_SPEAKER_MATCH`	`true`	When `false`, unmatched audio is forwarded to ASR instead of being rejected — enrolled speakers still get verification and extraction
`TAG_SPEAKER`	`false`	Prepend `[speaker_name]` to transcripts (useful for LLM-based conversation agents)
`SAVE_REJECTED`	`false`	Save rejected audio clips and metadata to `/data/rejections/` for review and re-enrollment
`LOG_LEVEL`	`DEBUG`	Logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`)
`DEVICE`	`cuda`	Inference device (`cuda` or `cpu`). Auto-detects: falls back to CPU if CUDA is unavailable
`HF_HOME`	`/data/hf_cache`	HuggingFace cache directory for model downloads (persisted via volume)
`MAX_VERIFY_SECONDS`	`5.0`	Seconds of audio to buffer before starting speaker verification
`VERIFY_WINDOW_SECONDS`	`3.0`	Sliding window size (in seconds) for the fallback verification pass
`VERIFY_STEP_SECONDS`	`1.5`	Step size (in seconds) between sliding windows

Tuning the Threshold

The VERIFY_THRESHOLD and EXTRACTION_THRESHOLD environment variables control how strict speaker matching is. Adjust them in docker-compose.yml and restart.

VERIFY_THRESHOLD determines whether the speaker is accepted at all. EXTRACTION_THRESHOLD determines which audio regions are kept when removing background audio. The extraction threshold should be lower than the verification threshold.

VERIFY_THRESHOLD	Behavior
`0.25`	Lenient - may accept some TV audio, but rarely rejects the enrolled speaker
`0.30`	Default - good balance for satellite mics with TV or background audio
`0.40`	Moderate - works well with high-quality mics (PC, laptop, phone)
`0.55`	Strict - fewer false accepts, but may reject quiet or distant speech

Start with debug logging enabled and observe the similarity scores:

docker compose logs -f wyoming-voice-match

You'll see output like:

INFO [handler] [971f8eb8] ── New audio session started ──
INFO [handler] [971f8eb8] Speaker verified: jx (similarity=0.3787, threshold=0.30), forwarding to ASR immediately
INFO [handler] [971f8eb8] ── Pipeline Summary ──
  Step           Duration
  ─────────────  ────────
  Verify              5ms
  Extract            39ms  (10.2s → 3.1s, 70% discarded)
  ASR               180ms
  Total           10649ms

  Transcript: "Tell me the weather"

INFO [handler] [3a2c1b9f] ── New audio session started ──
INFO [handler] [3a2c1b9f] ── Pipeline Summary ──
  Step           Duration
  ─────────────  ────────
  Verify            252ms
  Total            5032ms

  Result: REJECTED (best=0.1847, threshold=0.30)
  Scores: jx=0.1847

Your voice will typically score 0.35-0.70 depending on mic quality and command length
TV/other speakers will typically score 0.05-0.25
Set the verification threshold in the gap between these ranges
Set the extraction threshold slightly below the verification threshold
If you're getting rejected when speaking quietly, lower the threshold or re-enroll with more samples recorded at different volumes and distances

Being rejected too often? The most effective fix is to add more enrollment samples. Record additional samples in the conditions where you're being rejected (e.g., speaking softly, further from the mic, different times of day) and re-run enrollment. More samples produce a more robust voiceprint that handles natural voice variation better. If using a satellite, record samples directly through it with enroll_record — see Recording from a Satellite.

Noisy Environment Tuning

The default settings are tuned for noisy environments (TV, radio, etc.). The speaker extraction automatically removes background audio by comparing each speech region against your voiceprint — only regions matching your voice are forwarded to ASR.

If you need to adjust further:

    environment:
      - VERIFY_THRESHOLD=0.30     # Accepts the speaker if any pass scores above this
      - EXTRACTION_THRESHOLD=0.25 # Keeps audio regions scoring above this

Note: The satellite may continue showing a "listening" animation after the command has been processed. This is cosmetic - the proxy waits for the full stream to capture your complete command, but Home Assistant will have the transcript as soon as extraction and ASR finish.

Speaker Tagging

When TAG_SPEAKER=true is set, the verified speaker's name is prepended to the transcript:

[jx] Tell me the weather and the current price of Bitcoin

This is useful when using an LLM-based conversation agent (e.g., OpenAI, Claude, or a custom integration) - the LLM can use the speaker's identity to personalize responses or trigger per-user automations. This setting has no effect on rejected speakers (empty transcripts).

Note: If you're using Home Assistant's built-in intent-based assistant, leave this disabled. The [speaker_name] prefix will interfere with intent matching. This feature is designed for LLM-based conversation agents that can parse the tag naturally.

Rejected Audio Logging

When SAVE_REJECTED=true is set, every rejected audio clip is saved to /data/rejections/ with a companion JSON metadata file. This is useful for harvesting voice samples from family members or guests for future enrollment without requiring a dedicated recording session.

    environment:
      - SAVE_REJECTED=true

Each rejection produces two files:

WAV file: The full audio clip (e.g., rejected_20260315_143045_a1b2c3d4.wav)
JSON file: Rejection metadata (e.g., rejected_20260315_143045_a1b2c3d4.json)

Example JSON — rejected with speaker scores (near-miss or known voice at a distance):

{
  "timestamp": "2026-03-15T14:30:45.123456+00:00",
  "session_id": "a1b2c3d4",
  "audio_file": "rejected_20260315_143045_a1b2c3d4.wav",
  "duration_seconds": 4.2,
  "sample_rate": 16000,
  "best_score": 0.1847,
  "threshold": 0.30,
  "margin": 0.1153,
  "all_scores": {
    "jx": 0.1847,
    "jane": 0.0921
  },
  "speech_start_sec": 1.2,
  "speech_end_sec": 3.4
}

Example JSON — rejected with no speaker detected (e.g., TV, background noise):

{
  "timestamp": "2026-03-15T14:30:45.123456+00:00",
  "session_id": "a1b2c3d4",
  "audio_file": "rejected_20260315_143045_a1b2c3d4.wav",
  "duration_seconds": 4.2,
  "sample_rate": 16000,
  "best_score": 0.0,
  "threshold": 0.30,
  "margin": 0.30,
  "all_scores": {},
  "speech_start_sec": null,
  "speech_end_sec": null
}

The margin field (threshold - best_score) is useful for triaging: sort by lowest margin to find near-misses that are good candidates for enrollment. To enroll a rejected clip, move it to the speaker's enrollment folder and re-run enrollment:

mv data/rejections/rejected_20260315_143045_a1b2c3d4.wav data/enrollment/john/
docker compose run --rm --entrypoint python wyoming-voice-match -m scripts.enroll --speaker john
docker compose restart wyoming-voice-match

Re-enrollment

To update a speaker's voiceprint, add more WAV files to data/enrollment/<speaker>/ and re-run enrollment. The script processes all WAV files in the folder to generate an updated voiceprint.

Record additional samples using the same method as initial enrollment, then re-run:

docker compose run --rm --entrypoint python wyoming-voice-match -m scripts.enroll --speaker john
docker compose restart wyoming-voice-match

Demo: See Extraction in Action

Want to hear exactly what gets sent to your ASR service? The demo script runs the full pipeline on a WAV file and writes the extracted audio so you can compare before and after.

Place a WAV file in your data/ folder (any sample rate or channel count - it will be converted automatically), then run:

docker compose run --rm --entrypoint python wyoming-voice-match \
  -m scripts.demo \
  --speaker john \
  --input /data/test_audio.wav \
  --output /data/cleaned.wav

The script uses VERIFY_THRESHOLD and EXTRACTION_THRESHOLD from your docker-compose.yml environment, so it behaves exactly like the main service.

The script will:

Verify the speaker against all enrolled voiceprints (showing similarity scores)
Run speaker extraction, showing each detected speech region and whether it was kept or discarded
Write the result as a WAV file containing only your voice

This is useful for understanding how the extraction works, tuning your thresholds, or just confirming that TV audio is being properly removed.

Performance

Speaker verification latency: ~5-25ms on GPU, ~200-500ms on CPU
Speaker extraction: ~15-35ms on GPU for a typical 10-15s buffer
Memory usage: ~500MB (model + PyTorch runtime)
Accuracy: ECAPA-TDNN achieves 0.69% Equal Error Rate on VoxCeleb1, state of the art for open-source speaker verification

Limitations

Short commands (under 1-2 seconds) produce less audio for verification, reducing accuracy
Voice changes from illness, whispering, or shouting may lower similarity scores - enroll with varied samples to improve robustness
Satellite listening animation may continue after the command has been processed, since the satellite's VAD doesn't know the proxy already responded
Multiple users are supported - enroll each person separately and the service accepts audio from any enrolled speaker

License

MIT License. See LICENSE for details.

Acknowledgments

SpeechBrain for the ECAPA-TDNN speaker verification model
Wyoming Protocol by the Open Home Foundation
Home Assistant voice pipeline ecosystem

Name		Name	Last commit message	Last commit date
.github		.github
scripts		scripts
tools		tools
wyoming_voice_match		wyoming_voice_match
.dockerignore		.dockerignore
.gitignore		.gitignore
DESIGN.md		DESIGN.md
Dockerfile		Dockerfile
Dockerfile.cpu		Dockerfile.cpu
Dockerfile.jetson		Dockerfile.jetson
LICENSE		LICENSE
README.md		README.md
docker-compose.cpu.yml		docker-compose.cpu.yml
docker-compose.jetson.yml		docker-compose.jetson.yml
docker-compose.yml		docker-compose.yml
requirements.cpu.txt		requirements.cpu.txt
requirements.txt		requirements.txt

Navigation Menu

Uh oh!

Repository files navigation

Wyoming Voice Match

Table of Contents

The Problem

Before and After

How It Works

Step by step

Requirements

Quick Start

1. Create a Project Directory

2. Create `docker-compose.yml`

3. Enroll Your Voice

Recording from a Satellite

4. Start the Service

5. Configure Home Assistant

Configuration

Environment Variables

Tuning the Threshold

Noisy Environment Tuning

Speaker Tagging

Rejected Audio Logging

Re-enrollment

Demo: See Extraction in Action

Performance

Limitations

License

Acknowledgments

About

Releases

Sponsor this project

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Wyoming Voice Match

Table of Contents

The Problem

Before and After

How It Works

Step by step

Requirements

Quick Start

1. Create a Project Directory

2. Create docker-compose.yml

3. Enroll Your Voice

Recording from a Satellite

4. Start the Service

5. Configure Home Assistant

Configuration

Environment Variables

Tuning the Threshold

Noisy Environment Tuning

Speaker Tagging

Rejected Audio Logging

Re-enrollment

Demo: See Extraction in Action

Performance

Limitations

License

Acknowledgments

About

Topics

Resources

Stars

Watchers

Forks

Releases

Sponsor this project

Packages

Contributors

Languages

2. Create `docker-compose.yml`