Open-vocabulary keyword spotting for a desktop command
By Shank
I was exploring options for a simple voice-invoked command trigger for my project. These days, wake words and keyword triggers are everywhereâimplemented in most IDEs and agent-style apps as a first layer in front of larger ASR and STT workflows, or to trigger other functions.
I wanted to try it in a desktop app: treat a spoken phrase the same way as a button click. The user says a short phrase. The app runs a command it already has. The focus is on performance and simplicity.
That is the first voice layer for an interactive or agent-style workflow. Streaming transcription is probably next. This example is the phrase listenerâopen-vocabulary keyword spotting on the machine, with phrases I can add while the app is running.
The POC is a .NET 10 Avalonia app. The listener is sherpa-onnx. The command is capture_still. Source for this walkthrough lives in CommandCaptureKWS inside the ASR_STT_Playground repo.
What this example does
The preview window shows a random number + pattern preview. Listen starts the microphone. Capture image saves the current frame as a PNG. These spoken phrases call that same save:
- âcapture imageâ
- âtake photoâ
- âsnap pictureâ
Add phrase appends another phrase to capture_still. If Listen is already on, the new phrase is armed without closing the microphone.
The spotter returns one of those phrases, or nothing.

flowchart LR mic["Microphone"] --> kws["Keyword spotter"] kws --> cmd["capture_still"] button["Capture image"] --> cmd cmd --> png["PNG file"]
Open-vocabulary keyword spotting
Open-vocabulary keyword spotting uses one small acoustic model and a phrase list you supply as text. You can change the list without retraining. Sherpaâs document description is direct: if the keyword is HELLO WORLD, the result is HELLO WORLD or empty. And yes you can add multiple phrases.
The model reads mono float audio at 16 kHz. The app converts the microphone to that rate and passes it to sherpa in chunks of about 100 ms.
Below example loads the English package sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01. It is a Zipformer transducer of about 3.3 million parameters, trained on the GigaSpeech XL subset. The units are byte-pair encodings from bpe.model.
In the example below, phrases are two words each. capture_still has an 800 ms cooldown, so one phrase utterance saves one file.
Keyword spotting options I compared
The requirement I had was specific. A new phrase is typed in (also during runtime). ideally I do not expect to retrain. The library has to run in-process from C#, on a Windows CPU. The output is a phrase or nothing.
| Approach | How a new phrase is added | In this app |
|---|---|---|
| Fixed classifier, such as WeKws or microWakeWord | Retrain a closed word list | No |
| Porcupine or openWakeWord | Train or download a file for that phrase | No |
| Vosk or PocketSphinx with a grammar | Edit the grammar text | No. Possible, but the acoustics are older, and the Vosk NuGet is the May 2022 build |
| Research matchers such as CORD-KWS, CTCAT, and PhonMatchNet | Enroll text or audio in the paperâs model | No desktop SDK for this stack |
| sherpa-onnx Zipformer keyword spotting | Tokenize the phrase into a keyword file | Yes |
Porcupine is an interesting option and was useful as an API inspiration: audio in, a command name out. Its models and runtime are proprietary, and the engine needs a license key.
Adding a phrase without retraining
k2-fsa publishes two artifacts for this Zipformer model. The catalog mentions that the PyTorch checkpoint is for fine-tuning and the ONNX file is for deployment. we use the ONNX file.
A typed phrase is encoded with the modelâs SentencePiece vocabulary and written into the keyword file. The weights do not change. sherpa reads that file when the spotter is created, so the app builds a new spotter and keeps the microphone open.
flowchart LR typed["Typed phrase"] --> tokens["SentencePiece tokens"] tokens --> kwfile["Keyword file"] kwfile --> spotter["KeywordSpotter"] onnx["Published ONNX model"] --> spotter
Fine-tuning is the other path. icefall pull request 1428 trained this English model, then measured it on open-commands against GigaSpeech test audio. âOriginalâ means the published model with the phrases supplied as text. âFine-tunedâ means further training on recordings of those commands.
| English set | Recall, published model | Recall after fine-tuning | False alarms per hour |
|---|---|---|---|
| Small, 307 utterances | 86% | 98.7% | 0.025, then 0.6 |
| Large, 3,994 utterances | 83.6% | 97.9% | 0.45, then 1.3 |
Fine-tuning raises recall and also raises false alarms. It needs command audio and a new export. Currently we donât want top do that.
Model files the app loads
The NuGet package is org.k2fsa.sherpa.onnx 2.0.1. ONNX Runtime runs the networks. The package includes the native library, so the app does not build sherpa from source.
On first Listen, the app downloads the archive from the kws-models release, checks its SHA-256, and unpacks it under %LocalAppData%\CommandCapture\models.
Sherpaâs pretrained KWS docs recommend an INT8 encoder, a float decoder, and an INT8 joiner on CPU. INT8 is the same trained network with quantized MatMul weights. This app loads that mix.
| File | Role | Size |
|---|---|---|
encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx |
Audio features over time, with streaming state | 4.6 MB |
decoder-epoch-12-avg-2-chunk-16-left-64.onnx |
Next-token prediction from the last two tokens | 1.1 MB |
joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx |
Scores each token from the encoder and decoder vectors | 160 KB |
tokens.txt |
Token ids | 4.9 KB |
bpe.model |
SentencePiece model used to encode a phrase | 240 KB |
The spotter is configured at 16 kHz, with 80 feature dimensions, one thread, and provider cpu. sherpa computes the filterbank inside AcceptWaveform.
How the Zipformer spotter runs
The model is a streaming transducer. The encoder is Zipformer, from Yao et al., Zipformer (ICLR 2024). In that paper the encoder is a full speech recognizer, tens of millions of parameters. This package keeps the same encoder design and uses a checkpoint of about 3.3 million parameters. sherpa then searches only the phrases in the keyword file, so the result is one of those phrases or nothing.
The app hands sherpa about 100 ms of 16 kHz audio at a time. sherpa turns those samples into filterbank frames, 80 numbers per frame, 100 frames per second. One encoder update needs more than one of those microphone chunks. The stream keeps its state between chunks, so âcapture imageâ can start in one chunk and finish in a later one.
Three ONNX networks run on each update:
- Encoder. Reads the new filterbank frames, plus caches from audio it has already seen. It outputs one vector per encoder frame.
- Decoder. Looks only at the last two tokens the search has accepted. It predicts which token could come next. It does not see the audio.
- Joiner. Adds the encoder vector and the decoder vector, applies
tanh, and a linear layer scores every token intokens.txt.
A beam search then keeps the paths that can still become a phrase in the keyword file. The library default is 4 active paths. A match also has to be followed by one blank token, so the spotter reports the phrase after it has ended, not halfway through a longer word. Reset then clears the stream, and listening continues.
The encoder does this work at more than one speed. Filterbank frames arrive at 100 Hz, one every 10 ms. A convolution at the front halves that to 50 Hz. Six stacks then run at 50 Hz, 25 Hz, 12.5 Hz, 6.25 Hz, 12.5 Hz, and 25 Hz. The middle stacks therefore step less often. The vector the joiner receives is at 25 Hz, about one vector every 40 ms.
flowchart LR fb["Filterbank, 100 Hz"] --> front["Conv front-end, 50 Hz"] front --> stacks["Six stacks, 50 Hz down to 6.25 Hz"] stacks --> frames["Encoder frames, 25 Hz"] frames --> join["Joiner"]
The English file name chunk-16-left-64 is that streaming setup. chunk-16 means each encoder update consumes 16 frames at the 50 Hz rate. That is 16 / 50 seconds, about 320 ms of new audio, so the encoder advances about three times a second while the microphone is still delivering 100 ms chunks. left-64 is the cache: 64 frames of audio already heard, also at 50 Hz, about 1.3 seconds. Those caches are why a phrase can cross a chunk boundary.
Keyword file format
Each phrase becomes one line. The tokens come from bpe.model. This app writes a boost, a threshold, and an alias:
TOKEN TOKEN TOKEN :1.5 #0.25 @capture_image
| Mark | Value here | Meaning |
|---|---|---|
: |
1.5 | Boost. Helps this phrase survive the beam. Higher triggers more easily. |
# |
0.25 | Trigger threshold, from 0 to 1. Lower triggers more easily. |
@ |
capture_image |
Alias returned to the app. Spaces in the phrase become underscores. |
capture image, take photo, and snap picture are three phrases for one command, capture_still. Each phrase gets its own line. The alias is the phrase with spaces turned into underscores, so the three aliases are @capture_image, @take_photo, and @snap_picture.
For every phrase the app stores three lookup keys, and each key points at capture_still: the words as typed, the alias, and the SentencePiece token string. sherpa returns the keyword line, including the alias. The app keeps the text after @ and looks that alias up, so any of those three phrases runs capture_still.
foreach (var command in catalog.Directives)
{
foreach (var phrase in command.Phrases)
{
var tokens = encoder.EncodePhrase(phrase);
var alias = phrase.Replace(' ', '_');
var line = string.Create(
CultureInfo.InvariantCulture,
$"{tokens} :{Boost} #{Threshold} @{alias}");
lines.Add(line);
map[phrase] = (command.Name, phrase);
map[alias] = (command.Name, phrase);
map[tokens] = (command.Name, phrase);
}
}
flowchart LR audio["16 kHz mono"] --> fbank["80-dim filterbank"] fbank --> enc["Zipformer encoder"] enc --> join["Joiner"] dec["Decoder"] --> join join --> beam["Beam search"] kwfile["Keyword file"] --> beam beam --> hit["Phrase, or nothing"]
How the solution is split
The UI is Avalonia. Audio capture is NAudio WASAPI.
flowchart TB
subgraph ui ["CommandCapture.App"]
window["Window: Listen, Capture, Add phrase"]
handler["StillCaptureHandler"]
end
subgraph voice ["CommandCapture.Voice"]
catalog["commands.json"]
pipe["VoicePipeline"]
spot["SherpaKeywordListener"]
disp["CommandDispatcher"]
end
subgraph audio ["CommandCapture.Audio"]
wasapi["WasapiPcmSource"]
end
subgraph cam ["CommandCapture.Camera"]
preview["PreviewCameraSession"]
end
window --> pipe
window --> preview
catalog --> pipe
pipe --> wasapi
pipe --> spot
pipe --> disp
disp --> handler
handler --> preview
The preview in this build is a moving test pattern. The voice code only calls CaptureStillAsync. The Capture button calls that same method.
public interface IVoiceDirectiveHandler
{
string CommandName { get; }
Task HandleAsync(VoiceCommand command, CancellationToken cancellationToken);
}
The catalog shipped with the app:
{
"directives": [
{
"name": "capture_still",
"phrases": ["capture image", "take photo", "snap picture"],
"cooldownMs": 800
}
]
}
From the microphone to a saved PNG
Listen loads the model if it is already on disk, writes commands-keywords.txt, creates one KeywordSpotter and one live stream, and starts WASAPI. Each chunk is about 1,600 samples. A keyword becomes a VoiceCommand. The dispatcher checks the cooldown. The handler saves the frame.
sequenceDiagram participant Mic as WASAPI participant Pipe as VoicePipeline participant Sherpa as Keyword stream participant Disp as Dispatcher participant App as StillCaptureHandler Mic->>Pipe: 100 ms of 16 kHz floats Pipe->>Sherpa: AcceptWaveform Sherpa->>Sherpa: Decode while IsReady Sherpa-->>Pipe: keyword, or nothing Pipe->>Disp: VoiceCommand Disp->>App: HandleAsync App->>App: CaptureStillAsync
The live stream stays open for the whole Listen session. AcceptLive creates it on the first chunk.
public IReadOnlyList<string> AcceptLive(float[] samples)
{
_live ??= _spotter.CreateStream();
return Drain(_live, samples, finish: false);
}
private List<string> Drain(OnlineStream stream, float[] samples, bool finish)
{
if (samples.Length > 0)
{
stream.AcceptWaveform(16_000, samples);
}
if (finish)
{
stream.AcceptWaveform(16_000, _tail);
stream.InputFinished();
}
var found = new List<string>();
var steps = 0;
while (_spotter.IsReady(stream))
{
_spotter.Decode(stream);
var keyword = _spotter.GetResult(stream).Keyword;
if (!string.IsNullOrEmpty(keyword))
{
_spotter.Reset(stream);
found.Add(keyword);
}
if (++steps > 10_000)
{
break;
}
}
return found;
}
Live listening calls this with finish: false, so the stream stays open. Tests pass finish: true, which appends a short silence and marks the input finished.
The pipeline maps the keyword to a command. Confidence is stored as 1 because the #0.25 line already decided the hit. The raw keyword text is kept so the Heard list can show it.
_dispatcher.TrySubmit(new VoiceCommand(
hit.Command,
hit.Phrase,
Confidence: 1f,
DateTimeOffset.UtcNow,
new Dictionary<string, string> { [PhraseTrace.KeywordSlot] = hit.Keyword }));
The dispatcher drops a repeat inside the commandâs cooldown. capture_still uses 800 ms. A full queue is reported as âDropped while busyâ.
var cooldown = _catalog.CooldownMs(command.Name);
if (_lastAccepted.TryGetValue(command.Name, out var last) &&
(now - last).TotalMilliseconds < cooldown)
{
Report(command, $"Dropped {command.Name} during cooldown.", "Cooldown");
continue;
}
await _handlers[command.Name].HandleAsync(command, cancellationToken);
The handler saves the latest frame and posts the result back to the UI thread.
public async Task HandleAsync(VoiceCommand command, CancellationToken cancellationToken)
{
var path = await _camera.CaptureStillAsync(cancellationToken);
var elapsed = DateTimeOffset.UtcNow - command.Timestamp;
var keyword = command.Slots.TryGetValue(PhraseTrace.KeywordSlot, out var raw) ? raw : command.Phrase;
var fileName = Path.GetFileName(path);
var trace = new PhraseTrace(
command.Timestamp,
keyword,
command.Phrase,
command.Name,
$"Saved in {elapsed.TotalMilliseconds:0} ms. {fileName}");
_ui.Post(() => _saved(
fileName,
$"Heard \"{command.Phrase}\" and saved in {elapsed.TotalMilliseconds:0} ms. {path}",
trace));
}
Adding a phrase while Listen is on
Add phrase writes the new line into commands.json. If the pipeline is running, ReloadCatalog encodes the list again, constructs a new SherpaKeywordListener, swaps it in, and disposes the previous spotter. WASAPI keeps running. The next chunk uses the new keyword file.
sequenceDiagram participant UI as Add phrase participant JSON as commands.json participant Pipe as VoicePipeline participant Old as Previous spotter participant New as New spotter UI->>JSON: append the phrase UI->>Pipe: ReloadCatalog Pipe->>Pipe: encode tokens and rewrite the keyword file Pipe->>New: create KeywordSpotter Pipe->>Old: dispose after the swap
The app running
The window has the preview, a level meter, Listen, Capture image, a box for a new phrase, and a Heard list.

Links
- CommandCaptureKWS source (
ASR_STT_Playground) - Keyword spotting in sherpa-onnx
- Pretrained keyword-spotting models
- kws-models release
- k2-fsa KWS catalog
- sherpa-onnx, C# API, NuGet package
- icefall PR 1428 and the open-commands test set
- GigaSpeech
- Yao et al., Zipformer, ICLR 2024
- CORD-KWS, CTCAT, PhonMatchNet, WeKws