Open-vocabulary keyword spotting for a desktop command

By Shank

Open-vocabulary keyword spotting for a desktop command

I was exploring options for a simple voice-invoked command trigger for my project. These days, wake words and keyword triggers are everywhere—implemented in most IDEs and agent-style apps as a first layer in front of larger ASR and STT workflows, or to trigger other functions.

I wanted to try it in a desktop app: treat a spoken phrase the same way as a button click. The user says a short phrase. The app runs a command it already has. The focus is on performance and simplicity.

That is the first voice layer for an interactive or agent-style workflow. Streaming transcription is probably next. This example is the phrase listener—open-vocabulary keyword spotting on the machine, with phrases I can add while the app is running.

The POC is a .NET 10 Avalonia app. The listener is sherpa-onnx. The command is capture_still. Source for this walkthrough lives in CommandCaptureKWS inside the ASR_STT_Playground repo.

What this example does

The preview window shows a random number + pattern preview. Listen starts the microphone. Capture image saves the current frame as a PNG. These spoken phrases call that same save:

  • “capture image”
  • “take photo”
  • “snap picture”

Add phrase appends another phrase to capture_still. If Listen is already on, the new phrase is armed without closing the microphone.

The spotter returns one of those phrases, or nothing.

Command Capture: Listen on, a spoken phrase triggers capture, and a phrase can be added while listening

flowchart LR
  mic["Microphone"] --> kws["Keyword spotter"]
  kws --> cmd["capture_still"]
  button["Capture image"] --> cmd
  cmd --> png["PNG file"]

Open-vocabulary keyword spotting

Open-vocabulary keyword spotting uses one small acoustic model and a phrase list you supply as text. You can change the list without retraining. Sherpa’s document description is direct: if the keyword is HELLO WORLD, the result is HELLO WORLD or empty. And yes you can add multiple phrases.

The model reads mono float audio at 16 kHz. The app converts the microphone to that rate and passes it to sherpa in chunks of about 100 ms.

Below example loads the English package sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01. It is a Zipformer transducer of about 3.3 million parameters, trained on the GigaSpeech XL subset. The units are byte-pair encodings from bpe.model.

In the example below, phrases are two words each. capture_still has an 800 ms cooldown, so one phrase utterance saves one file.

Keyword spotting options I compared

The requirement I had was specific. A new phrase is typed in (also during runtime). ideally I do not expect to retrain. The library has to run in-process from C#, on a Windows CPU. The output is a phrase or nothing.

Approach How a new phrase is added In this app
Fixed classifier, such as WeKws or microWakeWord Retrain a closed word list No
Porcupine or openWakeWord Train or download a file for that phrase No
Vosk or PocketSphinx with a grammar Edit the grammar text No. Possible, but the acoustics are older, and the Vosk NuGet is the May 2022 build
Research matchers such as CORD-KWS, CTCAT, and PhonMatchNet Enroll text or audio in the paper’s model No desktop SDK for this stack
sherpa-onnx Zipformer keyword spotting Tokenize the phrase into a keyword file Yes

Porcupine is an interesting option and was useful as an API inspiration: audio in, a command name out. Its models and runtime are proprietary, and the engine needs a license key.

Adding a phrase without retraining

k2-fsa publishes two artifacts for this Zipformer model. The catalog mentions that the PyTorch checkpoint is for fine-tuning and the ONNX file is for deployment. we use the ONNX file.

A typed phrase is encoded with the model’s SentencePiece vocabulary and written into the keyword file. The weights do not change. sherpa reads that file when the spotter is created, so the app builds a new spotter and keeps the microphone open.

flowchart LR
  typed["Typed phrase"] --> tokens["SentencePiece tokens"]
  tokens --> kwfile["Keyword file"]
  kwfile --> spotter["KeywordSpotter"]
  onnx["Published ONNX model"] --> spotter

Fine-tuning is the other path. icefall pull request 1428 trained this English model, then measured it on open-commands against GigaSpeech test audio. “Original” means the published model with the phrases supplied as text. “Fine-tuned” means further training on recordings of those commands.

English set Recall, published model Recall after fine-tuning False alarms per hour
Small, 307 utterances 86% 98.7% 0.025, then 0.6
Large, 3,994 utterances 83.6% 97.9% 0.45, then 1.3

Fine-tuning raises recall and also raises false alarms. It needs command audio and a new export. Currently we don’t want top do that.

Model files the app loads

The NuGet package is org.k2fsa.sherpa.onnx 2.0.1. ONNX Runtime runs the networks. The package includes the native library, so the app does not build sherpa from source.

On first Listen, the app downloads the archive from the kws-models release, checks its SHA-256, and unpacks it under %LocalAppData%\CommandCapture\models.

Sherpa’s pretrained KWS docs recommend an INT8 encoder, a float decoder, and an INT8 joiner on CPU. INT8 is the same trained network with quantized MatMul weights. This app loads that mix.

File Role Size
encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx Audio features over time, with streaming state 4.6 MB
decoder-epoch-12-avg-2-chunk-16-left-64.onnx Next-token prediction from the last two tokens 1.1 MB
joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx Scores each token from the encoder and decoder vectors 160 KB
tokens.txt Token ids 4.9 KB
bpe.model SentencePiece model used to encode a phrase 240 KB

The spotter is configured at 16 kHz, with 80 feature dimensions, one thread, and provider cpu. sherpa computes the filterbank inside AcceptWaveform.

How the Zipformer spotter runs

The model is a streaming transducer. The encoder is Zipformer, from Yao et al., Zipformer (ICLR 2024). In that paper the encoder is a full speech recognizer, tens of millions of parameters. This package keeps the same encoder design and uses a checkpoint of about 3.3 million parameters. sherpa then searches only the phrases in the keyword file, so the result is one of those phrases or nothing.

The app hands sherpa about 100 ms of 16 kHz audio at a time. sherpa turns those samples into filterbank frames, 80 numbers per frame, 100 frames per second. One encoder update needs more than one of those microphone chunks. The stream keeps its state between chunks, so “capture image” can start in one chunk and finish in a later one.

Three ONNX networks run on each update:

  1. Encoder. Reads the new filterbank frames, plus caches from audio it has already seen. It outputs one vector per encoder frame.
  2. Decoder. Looks only at the last two tokens the search has accepted. It predicts which token could come next. It does not see the audio.
  3. Joiner. Adds the encoder vector and the decoder vector, applies tanh, and a linear layer scores every token in tokens.txt.

A beam search then keeps the paths that can still become a phrase in the keyword file. The library default is 4 active paths. A match also has to be followed by one blank token, so the spotter reports the phrase after it has ended, not halfway through a longer word. Reset then clears the stream, and listening continues.

The encoder does this work at more than one speed. Filterbank frames arrive at 100 Hz, one every 10 ms. A convolution at the front halves that to 50 Hz. Six stacks then run at 50 Hz, 25 Hz, 12.5 Hz, 6.25 Hz, 12.5 Hz, and 25 Hz. The middle stacks therefore step less often. The vector the joiner receives is at 25 Hz, about one vector every 40 ms.

flowchart LR
  fb["Filterbank, 100 Hz"] --> front["Conv front-end, 50 Hz"]
  front --> stacks["Six stacks, 50 Hz down to 6.25 Hz"]
  stacks --> frames["Encoder frames, 25 Hz"]
  frames --> join["Joiner"]

The English file name chunk-16-left-64 is that streaming setup. chunk-16 means each encoder update consumes 16 frames at the 50 Hz rate. That is 16 / 50 seconds, about 320 ms of new audio, so the encoder advances about three times a second while the microphone is still delivering 100 ms chunks. left-64 is the cache: 64 frames of audio already heard, also at 50 Hz, about 1.3 seconds. Those caches are why a phrase can cross a chunk boundary.

Keyword file format

Each phrase becomes one line. The tokens come from bpe.model. This app writes a boost, a threshold, and an alias:

TOKEN TOKEN TOKEN :1.5 #0.25 @capture_image
Mark Value here Meaning
: 1.5 Boost. Helps this phrase survive the beam. Higher triggers more easily.
# 0.25 Trigger threshold, from 0 to 1. Lower triggers more easily.
@ capture_image Alias returned to the app. Spaces in the phrase become underscores.

capture image, take photo, and snap picture are three phrases for one command, capture_still. Each phrase gets its own line. The alias is the phrase with spaces turned into underscores, so the three aliases are @capture_image, @take_photo, and @snap_picture.

For every phrase the app stores three lookup keys, and each key points at capture_still: the words as typed, the alias, and the SentencePiece token string. sherpa returns the keyword line, including the alias. The app keeps the text after @ and looks that alias up, so any of those three phrases runs capture_still.

foreach (var command in catalog.Directives)
{
    foreach (var phrase in command.Phrases)
    {
        var tokens = encoder.EncodePhrase(phrase);
        var alias = phrase.Replace(' ', '_');
        var line = string.Create(
            CultureInfo.InvariantCulture,
            $"{tokens} :{Boost} #{Threshold} @{alias}");
        lines.Add(line);
        map[phrase] = (command.Name, phrase);
        map[alias] = (command.Name, phrase);
        map[tokens] = (command.Name, phrase);
    }
}
flowchart LR
  audio["16 kHz mono"] --> fbank["80-dim filterbank"]
  fbank --> enc["Zipformer encoder"]
  enc --> join["Joiner"]
  dec["Decoder"] --> join
  join --> beam["Beam search"]
  kwfile["Keyword file"] --> beam
  beam --> hit["Phrase, or nothing"]

How the solution is split

The UI is Avalonia. Audio capture is NAudio WASAPI.

flowchart TB
  subgraph ui ["CommandCapture.App"]
    window["Window: Listen, Capture, Add phrase"]
    handler["StillCaptureHandler"]
  end
  subgraph voice ["CommandCapture.Voice"]
    catalog["commands.json"]
    pipe["VoicePipeline"]
    spot["SherpaKeywordListener"]
    disp["CommandDispatcher"]
  end
  subgraph audio ["CommandCapture.Audio"]
    wasapi["WasapiPcmSource"]
  end
  subgraph cam ["CommandCapture.Camera"]
    preview["PreviewCameraSession"]
  end
  window --> pipe
  window --> preview
  catalog --> pipe
  pipe --> wasapi
  pipe --> spot
  pipe --> disp
  disp --> handler
  handler --> preview

The preview in this build is a moving test pattern. The voice code only calls CaptureStillAsync. The Capture button calls that same method.

public interface IVoiceDirectiveHandler
{
    string CommandName { get; }
    Task HandleAsync(VoiceCommand command, CancellationToken cancellationToken);
}

The catalog shipped with the app:

{
  "directives": [
    {
      "name": "capture_still",
      "phrases": ["capture image", "take photo", "snap picture"],
      "cooldownMs": 800
    }
  ]
}

From the microphone to a saved PNG

Listen loads the model if it is already on disk, writes commands-keywords.txt, creates one KeywordSpotter and one live stream, and starts WASAPI. Each chunk is about 1,600 samples. A keyword becomes a VoiceCommand. The dispatcher checks the cooldown. The handler saves the frame.

sequenceDiagram
  participant Mic as WASAPI
  participant Pipe as VoicePipeline
  participant Sherpa as Keyword stream
  participant Disp as Dispatcher
  participant App as StillCaptureHandler
  Mic->>Pipe: 100 ms of 16 kHz floats
  Pipe->>Sherpa: AcceptWaveform
  Sherpa->>Sherpa: Decode while IsReady
  Sherpa-->>Pipe: keyword, or nothing
  Pipe->>Disp: VoiceCommand
  Disp->>App: HandleAsync
  App->>App: CaptureStillAsync

The live stream stays open for the whole Listen session. AcceptLive creates it on the first chunk.

public IReadOnlyList<string> AcceptLive(float[] samples)
{
    _live ??= _spotter.CreateStream();
    return Drain(_live, samples, finish: false);
}

private List<string> Drain(OnlineStream stream, float[] samples, bool finish)
{
    if (samples.Length > 0)
    {
        stream.AcceptWaveform(16_000, samples);
    }

    if (finish)
    {
        stream.AcceptWaveform(16_000, _tail);
        stream.InputFinished();
    }

    var found = new List<string>();
    var steps = 0;
    while (_spotter.IsReady(stream))
    {
        _spotter.Decode(stream);
        var keyword = _spotter.GetResult(stream).Keyword;
        if (!string.IsNullOrEmpty(keyword))
        {
            _spotter.Reset(stream);
            found.Add(keyword);
        }

        if (++steps > 10_000)
        {
            break;
        }
    }

    return found;
}

Live listening calls this with finish: false, so the stream stays open. Tests pass finish: true, which appends a short silence and marks the input finished.

The pipeline maps the keyword to a command. Confidence is stored as 1 because the #0.25 line already decided the hit. The raw keyword text is kept so the Heard list can show it.

_dispatcher.TrySubmit(new VoiceCommand(
    hit.Command,
    hit.Phrase,
    Confidence: 1f,
    DateTimeOffset.UtcNow,
    new Dictionary<string, string> { [PhraseTrace.KeywordSlot] = hit.Keyword }));

The dispatcher drops a repeat inside the command’s cooldown. capture_still uses 800 ms. A full queue is reported as “Dropped while busy”.

var cooldown = _catalog.CooldownMs(command.Name);
if (_lastAccepted.TryGetValue(command.Name, out var last) &&
    (now - last).TotalMilliseconds < cooldown)
{
    Report(command, $"Dropped {command.Name} during cooldown.", "Cooldown");
    continue;
}

await _handlers[command.Name].HandleAsync(command, cancellationToken);

The handler saves the latest frame and posts the result back to the UI thread.

public async Task HandleAsync(VoiceCommand command, CancellationToken cancellationToken)
{
    var path = await _camera.CaptureStillAsync(cancellationToken);
    var elapsed = DateTimeOffset.UtcNow - command.Timestamp;
    var keyword = command.Slots.TryGetValue(PhraseTrace.KeywordSlot, out var raw) ? raw : command.Phrase;
    var fileName = Path.GetFileName(path);
    var trace = new PhraseTrace(
        command.Timestamp,
        keyword,
        command.Phrase,
        command.Name,
        $"Saved in {elapsed.TotalMilliseconds:0} ms. {fileName}");
    _ui.Post(() => _saved(
        fileName,
        $"Heard \"{command.Phrase}\" and saved in {elapsed.TotalMilliseconds:0} ms. {path}",
        trace));
}

Adding a phrase while Listen is on

Add phrase writes the new line into commands.json. If the pipeline is running, ReloadCatalog encodes the list again, constructs a new SherpaKeywordListener, swaps it in, and disposes the previous spotter. WASAPI keeps running. The next chunk uses the new keyword file.

sequenceDiagram
  participant UI as Add phrase
  participant JSON as commands.json
  participant Pipe as VoicePipeline
  participant Old as Previous spotter
  participant New as New spotter
  UI->>JSON: append the phrase
  UI->>Pipe: ReloadCatalog
  Pipe->>Pipe: encode tokens and rewrite the keyword file
  Pipe->>New: create KeywordSpotter
  Pipe->>Old: dispose after the swap

The app running

The window has the preview, a level meter, Listen, Capture image, a box for a new phrase, and a Heard list.

Command Capture: Listen on, a spoken phrase triggers capture, and a phrase can be added while listening