The Speech Model Is Not the Dictation App

Connect a microphone to a speech-to-text model such as Whisper, and a working dictation demo can take an afternoon. The transcript comes back clean. It feels finished.

Daily use tells a different story. A dictation app can recognize every word correctly and still change what you meant. It can use a fast model and still keep you waiting. It can produce an excellent transcript on the first try, then lose your next recording when the connection drops.

These are problems with the system around the model: the harness. It handles everything from your microphone to the text that lands in your editor.

Dictating a deployment instruction into Claude Code makes those jobs concrete. Cleanup must preserve "only staging" and "leave production alone." Bluetooth microphone changes and failed uploads need recovery paths. Processing during recording can reduce the final wait, while an encrypted local queue keeps unfinished recordings available to retry.

Those details explain why choosing a speech model is only the beginning of building a dictation app.

The recording that started AICHE

I started building AICHE after losing about twenty minutes of a dictated software specification. Processing failed, and the recording was gone, along with the thinking already done inside it. The second attempt was not a repeat; it was a reconstruction from memory.

That gave the product its first requirement: a processing failure must not destroy the recording.

In AICHE, a recording that cannot be processed goes to Later, an encrypted local queue, ready for another attempt. You can also record there deliberately with no connection at all, and the queue processes itself when connectivity returns. Capture has succeeded; transcription can wait. The full behavior is documented in Never Lose a Recording and Offline Recording with Auto-Resume.

Behind that simple behavior sits a lot of client-side work: Bluetooth devices coming and going, interrupted uploads, crashes, operating systems that suspend background activity. No product can promise that no recording is ever lost. The mission is to treat every new way of losing one as the next thing to fix.

The user should not need to understand any of this. They should know where their recording is.

What is a speech-to-text harness?

A speech model takes audio and produces text. The harness makes that capability useful inside an application.

Coding agents offer a familiar parallel. A language model can write code, but the agent around it also needs files, tools, instructions, context, and ways to check its work. Anthropic's engineering work on agent harnesses shows how much the surrounding system contributes to the result.

Dictation has its own set of tasks. The application must capture the right audio, survive failures, preserve meaning during cleanup, and deliver the result where the user expects it. The workflow looks simple (press ⌃+⌥+R on Mac or Ctrl+Alt+R on Windows and Linux, speak, press the same keys again), but the harness starts before the first press and keeps working after the second.

Credit where it is due: OpenAI's Whisper, with its unusually broad language coverage, helped make sophisticated speech products practical for small teams, ours included. We have benchmarked speech-to-text models for developer workflows, and model choice matters. But no model can fix audio that was never captured, or a constraint deleted by a later rewrite.

Cleaner text can lose your meaning

This is one of the hardest parts of building AI dictation.

Spoken thoughts contain repetitions, corrections, unfinished phrases, and unfamiliar names. A language model can make them easier to read. It can also remove something important while doing so.

Consider this illustrative instruction to a coding agent:

Restart the staging worker, only staging, and leave production alone even if it has the same error.

An overly aggressive rewrite might produce:

Restart the affected worker to resolve the error.

The second sentence reads smoothly. It has lost the environment restriction and the instruction to leave production alone. For an AI coding workflow, that is the difference between an awkward instruction and a perfectly clear instruction to do the wrong thing.

Most changes are less dramatic. "Probably" disappears. "After approval" becomes an unconditional step. A repeated warning gets treated as filler. An unusual company name gets replaced with a familiar word.

One user described the repetition problem neatly:

I use repetition deliberately, for emphasis and rhythm.

They reported that Wispr Flow's cleanup removed those repetitions. One person's experience does not say how often this happens, but it captures the trade-off exactly: repetition can be noise or meaning, and a rule like "remove repeated words" cannot tell the difference on its own.

For people dictating prompts to AI, this matters in a subtler way too. An assistant may work out an unfamiliar name from its rough phonetic spelling. Once cleanup replaces that spelling with an unrelated ordinary word, the clue is gone.

The goal is nuance preservation: fix what can be fixed while keeping the speaker's constraints, uncertainty, emphasis, and intent.

The cleanup model must know its job

Another failure happens when the polishing model responds to the transcript instead of formatting it.

A Superwhisper user who wanted to dictate into GPT or Claude wrote:

If I give it a problem, I don't want it to do AI work to try and give me a solution.

Their request was punctuation and formatting, then pasting into another AI. Superwhisper also covers this failure category, AI answering instead of transcribing, in its own troubleshooting documentation.

The distinction is simple to explain and surprisingly hard to enforce. "Explain this error" may be text the user wants to send elsewhere. The cleanup model must preserve it as content, not start explaining the error itself. That takes careful instructions and testing across real examples. A prompt that behaves well with a friendly email may behave differently with a technical specification or a dictated command.

Each rewrite needs a reference point

Several processing steps can compound the problem.

If the first rewrite drops a qualification, and the next step sees only that rewrite, it has no reliable way to recover what disappeared. It may make the text even clearer while carrying the omission forward.

In AICHE, later refinement stays grounded in source material. We do not treat the previous polished version as unquestionable truth. The raw transcript can itself contain recognition errors, so it is evidence rather than infallible ground truth, but keeping it available helps prevent later stages from adding new losses on top of old ones. Different transformations also need different rules: punctuation, translation, and summarization have different jobs and different permissions to change things.

Our testing has to follow meaning through the whole chain. A prohibition should remain a prohibition. An uncertain claim should stay uncertain. A proper name should not become a confident invention.

This also exposes a weakness in measuring quality only by user edits. People can miss a fluent mistake. Text sent without correction is not automatically faithful text.

Fast dictation starts before you stop speaking

A two-hour recording should not mean a long wait when you finish.

The key is when processing begins. If an app waits for the entire recording before starting transcription, all the work lies ahead. AICHE processes live recordings progressively while you speak, so most of the work is already done when you stop. In that workflow, the wait after you stop speaking is typically a few seconds, refinement included.

That is what we observe in our own daily use, not a fixed guarantee: connectivity problems and recovery attempts can make it longer. It also describes live recording. Importing an existing two-hour file is a different job, because none of the processing could happen while you were creating it.

Making this work takes more than splitting up audio. The final text still needs continuity across the pieces: no duplicated phrases, no broken sentences, no qualification lost at a join.

The measure that matters is the time between finishing your thought and having usable text.

Recognition needs recovery paths too

Even with good audio, recognition can return an empty result, mishandle a language, or invent words during silence.

A September 2026 research preprint on Whisper hallucinations shows the trade-off clearly: an inference-time intervention that sharply reduced hallucinations on non-speech benchmarks also cost recognition accuracy on real speech. Suppressing invented words and preserving genuine ones are competing objectives, so "just filter the hallucinations" is not a complete solution.

The pipeline around the model can cause errors too. In its release notes, Wispr Flow described vendor outages and scaling problems in June 2026, UK English and Swiss German routed to the wrong place, audio compression that degraded transcription, and an auto-cleanup setting aggressive enough to hurt accuracy. Publishing that is to their credit, and it is a useful public record of how much of a dictation product's reliability sits outside the model.

This is why a harness needs more judgment than "retry if empty." Silence may correctly produce nothing. A temporary outage may justify another attempt. A consistently wrong result may need a different route. And there has to be a limit to how long the user waits. If processing cannot finish promptly, preserve the recording, say what happened, and make another attempt possible. A recoverable state beats an indefinite spinner.

Privacy is part of the design

Context can improve transcription. A user-supplied dictionary helps with technical terms. Nearby text may help identify a name. But collecting more context also gives the application access to more of the user's activity, so the harness defines a data boundary, not just a processing pipeline.

Wispr's context-awareness documentation describes reading text near the cursor, app information, on-screen text, and file names in code editors. Its data controls page separately covers cloud processing, server-side storage, and model training choices. Those are independent decisions, and a single "private" label cannot describe all of them.

We made different trade-offs. AICHE's desktop apps ship with no analytics, no trackers, and no telemetry, as stated in our privacy policy. In the cloud path, audio streams to the transcription provider, is processed, and is discarded. Fully local processing is already available across all our Linux distributions: speech recognition, refinement, meeting transcription with speaker identification, summaries, and note search, all running on your device without sending your audio or text to the cloud. Other platforms are next. Recordings that cannot be processed wait in the encrypted Later queue on your device, and note sync between devices is end-to-end encrypted with a key you set. The harness has to respect those choices during normal operation and during recovery: a retry or fallback must follow the same data rules as the first attempt.

What makes a good dictation app

Model accuracy, language coverage, and speed all matter. We benchmark them ourselves. Daily use adds a few more tests:

• Did the recording survive when something failed?
• Did the important meaning survive cleanup?
• How long was the wait after speaking?
• Did the text arrive in the right place?
• Could the user recover, and did recovery respect their data choices?

That is the work around the model. It usually becomes visible only when something goes wrong. The point of building a good harness is to make those moments recoverable, and to keep the ordinary ones simple: speak, finish, keep working.