MLX Whisper: chunked decoding with punctuation persistence, beam search, and subtitle standard cues#12245
Conversation
…via python3) Brings faster-whisper back to macOS as its own engine. CTranslate2's Whisper decoding gives noticeably better results in some languages (e.g. Arabic) than the other local engines, which makes it worth offering despite being CPU-only (CTranslate2 has no Apple GPU backend). Follows the MLX Whisper pattern exactly: the pip-installed faster-whisper library is driven through a bundled helper script via python3, installation is detected by probing several Python installs (Homebrew, python.org, pyenv, system) for "import faster_whisper", and models download automatically from Hugging Face on first use with newline progress lines so the first run does not look frozen. To compensate for the missing GPU, the helper uses faster-whisper's BatchedInferencePipeline when available (typically 2-4x faster than sequential decoding with identical weights and accuracy, with a sequential fallback for older versions) and pins CTranslate2 to one thread per CPU core. int8 quantization (~2x more) is left opt-in via --compute-type so default accuracy stays exactly stock. Unknown extra arguments from the advanced-settings box are warned about and ignored instead of failing the run. Verified end to end on an Apple Silicon Mac with faster-whisper 1.2.1: model pre-download progress, batched decode, language detection, live segment output, and the SRT result loading path.
…ty timing) Raw Whisper segments make poor subtitles: lines come out far too long or too short and their boundaries ignore sentence structure, which is exactly what made this engine's output feel worse than Purfview's Faster-Whisper-XXL on Windows despite identical recognition quality (XXL's --standard mode does its own sentence-based resegmentation). The helper now requests word-level timestamps and rebuilds the cues itself: words group into sentences (closed at sentence-ending punctuation, including Arabic and CJK marks, or at silence gaps of 1s+), and any sentence exceeding the length or duration limit is balance-split at the soft-punctuation boundary nearest its middle (comma, semicolon, Arabic comma), falling back to the word boundary nearest the middle, recursing until every part fits. Balanced splitting avoids the orphan cues (a lone trailing word half a second long) that greedy last-word-that-fits breaking produces. Each cue is timed from its first word's start to its last word's end, so timecodes hug the actual speech instead of inheriting Whisper's coarse segment bounds. Defaults: 84 chars (two 42-char lines) and 6 seconds per cue, overridable via --max-cue-chars / --max-cue-duration; --raw-segments restores the old output. Verified on synthesized multi-sentence English and Arabic speech: a single 30-second raw segment becomes complete-thought cues splitting on commas with no orphans, and Arabic yields evenly sized 4-5 second cues.
… pauses Testing on real Arabic material showed the batched pipeline produces no punctuation at all in some languages (verified on large-v3: zero punctuation marks batched vs normal sentence punctuation sequential), which in turn broke the sentence-based cue splitting and forced mid-sentence length cuts. A short punctuated prompt in the audio's language restores the punctuation under batching at no speed cost (verified: same run goes from zero marks to normal sentence marks). The helper now applies such a prompt automatically when the language is explicitly chosen (Arabic, English, Turkish table), overridable with --initial-prompt and disabled with --no-punctuation-prompt. Forced splits of oversized runs now also prefer the largest silence gap between words near the middle (a real speech pause, where a subtitler would cut), after soft punctuation and before the plain nearest-to-middle fallback. End-to-end on large-v3 Arabic: cues now end at sentence marks and forced splits land on audible pauses instead of arbitrary word boundaries.
…ed timing The curated punctuation prompts only covered a few languages. Since sequential decoding punctuates correctly in every language (it keeps running text context), the helper now bootstraps the prompt from the model itself for any language without a curated entry, and for language auto-detection: a sequential pass over the first 45 seconds yields in-language punctuated text that primes the batched run. Verified on German (not in the table): full punctuation and sentence-shaped cues via the bootstrap path. The cue pass now applies the industry conventions from the Netflix Timed Text Style Guide and the BBC subtitle guidelines rather than ad-hoc numbers: 84 characters (two 42-character lines) per cue, 7 seconds maximum (was 6), 5/6 second minimum with too-short cues merging into a neighbor when the result still fits, a two-frame minimum gap between cues, and cue ends extended when the reading speed would exceed 20 characters per second and there is room. New flags: --max-cps, --initial-prompt; --max-cue-duration default is now 7. Verified end to end on large-v3 Arabic (regression: sentence cues intact, short cues now merged, gaps enforced) and tiny German (bootstrap path).
MLX runs on the Apple GPU / Neural Engine and is the speed pick on Apple Silicon, but it wrote Whisper's raw segments, which make poor subtitles (lines far too long or too short, boundaries ignoring sentences). Port the Faster Whisper Mac helper's cue layer: word-level timestamps, sentence and silence-gap grouping, balanced splitting at soft punctuation or the largest speech pause, then the Netflix Timed Text Style Guide / BBC conventions (84 chars and 7s max per cue, 5/6s minimum with short-cue merging, two-frame minimum gap, reading speed capped at 20 chars/second). Same flags: --max-cue-chars, --max-cue-duration, --max-cps, --initial-prompt, --raw-segments. MLX decodes sequentially with running text context, so punctuation (which drives the sentence splitting) is intact in every language without the batched-pipeline prompt workaround the faster-whisper helper needs.
…nes live Real-world testing on a 45 minute Arabic video showed Whisper drops punctuation on dialectal Arabic speech even with sequential decoding, so the MLX helper had the same broken sentence splitting the batched faster-whisper path had (English was fine). Verified on this machine: zero punctuation marks without a prompt, normal sentence punctuation with one. The MLX helper now applies the same language-matched punctuation prompt automatically (--initial-prompt overrides, --no-punctuation-prompt disables). For speech where even the prompt cannot recover punctuation, both helpers now fall back to the audio's own rhythm: when a decode yields no sentence marks at all, the pause threshold tightens from 1s to 0.6s so cue grouping follows the speaker's actual pauses instead of degrading to pure length cuts. The MLX console also showed only tqdm frame bars, no transcription, because segments were printed after the whole decode finished (mlx_whisper.transcribe is not a generator). verbose=True makes mlx-whisper print each segment as it is decoded, which with unbuffered stdout streams live lines like the other engines; the helper's own duplicate after-the-fact prints are dropped.
The cue rebuilding the MLX Whisper and Faster Whisper Mac helpers perform (word-timestamp cues per the Netflix Timed Text Style Guide / BBC guideline limits) was only tunable through command-line flags in the advanced-settings box. Surface it in the Whisper post-processing dialog instead, where the related options already live: a toggle for the rebuilding itself plus the three limits (max characters per cue, max seconds per cue, max characters per second), persisted in settings and translated into the helper flags when the engines are invoked. Disabling the toggle maps to --raw-segments. Only these two engines understand the flags, so they are appended only in their invocation blocks; the other engines' post-processing is unchanged.
…rds) A real 45 minute Arabic episode exposed three failure modes, all rooted in mlx-whisper having no voice-activity detection, so intro music and long instrumental stretches reach the decoder: - the punctuation initial prompt echoed as phantom cues over the intro music - degenerate hallucination loops (the same foreign-script token repeated with zero-length timestamps, phantom "music" labels) - the actual dialogue in between decoded fine Layered fix, verified against a synthetic music + Arabic speech + music file that reproduced all three artifacts: - Silero VAD (imported from faster-whisper when installed, skipped otherwise) detects the speech regions and the decode is clipped to them via clip_timestamps, so music never reaches the decoder at all. Timestamps stay absolute: the first cue lands exactly where speech starts after the intro. - hallucination_silence_threshold engages mlx-whisper's own guard for silent stretches inside speech regions. - A sanitize pass (both helpers) drops what still gets through on VAD-less runs: zero-length cues, third-plus identical adjacent cues (loops), cues in a script that cannot belong to the chosen language, and leading cues that only quote the injected punctuation prompt.
The punctuation prompt used to reach the model only once. mlx-whisper applies initial_prompt to the first 30 second window, and with condition_on_previous_text disabled every later window decodes with an empty prompt (prompt_reset_since is advanced past the injected tokens), so on long recordings the model quickly reverts to unpunctuated output in languages like Arabic where training data is mostly unpunctuated. Users saw whole 45 minute episodes with almost no periods or commas while short clips looked fine. Instead of one transcribe call over the whole file, the script now merges the detected speech regions into chunks (gaps up to 3 seconds bridged, chunks capped at 240 seconds) and decodes each chunk separately. Every call gets the punctuation prompt again plus the tail of the previous chunk's text, so both the punctuation hint and the local context survive across the entire recording. Segment timestamps are offset by the chunk start, progress lines print absolute times, and silence and music between regions is never decoded at all, which also removes the hallucination loops those passages used to produce. Files without VAD available fall back to the previous single call.
Whisper's documented way to get names, places, and technical terms spelled correctly is the initial prompt, which biases decoding toward the vocabulary it contains. The post processing dialog gets a "Vocabulary prompt" text box whose value is passed to the Mac engine scripts as --initial-prompt. In the scripts a user prompt used to replace the automatic punctuation prompt, which would have traded punctuation for vocabulary on long recordings. The two now combine (punctuation prompt first, user vocabulary after), and since the MLX script re-applies the prompt on every VAD chunk, the vocabulary reaches the whole recording rather than only the first window. Whisper reads the final 224 prompt tokens, which both parts together stay well within. Also hardens the model download progress shim: huggingface_hub can drive the tqdm subclass through paths where .desc is not populated, which previously raised and forced a fallback download without progress lines.
faster-whisper decodes with beam search (beam_size 5) by default, which is one reason Purfview's Windows build reads better on difficult speech. mlx-whisper has no beam search implementation, but it does support best_of sampling on the temperature fallback path: when a segment fails the compression ratio or log probability checks and is retried at a higher temperature, it can sample several candidates and keep the most likely one. Pass best_of 5, matching openai/whisper's default. mlx-whisper drops the option at temperature 0, so the normal greedy pass is exactly as fast as before; only segments that already failed once pay the extra cost.
Beam search is the main quality lever the Windows Faster-Whisper builds have that MLX lacks: faster-whisper decodes with beam size 5 by default, keeping the five best hypotheses at every step, while mlx-whisper only implements a greedy decoder (requesting a beam raises NotImplementedError upstream). The library is almost ready for it: the group batch tiling, the maximum likelihood ranker, and an Inference.rearrange_kv_cache hook all exist; only the decoder class is missing. This ports openai/whisper's reference BeamSearchDecoder to MLX arrays and installs it by swapping the decoder inside DecodingTask at runtime, so no fork of the pip package is needed. Two MLX specific adaptations were required beyond the straight port. The library's rearrange_kv_cache cannot be used as is: the cache mixes self attention entries (batch = beam count) with cross attention entries (batch = audio count), and reindexing the latter with beam indices reads out of bounds, which MLX does not check on the GPU; the decoder therefore reorders only the arrays whose batch dimension matches the beam count. And the top k candidate selection runs on the GPU, so each step transfers beam_size + 1 values per beam instead of a vocabulary sized probability matrix. The finalize step pads candidates with EOT to satisfy the caller's rectangular array contract, and the temperature fallback path is untouched (mlx-whisper already drops beam_size on retries and best_of at temperature zero, so the options never collide). With the cross attention arrays left in place and the candidate transfer reduced, beam search decodes 44 seconds of Arabic with large-v3-turbo in the same wall time as greedy on an M series GPU, so it is enabled by default with beam size 5, matching faster-whisper. A "Beam size" field in the post processing dialog tunes it (0 gives the old greedy path); the value is passed as --beam-size to both Mac helpers.
The settings grid declares its rows explicitly, and the vocabulary prompt and beam size rows were added without extending the definitions, so those two rows plus the button panel all collapsed into the last defined row and rendered on top of each other. Two more auto rows bring the definitions back in line with the content.
Whisper sometimes closes a sentence with a period even though the previous token already ended it with a question mark, comma, or exclamation mark, leaving cues ending in "?." or ",." in the subtitle. The cue sanitizing pass now drops a period that directly follows another punctuation mark, in both Mac helpers. Ellipses are preserved.
Testing on Apple Silicon showed the CTranslate2 based engine is too slow to be practical (CTranslate2 has no Apple GPU backend, so it always decodes on the CPU), and with beam search and per chunk punctuation prompts the MLX engine now matches its accuracy while being several times faster. Removing the engine keeps the engine list to choices that make sense on the platform.
The engine list shows a green or grey dot for engines Subtitle Edit can download, but MLX Whisper is installed via pip and was excluded, so it never showed whether it was ready to use. The dot logic now surfaces the pip install state for it: green when the mlx-whisper package is importable, grey when not. The beam size option also gets a short explanation line in the post processing dialog: 0 decodes fastest, 5 gives the best accuracy at roughly 20-30% more time, and values above 5 rarely help.
|
Console Log: |
|
Thanks for this — reviewed against current A few things worth addressing before merge: 1. (blocking-ish) Version-guard the new 2. 3. 4. Cross-chunk prompt echo not fully sanitized. 5. (low) 6. (low) Single VAD region > 240 s isn't capped. The Empty-audio / last-chunk / all-silence paths are handled safely, and arg quoting is fine ( Heads-up on default behavior: with the new defaults ( Net: sound design and the merge is clean — please address #1 (and ideally #2–#4) and I think it's good to go. The +560 lines of Python are hard to fully validate without an Apple-Silicon runtime, so a note on how you tested end-to-end would help. (Review assisted by Claude Code.) |
- Feature-detect the installed mlx-whisper (transcribe signature and DecodingOptions fields) and drop unsupported options with a printed note instead of aborting the run with a TypeError: word_timestamps falls back to raw segments, hallucination_silence_threshold relies on VAD clipping alone, best_of and beam_size decode greedily, and initial_prompt is guarded the same way. - Cap the reading speed end extension at the gap before the next cue; with near simultaneous starts the old formula could push a cue past the next cue's start and create an overlap. - Drop leading segments of a chunk whose text is contained in that chunk's injected prompt, covering echoes of the previous chunk's tail as well as the punctuation prompt. - Keep cues that contain no letters or digits at all; the script check has no signal to judge them by. - Split single VAD regions longer than the 240 second cap after merging, so the prompt refresh interval also holds for uninterrupted speech.
|
Thanks for the thorough review. All six points are addressed in 918ac71:
The PR description now has a "Default behavior change" section and a note on how the end-to-end testing was done. Regression run after the changes (two speech regions separated by music, beam size 5): 2 VAD chunks, correct absolute timestamps, punctuation in both regions, 16 cues, zero overlaps. |
2 participants
Problem
On macOS the MLX Whisper engine produced noticeably worse subtitles than Purfview's Faster-Whisper-XXL on Windows, especially on long recordings and in languages like Arabic:
initial_promptonly to the first 30 second window, and with conditioning reset the later windows decode with an empty prompt, so the model falls back to unpunctuated output in languages whose training data is mostly unpunctuated. Since punctuation drives sentence segmentation, cue quality degraded with it.What changed
The transcribe helper now runs a proper subtitling pipeline:
best_of 5is also passed for the temperature fallback path, matching openai/whisper defaults.--raw-segmentsrestores plain Whisper segmentation.[MM:SS.mmm --> MM:SS.mmm] textwith absolute timestamps while decoding, plus model download progress lines with percentages.UI additions (post processing dialog, applying to this engine):
The engine list also shows the usual install dot for MLX Whisper now (green when the pip package is importable, grey when not).
Practical results
Measured on an Apple M5, whisper-large-v3, beam size 5, a 45.8 minute Arabic TV episode:
Screenshots of the new dialog section and sample output follow in a comment.
Default behavior change
With the new defaults (cue rebuild on, beam size 5), existing MLX users' next run switches to beam search, word timestamp resegmentation, and VAD chunking. VAD engages only when the faster-whisper package is importable; otherwise the script prints a note and decodes in one pass. Both behaviors can be turned off in the post processing dialog (cue rebuild checkbox, beam size 0), and
--raw-segmentsrestores plain Whisper segmentation. On older mlx-whisper versions the script feature-detects what the installed version supports and degrades gracefully with a printed note instead of failing.How this was tested end to end
sayplus generated music, arranged as music / speech / music / speech / music, verifying VAD chunk boundaries and absolute timestamps (first cues at 12.0 s and 75.3 s), punctuation present in both regions, no prompt echo cues over the music, no overlapping cues, and deterministic beam output at temperature 0.