# Fast Accuracy-Preserving RNNT Beam Search on Apple Silicon > **Collective Library edition.** This is the complete technical report. Private filesystem paths, internal run identifiers, campaign-control notes, and repository navigation were removed. Technical claims, code, measurements, evidence labels, citations, corrections, and falsification criteria are preserved. **Date:** 2026-09-06 **Status:** Ingested external field guide; Argmax claims normalized against public primary sources, the MIT-licensed local Parakeet v2 and v3 Core ML artifacts inspected directly, Core ML APIs checked through current coremltools documentation, and `malsd_batch` semantics corrected against pinned NeMo Speech source ## Executive Answer Argmax publicly demonstrates a fast **Parakeet TDT** deployment, not a fast beam decoder for the reference implementation's **Parakeet Unified RNNT**. Argmax reports more than 100 seconds of audio transcribed per wall-clock second on an 8 GB M1 MacBook Air, 10+ TFLOPS on the Apple Neural Engine for Parakeet v2, and 15+ TFLOPS for a ten-minute Parakeet v3 workload on an iPhone 16 Pro. It also reports a 476 MB compressed v2 package and a 494 MB v3 package. Those are vendor measurements of an encoder-heavy TDT system. Public material does not identify its search algorithm or beam width, but the inspectable MIT-licensed v2 and v3 artifacts now settle the model-side implementation: four static Core ML programs, 6-bit-palettized encoder weights, ANE-friendly rank-4 encoder tensors, both joint projections hoisted out of the recurrent joint call, explicit FP16 LSTM state, and a batch-one host-driven search boundary. TDT contributes real but bounded architectural leverage. Its duration head can jump over encoder frames; the TDT paper reports up to 2.82× faster ASR inference than a conventional transducer on its experiments.[4] That cannot explain a 100× speed factor by itself, and it cannot be copied into frozen Unified RNNT weights. The artifact shows the transferable model split and compression choices. The SDK's host search algorithm and benchmark inclusion boundary remain undisclosed. For the reference implementation's RNNT, the shortest credible exact path is: 1. Hoist the encoder and predictor projections out of the joint inner loop. NeMo's current `malsd_batch` does this already; the reference implementation's current `JointDecisionSingleStep` recomputes both projections on every call. 2. Export fixed-batch FP32 predictor and projected-joint graphs for the selected beam width, initially width 8. Never dispatch one Core ML prediction per hypothesis. 3. Reproduce the pinned `malsd_batch` state machine exactly: top-K per hypothesis, global K-of-K² pruning, max-symbol blank forcing, pointer-based history, post-prune duplicate recombination by log-sum-exp, blank-state preservation, and final score normalization. 4. Keep state tensors explicit. `MLState` is a poor first fit for branching hypotheses; output backings are an optional measured optimization, not a zero-copy guarantee. 5. If two Core ML calls per alignment step miss the budget, test one fixed-shape fused transition-plus-joint graph or a native CPU recurrent kernel. Do not start with custom kernels. The most important new implementation observation is not an exotic beam algorithm. It is **projection hoisting**: the current runtime repeatedly pays work that depends only on the encoder frame or current token prefix. Remove that first. ## Evidence Labels | Label | Meaning | | --- | --- | | **Vendor claim** | Argmax publishes the number, but the implementation or full benchmark protocol is not public. | | **Documented upstream** | Current Apple/coremltools or NeMo documentation establishes the API or behavior. | | **Pinned source** | Read directly from NeMo Speech `3.1.0+95f92737cf` or the reference implementation's pinned exporter source. | | **Inspectable Argmax artifact** | Read directly from the MIT-licensed compiled Parakeet v2 and v3 Core ML models named above. | | **Peer reviewed** | A cited paper establishes the result on its models and data, not necessarily on the reference implementation. | | **Implementation artifact** | Read from the reference implementation's code or an immutable/reproducible receipt. | | **Hypothesis** | Plausible but unmeasured here; the settling experiment is named. | ## 1. What Argmax Demonstrably Did ### 1.1. Public performance claims | Claim | What the source actually establishes | Confidence and boundary | | --- | --- | --- | | More than 100× speed factor on an 8 GB M1 MacBook Air | Parakeet v2 on Argmax Pro SDK; speed factor means audio duration divided by processing time.[1] | **Vendor claim, high provenance.** Exact clip, warm/cold state, preprocessing, tokenization, and search policy are not stated. | | 10+ TFLOPS on the ANE | Argmax's Parakeet v2 reimplementation reached “near-peak” ANE utilization.[1] | **Vendor claim.** It establishes an optimized encoder path, not beam-search latency. | | 15+ TFLOPS on iPhone 16 Pro | Ten-minute Parakeet v3 file transcription in Argmax Playground, FP16 ANE, iOS 26; Argmax calls the workload encoder-heavy.[3] | **Vendor claim with a useful boundary.** This is burst file transcription, not end-of-audio finalization for a 30-second Unified window. | | 476 MB v2 and 494 MB v3 packages | Argmax's model-management documentation names those downloadable variants and says v2 compression stays within 0.5% accuracy of the uncompressed model.[2] | **Vendor claim.** “0.5%” is not defined there as absolute or relative WER. | | 160 ms real-time latency | Argmax defines this as lips-to-screen latency and says the slowest measurement was an iPhone 12.[5] | **Vendor claim.** Streaming partial latency is not offline finalization latency. | | Public beam width or beam implementation | No public Argmax source found in this research names one. | **Data unavailable.** High WER or leaderboard agreement does not reveal the decoder. | The raw report asserted a roughly 66 MB memory footprint. No supporting Argmax primary source was found, and it conflicts with the public 476/494 MB package sizes if read as model residency. It is excluded. ### 1.2. Architecture is part of the speed result Parakeet v2 and v3 are TDT models. A TDT predicts a token distribution and a duration distribution, then advances by the selected duration instead of necessarily visiting the next encoder frame. The original paper reports up to 2.82× faster ASR inference than its conventional transducer baselines.[4] NVIDIA's explanation is concrete: long predicted durations skip blank-heavy regions.[6] Three conclusions follow: - TDT frame skipping is not an RNNT beam optimization. - The paper's “up to 2.82×” does not explain Argmax's full speed factor. - A 0.6B encoder at high ANE utilization can dominate the public throughput number while saying little about the recurrent decoder. ### 1.3. What the MIT-licensed v2 and v3 artifacts reveal **Inspectable Argmax artifact.** Both inspected `LICENSE_NOTICE.txt` files now identify the weights as MIT licensed and permit reverse engineering and modification. Each variant is four compiled ML Programs with readable `metadata.json` and MIL. All eight programs target Core ML specification version 8 with iOS 17/macOS 14 availability. The v3 conversion metadata records TorchScript conversion with PyTorch 2.5.0 and coremltools 9.0b1 on 2025-08-14; v2's decoder, joint, and mel graphs record coremltools 8.2 and PyTorch 2.6.0, while its encoder records coremltools 8.3.0 and PyTorch 2.5.0. The two variants share the same split and the same 15-second, 16 kHz window (`audio [240000]` → mel `[1,1,1501,128]` → encoder frames `[1,1024,1,188]` plus projected encoder `[1,640,1,188]`). They differ in vocabulary size, joint width, and how aggressively the frontend is palettized: | Component | Shared interface | v2 stored precision | v3 stored precision | Mechanism visible in MIL | | --- | --- | --- | --- | --- | | `MelSpectrogram` | audio `[240000]` → mel `[1,1,1501,128]` | Dense FP16 | Mixed FP16 and 6-bit palette | Fixed 15-second front end; rank-4 output goes directly into the encoder | | `AudioEncoder` | mel `[1,1,1501,128]` → encoder `[1,1024,1,188]` and projected encoder `[1,640,1,188]` | Mixed FP16 and 6-bit palette | Mixed FP16 and 6-bit palette | Rank-4 channel-first graph; encoder-side joint projection is a final 1×1 convolution computed once over all 188 frames | | `TextDecoder` | token `[1]`, hidden/cell `[2,640]` → projected predictor `[1,640]`, new hidden/cell `[2,640]` | FP16 | FP16 | Two 640-wide LSTMs, then the predictor-side 640→640 joint projection. Embedding table is `[1025, 640]` on v2 and `[8193, 640]` on v3. v3 remaps negative token ids by `+8193` before gather | | `MultimodalLogits` | projected encoder `[1,640]` + projected predictor `[1,640]` → log probabilities `[1, C]` | FP16, `C = 1030` | FP16, `C = 8198` | Only add, ReLU, linear, softmax, and log remain in the recurrent joint graph | The v3 dimensions are strong structural evidence for a TDT joint: 8,193 embedding rows versus 8,198 joint outputs—five extra values consistent with duration choices. v2 is the same transducer split with a small-vocab joint: 1,025 embedding rows versus 1,030 joint outputs, also five extra classes. Neither artifact includes a tokenizer or decoder configuration, so exact blank, token, and duration semantics—and how the host separates or renormalizes their scores—remain unconfirmed. The useful reverse-engineered facts are: 1. **Projection hoisting is real on both variants.** Argmax computes the encoder projection once for the whole utterance and the predictor projection once per decoder update. Its recurrent joint graph never repeats either 640-wide projection. 2. **The encoder is shaped for Core ML's image-oriented execution path.** Every major encoder tensor is rank 4, with sequence in the last dimension; dense channel projections are represented as 1×1 convolutions. 3. **Compression is concentrated in the heavy path.** Both encoders use 64-entry lookup tables—6-bit palettization. v3 also palettizes the mel frontend; v2 leaves mel dense FP16. The small recurrent decoder and joint stay plain FP16 on both. 4. **The Core ML graphs contain no search.** There is no `argmax`, `topk`, beam state, pruning, or prefix operation. `MultimodalLogits` returns the full vector to host code. 5. **The decoder interface is batch one.** The artifacts expose no hypothesis-batch dimension, so they are not evidence of a batched Core ML beam implementation. 6. **The advertised file throughput uses a fixed 15-second model window.** Long-file speed can amortize model load and host setup over many windows; it is a different boundary from the reference implementation's 30-second end-of-audio finalization gate. These facts explain a substantial part of the speed without inventing an SDK algorithm: compact accelerator-friendly encoding, projections removed from the recurrent hot loop, a small FP16 host-driven decoder boundary, and TDT frame skipping. ### 1.4. What remains unknown Public evidence does not settle: - greedy, beam, or another TDT search; - beam width, pruning, score normalization, or maximum symbols; - native Swift, C++, Metal, or mixed host bookkeeping around the four Core ML programs; - whether tokenization, punctuation, timestamps, and segmentation are inside every speed number; - cold load and specialization cost; - per-component latency and memory. Do not replace those unknowns with a story. Inspect the model graph for model-side facts; measure the SDK end to end only when the same audio, device, output contract, and timing boundary can be used. ## 2. The Actual `malsd_batch` Target ### 2.1. Pin implementation, not algorithm name The selected quality result uses NeMo `malsd_batch`, but an algorithm name is not a conformance specification. In pinned NeMo Speech `3.1.0+95f92737cf`, `ModifiedALSDBatchedRNNTComputer` states two modifications to the paper: - it does **not** support prediction-network caching; - it does **not** estimate target length; it limits label expansions per frame.[7] The raw report described a trie-based predictor cache as mandatory. That is incorrect for the pinned target. `BatchedBeamHyps` stores dense tensors of current labels and predecessor beam pointers, plus transcript hashes used for duplicate detection. The history is tree-like, but the MALSD computer does not cache predictor results by prefix.[8] ### 2.2. One pinned iteration For batch `B`, beam `K`, vocabulary-plus-blank `V`, and projected width 640, the source performs: ```text project encoder once [B,T,640] initialize predictor for B*K hypotheses [B*K,1,640] while any hypothesis is active: gather one projected encoder frame per hypothesis [B*K,1,640] joint-after-projection [B*K,1,1,V] log_softmax in the active floating dtype [B,K,V] top K labels for each hypothesis [B,K,K] add previous hypothesis scores [B,K,K] force blank where max-symbol limit fired select global top K from K*K candidates [B,K] append labels and predecessor pointers recombine identical surviving transcripts by log-sum-exp gather parent predictor states run predictor for all B*K selected labels restore previous output/state for blank transitions advance acoustic time only on blank ``` This ordering matters. NeMo prunes K² candidates to K **before** recombining identical surviving hypotheses. Recombination sets all but the highest-scoring representative inactive and log-sum-exp merges their scores; it does not refill the vacated beam slots from discarded candidates.[7][8] An implementation that merges before pruning is a different decoder even if it looks mathematically cleaner. ### 2.3. Equality and recombination For RNNT, pinned `BatchedBeamHyps.recombine_hyps_()` considers hypotheses equal when all three match: - transcript hash; - last label; - non-blank transcript length. It computes `torch.logsumexp` over equal scores and keeps the slot that previously held the highest score.[8] Hash collision is theoretically possible. A Swift port should either reproduce the hash and verify tokens on collision, or compare pointer-chain tokens directly for the at-most-eight survivors. With `K=8`, collision-safe verification is cheap. ### 2.4. Configuration fields that can change output Freeze these in every receipt: | Field | Pinned campaign meaning | | --- | --- | | `strategy` | `malsd_batch` | | `beam_size` | Selected quality width; currently under full-development evaluation | | `max_symbols` | Defaults to 10 in NeMo's `malsd_batch` construction and forces blank when reached | | `score_norm` | Defaults true; changes final hypothesis ordering by normalized score | | `softmax_temperature` | Not passed into `BeamBatchedRNNTInfer` for pinned `malsd_batch`; record the config but do not treat it as an active knob | | `ngram_lm_model`, `ngram_lm_alpha` | None / zero for the no-LM campaign | | `boosting_tree`, `boosting_tree_alpha` | Disabled | | `pruning_mode`, `blank_lm_score_mode` | Relevant when fusion models are present; still record them | | blank index and vocabulary order | 1,032 and the exact 1,033-class order for the tuned archive | | floating dtype | FP32 for the the reference implementation quality-preserving path unless a separate experiment proves otherwise | `BeamRNNTInfer` and `BeamBatchedRNNTInfer` are different code paths. Beam size one is also not the campaign's `greedy_batch`; NeMo warns that cached beam-one greedy may differ from the dedicated greedy decoder.[9] ### 2.5. Test the state machine with a scorer, not a static logit tensor An RNNT does not have one prefix-independent `[T,V]` logit matrix. Its joint logits depend on acoustic time **and** predictor state, which depends on the emitted prefix. The raw report's proposed “extract raw `log_probs` once, then feed both decoders” test cannot establish RNNT beam parity. Use a deterministic toy scorer instead: ```python class ToyRnntScorer: def log_probs(self, time_index: int, token_prefix: tuple[int, ...]) -> list[float]: return self.table[(time_index, token_prefix)] ``` Run the pinned Python state machine and the Swift state machine against the same table. Include fixtures for: - blank versus label ties; - two alignments reaching the same transcript; - recombination after pruning; - max-symbol blank forcing; - an inactive final hypothesis retained in slot zero; - final score normalization changing the winner; - hash collision fallback. Only after the toy state machine matches should model numerics enter the experiment. ## 3. Delete Repeated Projection Work First ### 3.1. Current the reference implementation hot path **Implementation artifact.** `scripts/coreml/components.py` currently implements a full joint wrapper: ```python enc_proj = self.module.enc(encoder_outputs) dec_proj = self.module.pred(decoder_outputs) x = enc_proj.unsqueeze(2) + dec_proj.unsqueeze(1) x = self.module.joint_net(x) ``` `JointDecisionSingleStep` invokes that wrapper for each decoder decision. Therefore: - the same encoder frame is projected from 1,024 to 640 for every label expansion at that frame; - the same predictor output is projected from 640 to 640 until its prefix changes; - the graph computes softmax, argmax, and top 64 for a greedy loop that needs less; - the interfaces are batch one. This is acceptable as a first correct exporter. It is the wrong boundary for a fast beam loop. ### 3.2. NeMo's source already shows the split Pinned `malsd_batch` calls `joint.project_encoder()` once before its loop and `joint.project_prednet()` after predictor updates. The loop calls only `joint_after_projection()`.[7] This is exact at the algorithm level and transfers directly to Core ML. The target graph split is: ```text audio -> mel -> encoder + encoder projection once | +--> projected encoder [1,T,640] selected tokens + parent states -> batched predictor + predictor projection [K,640] + new states projected encoder frames [K,640] + projected predictor [K,640] -> add -> ReLU -> final linear -> log-softmax -> top K + blank score ``` Moving operations across Core ML graph boundaries can change floating-point fusion and accumulation. “Same PyTorch operators” is not a numerical parity result. Compare component outputs and transcript identities before calling the split exact. ### 3.3. Minimal fixed-batch export shape Start with one model per selected beam width. Separate fixed models are simpler to compile, profile, and reject than a multifunction or flexible-shape package. ```python class ProjectedPredictor(torch.nn.Module): def __init__(self, decoder, joint): super().__init__() self.decoder = decoder self.pred_projection = joint.pred def forward(self, tokens, lengths, h, c): y, _, (h_new, c_new) = self.decoder( targets=tokens.long(), target_length=lengths.long(), states=[h, c], ) projected = self.pred_projection(y.transpose(1, 2)).squeeze(1) return projected, h_new, c_new class ProjectedJointTopK(torch.nn.Module): def __init__(self, joint, beam_size: int, blank_idx: int): super().__init__() self.activation = joint.joint_net[0] self.output = joint.joint_net[2] self.beam_size = beam_size self.blank_idx = blank_idx def forward(self, encoder_projected, predictor_projected): logits = self.output(self.activation(encoder_projected + predictor_projected)) log_probs = torch.log_softmax(logits, dim=-1, dtype=torch.float32) scores, ids = torch.topk(log_probs, k=self.beam_size, dim=-1) return scores, ids.to(torch.int32), log_probs[:, self.blank_idx] ``` For beam 8, trace fixed inputs: ```python inputs = [ ct.TensorType(name="tokens", shape=(8, 1), dtype=np.int32), ct.TensorType(name="lengths", shape=(8,), dtype=np.int32), ct.TensorType(name="h_in", shape=(2, 8, 640), dtype=np.float32), ct.TensorType(name="c_in", shape=(2, 8, 640), dtype=np.float32), ] predictor = ct.convert( traced_predictor, convert_to="mlprogram", inputs=inputs, compute_precision=ct.precision.FLOAT32, minimum_deployment_target=ct.target.iOS17, ) ``` Current coremltools also supports `ct.EnumeratedShapes` directly on `ct.TensorType` for a finite shape set.[10] Do not add it initially. Coupled token and state batch dimensions are easier to reason about as separate fixed packages, and only the development winner needs to ship. ### 3.4. Keep top-K inside the joint graph Returning `[K,1,033]` FP32 log probabilities moves about 33 KB per iteration at beam 8. Returning K top scores, K token IDs, and one blank score per hypothesis moves hundreds of bytes. The existing exporter already proves Core ML can compile `topk`. Top-K inside the graph is therefore the first arm to test. Preserve: - `log_softmax` before `topk`; - the same FP32 arithmetic target; - blank score even when blank is not in top K; - deterministic tie handling checked on fixtures. A native partial selector remains a fallback if Core ML's `topk` placement or latency is bad. Do not full-sort 8,264 values per iteration, and do not assume a named Accelerate routine is faster without a microbenchmark. ## 4. The Native Beam Loop ### 4.1. Fixed storage At beam 8, use fixed-capacity storage: ```text scores Float[K] time_indices Int32[K] last_labels Int32[K] nonblank_lengths Int32[K] symbols_at_current_time Int32[K] h, c Float[2,K,640] predictor_projected Float[K,640] history token arena Int32[max_steps,K] history predecessor arena UInt8[max_steps,K] transcript hashes UInt64[K] ``` The predecessor arena avoids copying token arrays when a hypothesis branches. Reconstruct only final transcripts. With eight survivors, recombination can compare hash, last label, and length first, then walk predecessor chains only on a potential match. ### 4.2. One model call per graph, not per hypothesis The safe first port mirrors NeMo: - one batched projected-joint call per alignment iteration; - one batched predictor call after pruning; - restore parent output/state for blank or inactive transitions; - all gathers, score arithmetic, K² pruning, history updates, and recombination in one native loop. Serially calling a batch-one decoder eight times is not a beam implementation worth benchmarking. It multiplies feature-provider construction, scheduling, output allocation, and dispatch. ### 4.3. Explicit states before `MLState` Core ML stateful models use `ct.StateType` and require iOS 18/macOS 15-era state APIs.[11] A state object is naturally a mutable linear history. Beam search branches and reorders eight histories every iteration. Cloning opaque `MLState` objects is not the simple path, and using one shared state would serialize or corrupt branches. Keep `[2,K,640]` hidden and cell tensors explicit until a physical experiment proves a stateful model helps. This preserves the iOS 17/macOS 14 floor and makes gather/scatter observable. ### 4.4. Output backings are a proposal `MLPredictionOptions.outputBackings` can propose client-allocated `MLMultiArray` or pixel-buffer storage for named outputs. Core ML may ignore the proposal; Apple requires checking object identity. It is unsupported for batch prediction, though a normal prediction whose tensor has a batch dimension is not the batch-prediction API.[12] ```swift let options = MLPredictionOptions() options.outputBackings = [ "h_out": hBacking, "c_out": cBacking, "top_scores": scoreBacking, "top_ids": idBacking, ] let result = try model.prediction(from: input, options: options) let usedHBacking = result.featureValue(for: "h_out")?.multiArrayValue === hBacking ``` Record `usedHBacking` rather than claiming zero-copy. Avoid touching a pixel-buffer-backed array's data pointer before prediction; doing so can force synchronization.[12] ### 4.5. A one-dispatch exact candidate If the two-call loop misses latency, test a fixed-shape graph that accepts: - selected labels; - parent hidden/cell states; - parent projected predictor output; - a blank/inactive preserve mask; - selected projected encoder frames. Inside one graph: 1. run the predictor on all K labels; 2. select parent versus new state/output by the preserve mask; 3. run predictor projection; 4. run projected joint, FP32 log-softmax, top-K, and blank extraction. This reduces the steady-state path to one Core ML dispatch per alignment iteration. It is algorithmically equivalent to NeMo's run-then-restore sequence. Core ML may fuse or round differently, so token-level parity—not operator inspection—decides whether it is acceptable. ## 5. Runtime Choices | Recurrent path | Dispatch shape | Main advantage | Main risk | Use when | | --- | --- | --- | --- | --- | | Fixed-batch FP32 Core ML predictor + joint on CPU | Two calls per iteration | Smallest change; portable; inspectable compute plan | Dispatch and state movement may dominate | First implementation | | Fused FP32 Core ML transition + joint | One call per iteration | Deletes one dispatch and intermediate handoff | Graph fusion can change numerics; larger graph | Two-call path misses budget | | Native Accelerate/BNNS recurrent kernel | No Core ML dispatch in loop | Lowest plausible CPU overhead; direct state control | Weight packing, LSTM semantics, and maintenance | Core ML loop is proven dispatch-bound | | MPSGraph or custom Metal | One or more GPU commands per iteration | High matrix throughput | Hundreds of sequential small launches and CPU/GPU synchronization | Only after a named-device trace wins | | ANE predictor/joint | Small repeated programs | Potential energy efficiency | Small recurrent graphs often lose to scheduling; FP32 is unavailable on ANE | Only with direct target-device evidence | | MLX decoder service | Framework-dependent | Fast prototyping | New runtime, memory, packaging, and parity surface | Research arm, not default | There is no source-backed universal winner. The encoder and recurrent loop are different workloads. The ANE residency guide shows why placement and speed must be measured separately on each device. The small-recurrent-graph dispatch guide covers the per-call dispatch floor by compute unit and the measured FP32-versus-FP16 CPU cost behind the "FP32 is unavailable on ANE" row above. ## 6. Exact and Approximate Optimizations | Technique | Classification | Required proof | | --- | --- | --- | | Hoist encoder projection once | Algorithmically exact | Component parity and transcript identity; graph-boundary rounding measured | | Project predictor only after token updates | Algorithmically exact | Same projected vectors and state preservation on blank | | Batch K hypotheses in one graph | Algorithmically exact | Toy-state-machine parity and component parity | | Preallocated history/state arenas | Exact | Same tokens, scores, and predecessor chains | | In-graph FP32 log-softmax and top-K | Exact in real arithmetic; backend numerics may differ | Tie fixtures, top-K membership/order, end-to-end transcript comparison | | Output backings | Exact | Identity check plus latency and allocation reduction | | Fuse predictor transition and joint | Algorithmically exact; numerically exposed | First token/score mismatch gate | | Predictor prefix cache | Potentially exact, but not pinned-NeMo behavior | Dedicated equivalence proof; do not include in first port | | Adaptive beam width | Approximate | Paired WER and retained fraction of beam-over-greedy gain | | Confidence-triggered greedy frames | Approximate unless a valid future-score bound is proven | Exact-hypothesis rate plus paired WER | | Lower per-hypothesis top-K | Approximate | Full-development WER and disagreement decomposition | | FP16 predictor or joint | Approximate on this stack | Component, transcript, and paired-WER receipts | | Lower `max_symbols` | Different decoder | Empty/truncated output checks and paired WER | | TDT, multi-blank RNNT, or CTC head | New model | Full training and untouched-base bakeoff | | Beam-to-greedy distillation | New model | Same frozen development/holdout protocol | An adaptive beam can save useful work, but only after an exact beam exists as an oracle. Cheap confidence signals include top-1/top-2 margin, blank probability, entropy, active-beam score gap, and beam agreement. A confident greedy path can still hide a deletion that beam repairs, so no signal is safe by intuition. Two-pass greedy-then-beam improves perceived partial latency only if greedy text is shown immediately. It does not reduce end-of-audio finalization latency unless beam is skipped or pruned. ## 7. Smallest Experiments That Settle the Design ### Experiment 0: Finish the quality rank Score greedy and beams 3, 4, and 8 on the sealed 4,182-row development role. Select a width before exporting several beam packages. The 500-clip rank is evidence for that slice, not authority to ship width 8. ### Experiment 1: Count the actual work Instrument pinned NeMo on probe50: - alignment iterations per clip; - joint calls; - predictor calls; - blank/non-blank transitions; - active-beam count; - duplicate recombinations; - forced blanks at `max_symbols=10`; - K² candidate selections; - final token count. This converts the 400 ms target into a per-iteration budget. ### Experiment 2: Hoist projections before building beam Export batch-one FP32 projected encoder, projected predictor, and projected joint. On a public ladder: - compare projected tensors to NeMo; - compare greedy tokens to the current unhoisted graph; - measure isolated p50/p95 and end-to-end greedy p95; - inspect CPU/GPU/ANE placement. **Kill:** no latency reduction, any unexplained top-1 change, or a larger package with no measured benefit. ### Experiment 3: Fixed batch-8 microbenchmark Use deterministic inputs and one process per configuration. Measure: - predictor B=1 versus B=8; - projected joint B=1 versus B=8; - full logits versus in-graph top-K outputs; - fresh versus reused feature providers; - default outputs versus verified output backings; - separate versus fused transition-plus-joint. Run on M1 8 GB, M1 16 GB, and iPhone 12 Pro. The M2 Ultra can debug, not certify. ### Experiment 4: State-machine parity Pass toy scorer fixtures through pinned Python and Swift. Compare after every iteration: - selected parent indices; - labels; - unnormalized scores; - normalized final scores; - time indices; - max-symbol counters; - active masks; - recombination groups; - predecessor pointers. **Kill:** first unexplained mismatch. ### Experiment 5: Component and probe50 parity Use frozen real encoder outputs. Compare NeMo FP32 projections to Core ML FP32, then compare a Python host loop and Swift host loop over the same Core ML components. This separates graph numerics from beam bookkeeping. **Kill:** first Swift-versus-Python Core ML hypothesis mismatch. NeMo-versus-Core ML differences are reported separately because existing FP32 conversion is already 496/500 exact, not 500/500. ### Experiment 6: Quality and latency escalation Run the exact candidate on parity500, then the full development role, with the untouched greedy baseline retained. Interleave latency arms on each device after warmup. Required receipt fields: ```json { "model_revision": "...", "package_digests": {}, "decoder_config": { "strategy": "malsd_batch", "beam_size": 8, "max_symbols": 10, "score_norm": true, "dtype": "fp32" }, "host": { "hardware": "...", "os_build": "...", "thermal_state": "..." }, "latency_ms": { "preprocess_p50": 0, "encoder_p50": 0, "predictor_p50": 0, "joint_p50": 0, "bookkeeping_p50": 0, "finalization_p50": 0, "finalization_p95": 0 }, "counts": { "alignment_iterations": 0, "predictor_calls": 0, "joint_calls": 0, "recombinations": 0, "bytes_copied": 0, "allocations": 0 }, "quality": { "raw_wer": 0, "greedy_minus_candidate_points": 0, "paired_interval": [0, 0], "exact_hypotheses": 0 } } ``` ### Experiment 7: Approximation only if exact misses Freeze the exact beam as teacher. Sweep one approximation at a time on probe50, then parity500. Reject any arm that loses a preregistered material fraction of beam's measured gain over greedy. Never tune and report on the same full-development role without recording that reuse. ## 8. Failure Modes - **Calling Core ML once per hypothesis.** Batch is the first requirement, not a later optimization. - **Recomputing both joint projections.** Encoder projection is once per utterance; predictor projection is once per changed prefix. - **Calling a static `[T,V]` tensor an RNNT lattice.** Joint scores depend on predictor prefix. - **Merging before pruning.** Pinned NeMo prunes K² to K, then recombines survivors. - **Implementing a predictor cache and calling it exact NeMo parity.** The pinned MALSD implementation explicitly does not cache predictor outputs. - **Treating beam one as `greedy_batch`.** NeMo says they can differ. - **Using `MLState` because it sounds zero-copy.** Branching and beam reordering are the actual state problem. - **Claiming output backings were used.** Check object identity and measure allocations. - **Returning all 1,033 logits without measuring transfer.** In-graph top-K plus blank is the smaller contract. - **Full-sorting the vocabulary for eight winners.** Use graph top-K or measured partial selection. - **Assuming FP16 is harmless.** the reference implementation measured 25 of 500 transcript changes from FP16 CPU components. - **Inferring Core ML beam cost from NeMo CPU wall time.** The shared PyTorch encoder dilutes the decoder ratio. - **Inferring an M1 or iPhone result from a 60-core M2 Ultra GPU.** Device ordering is not portable. - **Calling Argmax's TDT speed a beam benchmark.** Public sources do not disclose a beam. ## 9. Ranked Optimization Ladder | Rank | Work | Expected upside | Correctness risk | Result that funds the next rung | | --- | --- | --- | --- | --- | | 1 | Count pinned MALSD iterations and calls | Converts speculation into a budget | None | Measured call distribution | | 2 | Hoist encoder/predictor projections | Removes repeated dense layers from every decision | Low, numerical boundary only | Faster greedy and component parity | | 3 | Fixed-batch FP32 projected joint with in-graph top-K | Deletes per-hypothesis dispatch and bulk logits transfer | Low | Batch-8 isolated p95 within budget | | 4 | Exact native MALSD bookkeeping with explicit states | Reproduces quality target | Medium | Toy and probe50 parity | | 5 | Fused transition-plus-joint graph | Halves steady-state Core ML dispatches | Medium | Same hypotheses, lower p95 | | 6 | Native CPU predictor/joint | Removes Core ML recurrent dispatch | High | Trace proves Core ML dispatch-bound | | 7 | Adaptive beam or speculative greedy | Can remove whole expansions | Quality risk | Paired WER retains sealed beam gain | | 8 | TDT/multi-blank/distilled model | Largest long-horizon reduction in search work | New training campaign | Untouched-base bakeoff wins | The simple stopping rule is physical: do not write rung 4 until rung 3's batch-8 microbenchmark can plausibly fit inside the per-iteration budget; do not write a custom kernel until a trace proves dispatch, not math or the encoder, is the limiter. ## 10. Ingest Corrections to the Raw Report - Removed the unsupported 66 MB Argmax memory claim. - Replaced “Argmax speed is primarily TDT” with the narrower evidence: TDT supplies up to 2.82× in its paper; the remaining system-speed attribution is not established. - Removed speculation that Argmax uses adaptive beam, custom Metal, or C++. - Corrected `BatchedBeamHyps`: dense tensors, predecessor pointers, and hashes; no predictor prefix cache in pinned MALSD. - Corrected recombination ordering to post-prune, matching source. - Removed the false static-logit RNNT parity recipe and replaced it with a prefix-aware toy scorer. - Replaced deprecated neural-network flexible-shape mutation with current `ct.EnumeratedShapes` guidance, while recommending fixed packages first. - Corrected `ct.ComputeUnits.ALL` to singular `ct.ComputeUnit`. - Removed the unsupported claim that Core ML compute plans report `bnns`, `mps_graph`, or `ane` backend names; they report preferred/supported compute-device objects. - Downgraded `MLState` and output backings from “mandatory zero-copy” to measured options with documented constraints. - Removed the unverified MLX 2 GB and 2.3 ms dispatch constants. - Removed the proposed `vDSP_vsorti` top-K snippet; a full sort is not the right default and the API/performance claim was not established. ## Sources 1. Argmax, [“Nvidia Frontier Speech Models on Argmax SDK”](https://www.argmaxinc.com/blog/nvidia-frontier-speech-models-on-argmax-sdk), 2025-06-19. 2. Argmax, [“Managing Model Files”](https://app.argmaxinc.com/docs/guides/managing-models), accessed 2026-09-06. 3. Argmax, [“iPhone 17: On-device inference benchmarks”](https://www.argmaxinc.com/blog/iphone-17-on-device-inference-benchmarks), 2025-09-21. 4. Xu et al., [“Efficient Sequence Transduction by Jointly Predicting Tokens and Durations”](https://proceedings.mlr.press/v202/xu23g.html), ICML 2023. 5. Argmax, [“Argmax Pro SDK”](https://www.argmaxinc.com/blog/pro-sdk-ga), 2025-07-24. 6. NVIDIA, [“Turbocharge ASR Accuracy and Speed with NVIDIA NeMo Parakeet-TDT”](https://developer.nvidia.com/blog/turbocharge-asr-accuracy-and-speed-with-nvidia-nemo-parakeet-tdt/), 2024-04-18. 7. NVIDIA NeMo Speech, [`rnnt_malsd_batched_computer.py`](https://github.com/NVIDIA-NeMo/Speech/blob/95f92737cf/nemo/collections/asr/parts/submodules/rnnt_malsd_batched_computer.py), pinned revision `95f92737cf`. 8. NVIDIA NeMo Speech, [`batched_beam_decoding_utils.py`](https://github.com/NVIDIA-NeMo/Speech/blob/95f92737cf/nemo/collections/asr/parts/utils/batched_beam_decoding_utils.py), pinned revision `95f92737cf`. 9. NVIDIA NeMo Speech, [`rnnt_decoding.py`](https://github.com/NVIDIA-NeMo/Speech/blob/95f92737cf/nemo/collections/asr/parts/submodules/rnnt_decoding.py), pinned revision `95f92737cf`. 10. Apple coremltools, [“Flexible Input Shapes”](https://apple.github.io/coremltools/docs-guides/source/flexible-inputs.html), current documentation checked through Context7 on 2026-09-06. 11. Apple coremltools, [“Stateful Models”](https://apple.github.io/coremltools/docs-guides/source/stateful-models.html), current documentation checked through Context7 on 2026-09-06. 12. Apple, [`MLPredictionOptions.outputBackings`](https://developer.apple.com/documentation/coreml/mlpredictionoptions/outputbackings), accessed 2026-09-06. 13. Saon, Tüske, and Audhkhasi, [“Alignment-Length Synchronous Decoding for RNN Transducer”](https://doi.org/10.1109/ICASSP40776.2020.9053040), ICASSP 2020.