External-Encoder, Decoder-Only Whisper Runtime on Apple Silicon

Collective Library edition. This is the complete technical report. Private filesystem paths, internal run identifiers, campaign-control notes, and repository navigation were removed. Technical claims, code, measurements, evidence labels, citations, corrections, and falsification criteria are preserved.

August 12, 2026

Executive Summary

Upstream whisper.cpp documents a hybrid configuration in which Core ML runs the encoder while the decoder remains in the GGML model. It does not document a decoder-only GGML container for this pinned loader. Structural encoder deletion is a local experimental derivative and must be qualified as such.

The opportunity is nevertheless concrete. The ordinary file initializer in the pinned runtime opens the model with std::ifstream. The loader declares the expected tensors, allocates backend buffers for them, then reads tensor bytes into those allocations. It is not an mmap-backed model path. Enabling Core ML changes encoder compute, but by itself does not delete the native encoder tensors from the model allocation.

The smallest safe decoder-only design therefore has four parts:

1. derive an authenticated artifact that preserves the original header, mel filters, vocabulary, and every decoder record byte-for-byte while omitting every native encoder record;
2. make the runtime declare and expect only those retained decoder tensors;
3. make state creation and encoder execution fail closed unless the exact external Core ML sidecar loads and returns the required tensor; and
4. prove exact tokens, text, EOT behavior, memory deletion, and repeated-call lifecycle behavior against the complete-model control.

Do not mix this lossless structural experiment with further quantization, cache shrinking, logits-API changes, or Core ML compression. Each changes a different causal mechanism.

1. Corrected Evidence Boundary

2. Architecture and Ownership

The target runtime has one acoustic path and one text path:

PCM
 |
 v
mel filters retained in GGML prefix
 |
 v
external Core ML encoder -----> [audio positions, audio state]
                                      |
                                      v
                           cross-attention K/V
                                      |
                                      v
                    retained GGML/Metal decoder
                                      |
                                      v
                             tokens and text

The Core ML package replaces native encoder compute. The decoder still needs encoder hyperparameters because they define the external output and cross-attention contract. Deleting native weights does not authorize changing those dimensions.

2.1. Keep the ABI-bearing prefix

A decoder-only derivative should preserve, byte-for-byte:

• file magic and model hyperparameters;
• n_audio_ctx, n_audio_state, n_audio_head, and n_audio_layer;
• text dimensions and quantization metadata;
• mel-filter dimensions and values;
• vocabulary bytes and token IDs; and
• every retained decoder record header and payload.

Preserving n_audio_layer may look odd after encoder deletion. In this legacy container it also participates in model-family identification and expected architecture. The runtime patch, not an invented header value, should make native encoder tensors unreachable.

2.2. Delete only native encoder records

Delete the records named by the pinned encoder inventory:

• positional embedding;
• two convolution weights and biases;
• final encoder layer normalization; and
• every tensor in every native encoder transformer block.

Retain all decoder self-attention and cross-attention tensors. Cross-attention is part of the decoder even though its keys and values are derived from the external encoder output.

2.3. One owner per resource

A memory-minimal serialized product starts with:

• one model context owning decoder weights;
• one state owning mutable caches and schedulers;
• one Core ML model handle associated with that state; and
• one request at a time through the context.

The public whisper.cpp contract does not make one context concurrently safe for arbitrary state use. Add concurrency only after proving it is a product requirement and measuring the incremental state and framework residency.

3. Deriving the Artifact

3.1. Prefer exact record filtering over reconversion

For a pinned, already-authenticated GGML source, the simplest inspected derivative is a strict parser and copier:

1. parse the prefix through vocabulary exact EOF boundaries;
2. parse every tensor record with checked dimensions, type, name, byte count, and offset arithmetic;
3. classify each name against frozen encoder and decoder inventories;
4. copy the prefix and decoder record bytes without numerical conversion;
5. reject unknown, duplicate, missing, malformed, or trailing records; and
6. reparse the emitted artifact independently before sealing it.

This design isolates structural deletion. Filtering a PyTorch state dictionary and rerunning conversion can also work, but introduces converter, source-model, and quantizer drift that the deletion experiment does not need.

3.2. Seal both what remains and what disappeared

The derivation receipt should bind:

• complete source-model SHA-256 and byte size;
• prefix SHA-256 and byte size;
• ordered source tensor inventory;
• ordered retained and removed inventories;
• each retained record's source offset, byte length, type, shape, and hash;
• output SHA-256 and exact EOF;
• total retained and removed payload and physical record bytes;
• derivation script and focused test hashes; and
• the compatible runtime source, patch, build options, and binary closure.

An output hash alone cannot prove that the right tensors were retained. An inventory alone cannot prove payload identity. Bind both.

3.3. Current 8e2d arithmetic is local experimental evidence

The current campaign's authenticated Q5 physics shell is 537,819,875 bytes. An exact parse identifies 487 native encoder records and 52 decoder records. Preserving its 611,057-byte prefix and all decoder records projects a stripped artifact of 84,778,619 bytes, removing 453,041,256 physical file bytes.

Those numbers are local binary-format evidence, not upstream guarantees, resident-memory measurements, quality evidence, or a shipping verdict. The campaign note owns their current receipts and status.

4. Making the Loader Decoder-Only

The pinned loader currently calculates tensor metadata, declares all native encoder and decoder tensors, allocates backend buffers for them, and then requires the number of loaded records to equal the declared tensor count. Merely truncating the file therefore fails with missing tensors.

4.1. Change the expected inventory, not error tolerance

In an explicit external-encoder-only build or mode:

• do not create native encoder tensor metadata;
• do not add native encoder names to model.tensors;
• resize and create only the decoder-layer structures needed by text and cross-attention graphs;
• retain the stock unknown-tensor, shape, size, and loaded-count checks for the smaller inventory, then add exact type, duplicate, missing-name, and EOF checks; and
• log that native encoder capability is absent.

Do not make the generic loader accept arbitrary missing tensors. That converts a precise derivative into a partially initialized model that can fail far from the load boundary.

4.2. Make native encoder execution unreachable

Encoder pointers may remain null only if every path that can build or execute the native encoder is structurally unavailable. The contract should require:

• Core ML compiled in;
• native fallback compiled out or rejected for this artifact class;
• Core ML sidecar load before a usable state is returned;
• an error return when prediction fails; and
• no attempt to build convolution or encoder graphs with missing tensors.

The model-only _no_state initializer may reasonably authenticate and load decoder weights before a Core ML state exists. If so, document that boundary: model-only load can succeed, while state creation and inference must fail when the sidecar is absent. A negative control must exercise the exact public entry point the product uses.

4.3. Validate the external encoder result

Do not treat a non-null MLMultiArray as sufficient. Check:

• prediction returned without error;
• output feature name is the expected one;
• element type is Float32 when the bridge copies Float32;
• rank and every dimension match the compiled contract;
• strides/layout match the copy path, or copy by checked strides; and
• element count exactly matches the destination allocation.

The external output shape and the preserved header's audio context must agree. A shorter or longer output is a contract failure, not a reason to silently truncate, pad stale memory, or invoke the absent native encoder.

4.4. The existing autorelease pool is not the missing feature

The checked Objective-C++ bridge already places the Core ML prediction call in an @autoreleasepool. Under ARC, locally owned objects also leave scope with the function. That does not prove the entire application is leak-free, but it invalidates the raw report's claim that this bridge lacks a pool and leaks a fixed 5.8 MB on every request.

If repeated inference grows memory, distinguish:

1. one-time Core ML specialization or allocator growth that plateaus;
2. reusable framework caching;
3. product-layer autoreleased objects outside the bridge; and
4. genuinely unbounded retained objects or buffers.

Use allocations and VM traces to identify the owner before adding another pool or changing bridge lifetime.

5. Mutable State Is a Separate Optimization

Structural deletion targets immutable native encoder weights. Self-K/V, cross-K/V, pad caches, schedulers, compute buffers, token vectors, and logits remain. Their bounds depend on the product contract and should be changed only after same-quantization parity passes.

5.1. Logits

The pinned state reserves capacity proportional to vocabulary size times text context. A greedy product may ultimately need only current-step logits, but a last-row-only change can alter public accessor assumptions, batching behavior, and downstream pointer lengths. Measure committed memory first, then introduce an explicit API contract and regression tests if the saving is material.

5.2. Text and audio caches

Bounding text length can reduce mutable state, but every caller must reject an oversized request before graph construction. Bounding audio positions is not valid merely because typical input is shorter: the external encoder output, header metadata, cross cache, and decoder graph must all share the same frozen maximum.

For the current full-span 8e2d experiment, preserve the 1,500-position external encoder contract. Short-context state is a different artifact and quality experiment.

5.3. Disable unused product features explicitly

Greedy, no-timestamp dictation can avoid beam candidates and optional DTW state when the public product contract does not consume them. Record each disabled feature in the receipt. Do not infer that a zero-valued request flag prevented an earlier initialization allocation; inspect the construction order.

6. Proving Memory Reduction on macOS

Artifact bytes, backend allocation logs, process footprint, and system pressure answer different questions. No single column proves the whole result.

Sample at named phases:

1. before model load;
2. after decoder weights load;
3. after state and Core ML sidecar initialization;
4. after the first prediction and specialization;
5. across warmed repeated predictions; and
6. after state/context release and an idle observation window.

Compare complete-model and stripped-model processes with the same backend, Core ML tree, input, request, thread count, warmup, host admission, and sample cadence. Do not overlap the old and new contexts during replacement.

6.1. ComputeUnits.all is eligibility, not placement proof

Core ML Tools documents ComputeUnits.ALL as allowing all available units: CPU, GPU, and Neural Engine. The same documentation exposes MLComputePlan.get_compute_device_usage_for_mlprogram_operation to inspect planned operation use. Therefore a successful load or prediction under ALL does not, by itself, prove ANE residency.

For a physical claim, bind the compiled-tree hash and capture an operation-level compute plan or Instruments trace on the exact target host. Report boundary CPU/GPU operations rather than collapsing a mixed graph into “runs on ANE.”

7. Correctness and Negative Controls

The structural-deletion control must use the same quantized decoder and same external Core ML encoder on both sides. Require:

• identical token IDs in order;
• identical EOT presence and stop reason;
• identical decoded-text bytes;
• identical token ceilings and request options;
• deterministic output across repetitions; and
• no latency regression beyond a preregistered tolerance.

Include full-span inputs near both 15 and 30 seconds. A load-only pass cannot prove that null native-encoder pointers stay unreachable during real prediction.

Named negative controls should reject:

• missing or renamed Core ML sidecar;
• Core ML prediction error or wrong output shape/type;
• one missing decoder record;
• one duplicate decoder record;
• an unknown encoder record reintroduced into the stripped artifact;
• wrong tensor type, dimensions, or payload length;
• truncated input and trailing bytes;
• native-fallback configuration; and
• more tokens or audio positions than the bounded state allows.

Q5-versus-FP16 WER and pathology qualification remains separate. Exact stripped-Q5 versus complete-Q5 parity proves deletion is lossless; it does not prove Q5 is good enough for production.

8. Failure Modes and Debugging Playbook

8.1. Loader reports an unknown or missing tensor

The derivative inventory and runtime declaration disagree. Print the first unexpected name, dimensions, type, record offset, and expected inventory hash. Do not suppress the error or permit partial loading.

8.2. Load succeeds, then native graph construction crashes

A fallback or graph-builder path still reaches a deleted encoder pointer. Reproduce with the Core ML sidecar missing, then make the external-encoder-only mode fail before any inference graph can be built.

8.3. Output repeats or hallucinates after the split

First compare the external encoder output shape and bytes, cross-cache construction, request reset, and exact decoder artifact against the complete control. Do not attribute the symptom to encoder deletion until those contracts match. Stale cross-K/V and audio-context mismatch can produce plausible text without a clean crash.

8.4. The file shrinks but peak memory does not

Check the model-load buffer log. If it still reports the complete weight size, the runtime still declares native tensors. If that buffer shrank, split the remaining peak among Core ML initialization, caches, compute arenas, temporary read buffers, dynamic-library load, and overlapping contexts.

8.5. Memory grows across repeated predictions

Run enough iterations to distinguish lazy plateau from linear growth. Mark pool boundaries, model/state lifetime, and request completion in the trace. Because the bridge already has a prediction-local autorelease pool, inspect actual retained owners before diagnosing an autorelease leak.

8.6. Core ML loads, but ANE evidence is absent

This is expected under a permissive compute-unit request. Capture the exact compiled model's compute plan or Instruments trace. Treat load logs as Core ML utilization evidence only, not device-placement evidence.

9. Anti-Patterns

10. Decision Framework

The clean promotion order is:

1. complete-model same-quantization control;
2. stripped artifact and fail-closed loader;
3. exact inference parity and malformed-input controls;
4. matched local physical-memory proof;
5. exact target-Mac latency, placement, memory, and swap proof;
6. mutable-state experiments one mechanism at a time; and
7. independent decoder quantization and product-quality qualification.

Works Cited

1. whisper.cpp pinned source tree — source pin used for the legacy loader, tensor, state, and Core ML boundary.
2. whisper.cpp Core ML support — upstream hybrid encoder setup and runtime load-log contract.
3. Core ML Tools compute-unit options — ALL, CPU, GPU, and Neural Engine eligibility choices.
4. Core ML Tools model utilities — MLComputePlan operation-device and estimated-cost inspection.
5. Low-memory Whisper Turbo inference — locally corrected Darwin accounting, pinned allocation topology, and memory receipt design.
6. Physical Core ML plus whisper.cpp verification — local fail-closed and placement-evidence contract.