Inside MetaScope 1.5's Vision Stack: The Models and How We Configured Them

This is the technical companion to MetaScope 1.5: The Intelligence Release.

That article explains what the new Intelligence features do. This one explains how they work: the models MetaScope runs on your Mac, why we selected them, and how we configured them for a metadata application rather than a chat interface.

Three principles shaped the system.

1. Everything runs on-device. No recognition path makes a network request. There is no cloud fallback or separate cloud-powered tier.
2. The output becomes metadata. A recognition result can become a durable keyword or description in a standard field. A wrong label is not merely a poor search result. It can become a permanent false claim written into the file.
3. Analysis must never destabilise the application. It runs away from the main thread and priority-escalation path, under thermal backpressure, with no more than two images processed concurrently.

Those constraints influenced every model, calibration, storage, and integration decision that follows.

The system at a glance

MetaScope 1.5 runs two independent on-device recognisers over a single image decode. Their outputs are then combined using one calibrated confidence scale.

                 one CGImage decode (RAW → embedded preview, capped)
                          │
        ┌─────────────────┴─────────────────┐
        │                                   │
  Apple Vision                        SigLIP2 (Core ML)
  8 requests, one perform()           1 raw MLModel.prediction()
  scene · animal · face · human       1024-d embedding + 1262 label scores
  OCR · saliency · horizon · aesth.   ("Enhanced Recognition", Pro, opt-in)
        │                                   │
        └─────────────────┬─────────────────┘
                          │
            calibrated, read-time merge layer
        (one shared [0,1] floor · agreement boost · live sensitivity)
                          │
             persisted locally (SQLite, schema v16)
             keywords · descriptions · embeddings

The image is decoded once. Both recognisers work from the same pixels, but neither depends on the other succeeding.

Their results are stored in a local SQLite index. A read-time layer converts those results into keyword suggestions and description inputs, applying the current sensitivity setting without requiring the image to be analysed again.

Layer 1: Apple Vision, calibrated by class

The base recogniser uses Apple’s Vision framework.

MetaScope runs eight requests against the decoded image in a single VNImageRequestHandler.perform call.

Request	Output
VNRecognizeTextRequest	Visible text through OCR
VNClassifyImageRequest	Approximately 1,300 scene and subject labels
VNRecognizeAnimalsRequest	Recognised pets
VNDetectFaceRectanglesRequest	Detected faces
VNDetectHumanRectanglesRequest	People without a visible face
VNGenerateAttentionBasedSaliencyImageRequest	The subject’s position in the frame
VNDetectHorizonRequest	Horizon angle
VNCalculateImageAestheticsScoresRequest	Aesthetic score and utility-image flag

The first five contribute to keyword suggestions. Saliency, horizon, and aesthetics contribute composition and tonal information to descriptions.

VNDetectHumanRectanglesRequest is particularly important for photographs where a person is facing away from the camera. Face detection alone cannot reliably identify that person as a subject.

MetaScope accepts a detected human at minHumanConfidence = 0.5, subject to a minimum width fraction. This prevents a small, distant bystander from automatically becoming the main subject.

Why one confidence threshold does not work

A metadata application needs to be more conservative than a recognition demonstration.

The raw confidence values produced by VNClassifyImageRequest are not directly comparable across approximately 1,300 classes.

A score of 0.5 for “beach” does not represent the same strength of evidence as 0.5 for “hound.” Individual classes have different precision and recall characteristics.

Applying one flat threshold would admit too many results for some classes while suppressing useful results for others.

MetaScope therefore asks Vision for class-specific calibration using hasMinimumPrecision(_:forRecall:).

Each result is evaluated against a calibrated precision tier at a fixed recall operating point:

precisionTierRecall = 0.7

The user-facing Recognition Sensitivity setting maps to these calibrated tiers:

• More results
• Balanced
• Higher accuracy

This allows each class to be admitted according to its own calibration rather than a global confidence number.

The previous continuous percentage slider was removed because adjacent positions often produced identical behaviour under the calibrated tier system. Keeping it would have implied a level of precision the control did not actually provide.

For classes where Apple does not publish a calibration curve, MetaScope falls back to a direct confidence check rather than dropping the label entirely.

A curated vocabulary, not raw machine labels

Vision produces machine-oriented identifiers such as outdoor, structure, material, and people_portrait.

Some are useful. Many are too broad to become meaningful photo metadata.

MetaScope maps these identifiers through a curated 1,262-keyword vocabulary stored in vision-label-vocabulary.json.

The mapping converts relevant identifiers into terms a photographer might actually apply. For example, beach becomes “Beach.”

The mapping is also a filter. Generic identifiers are deliberately omitted, so a recognised Chihuahua can surface “Chihuahua,” while a recognised structure produces no keyword.

This same 1,262-keyword vocabulary is also used by the second recognition layer.

Sensitivity is applied at read time

One less visible design decision has an important practical effect: MetaScope applies sensitivity when results are read, not only when they are extracted.

The raw pre-threshold signal is persisted in the labelsJSON column introduced with schema version 14. It includes every stored label with its confidence and provenance.

A shared function, VisionLabelSelection.selectIdentifiers, filters that raw signal using the current confidence floor whenever results are read.

Changing Recognition Sensitivity therefore updates suggestions for previously analysed photos immediately.

There is no need to decode or analyse those images again.

Re-analysis is only required when MetaScope introduces a genuinely new recogniser, not when the user changes a threshold.

Layer 2: SigLIP2, configured for MetaScope

Apple’s classifiers are limited by their fixed taxonomy.

Many terms in MetaScope’s curated 1,262-keyword vocabulary are structurally unreachable because Apple’s models never emit a corresponding identifier, regardless of how MetaScope maps the output.

Enhanced Recognition, available in MetaScope Pro and disabled by default, adds a second on-device model to improve coverage within that vocabulary.

Why SigLIP2 replaced MobileCLIP

The initial candidate was MobileCLIP-S. It is compact, fast, and an obvious model to consider for local execution.

Licensing made it unsuitable.

Apple’s ML Research Model terms are non-commercial and extend to derivatives. The DataCompDR training data is licensed under CC-BY-NC-ND. Neither condition is compatible with distributing the model inside a paid application.

MetaScope instead uses ViT-L-16-SigLIP2-256.

SigLIP2 is available under Apache 2.0, which provides a clean path for redistribution, and it performs better than MobileCLIP and OpenAI CLIP on zero-shot benchmarks.

The trade-off is size.

The bundled model is approximately 243 MB, compared with roughly 20 to 80 MB for MobileCLIP variants. We accepted the larger download in favour of stronger recognition and an appropriate commercial licence.

MetaScope follows the requirements of Apache 2.0 §4. The bundled weights are identified as a modified derivative, and the application includes the relevant LICENSE and NOTICE files.

A closed-set scorer, not open-vocabulary recognition

It is important to be precise about what the bundled model can do.

MetaScope’s SigLIP2 implementation is not an open-vocabulary recognition system.

We created a frozen classification head over exactly the same 1,262 curated keywords used elsewhere in the application.

The model exposes two named outputs:

• embedding: a 1,024-dimensional, L2-normalised image vector stored for a future semantic-search feature
• labelScores: 1,262 sigmoid scores, one for each curated keyword, aligned by index to a pinned label-order.json

labelScores can only score those predefined terms.

It does not accept arbitrary text queries. Its purpose is to improve reachability inside MetaScope’s existing keyword taxonomy using a second, complementary recognition model.

Free-text querying is a separate future capability that can be built on the persisted image embedding.

How the classification head was baked

The classification head is not part of SigLIP2’s stock output.

It was produced offline from MetaScope’s vocabulary.

1. Load SigLIP2’s webli-pretrained checkpoint through open_clip.
2. For each of the 1,262 curated keywords, run eight prompt templates through the text tower. Examples include "a photo of a {}.", "a close-up photo of a {}.", and "a picture of a {}.".
3. L2-normalise each text embedding.
4. Average the eight embeddings into a single weight row, W[keyword].
5. Bake SigLIP2’s trained sigmoid parameters alongside the weight matrix.

The trained parameters are:

logit_scale = 108.05
logit_bias  = -16.35

The final scoring operation runs inside the compiled Core ML graph:

labelScores = σ(logit_scale · (embedding · Wᵀ) + logit_bias)

Prompt ensembling produces a more stable text anchor for each keyword than relying on one phrasing.

The Swift application does not implement any additional scoring logic. It reads the output array produced by the model.

In practical terms, MetaScope’s entire keyword taxonomy is compiled into the model’s output head.

Bundled, compiled, and never downloaded at runtime

The original fp16 model is 633 MB.

MetaScope applies 6-bit k-means palettisation at per_grouped_channel granularity with group_size=8.

A naïve per-tensor 6-bit conversion produced a more significant fidelity loss, measuring 0.968 cosine similarity. Per-grouped-channel quantisation with a group size of eight recovered fidelity at the same bit depth.

The resulting bundled model is 243.5 MB.

It is stored through git LFS, including the contents of the .mlpackage directory bundle. Xcode compiles it into .mlmodelc format during the application build.

At runtime, MetaScope loads the compiled artefact directly from the signed application bundle.

There is:

• No runtime compilation
• No first-launch model compilation delay
• No relocation into Application Support
• No model download
• No network dependency

The model ships with the application.

Why SigLIP2 runs outside Vision’s perform

MetaScope does not wrap SigLIP2 in a VNCoreMLRequest.

Two implementation rounds explored versions of that design. Testing against the macOS 26.5 SDK exposed an unacceptable failure mode.

When a Core ML request throws inside a shared Vision perform call, the other eight requests can be left with nil results that cannot be recovered.

A failure in the secondary recogniser could therefore destroy scene, animal, face, human, OCR, saliency, horizon, and aesthetics results from the primary stack.

That was not acceptable.

SigLIP2 instead runs through a separate raw MLModel.prediction(from:) call using the same previously decoded CGImage.

This gives the integration three useful properties.

One image decode

SigLIP2 reads the pixels already decoded for Vision. It does not trigger an additional source-image decode.

Structural isolation

The model does not participate in Vision’s shared perform call.

A SigLIP2 failure returns nil, while the eight Apple Vision results remain identical to a run where Enhanced Recognition was disabled.

Controlled preprocessing

Testing showed that Vision’s internal resize path, which behaves like a bilinear-class kernel, achieved approximately 0.989 cosine parity with the model’s reference preprocessing.

That difference disproportionately affected lower-ranked results.

MetaScope instead resizes the image to the model’s 256 × 256 input using CGContext with interpolationQuality = .high.

The bicubic resize path achieved 0.9999 cosine parity.

For this recognition path, MetaScope controls preprocessing directly rather than relying on Vision’s resize kernel.

Single-flight model loading

The SigLIP2 service is implemented as an actor with a single-flight loading path.

The first analysis task that needs the model creates the loading Task. Any concurrent callers await the same result.

This prevents the 243 MB model from being loaded more than once.

The service is deliberately not isolated to @MainActor.

Image extraction already runs away from the main actor, and introducing a main-thread hop for every image would add avoidable overhead to the processing path.

Bringing both models onto one confidence scale

The baked SigLIP2 head introduced a calibration problem.

Its ranking was useful, but its absolute sigmoid probabilities were much smaller than the scores produced by Apple’s classifier.

A correct top-ranked result might score anywhere from approximately 0.003 to 0.4.

With 1,262 sigmoid outputs, the prompt-ensembled and renormalised text anchors dilute the cosine alignment relative to the distribution used to train the original logit_bias.

A conventional threshold such as 0.5 would therefore suppress almost every valid match.

The ranking remained correct. The operating point had shifted.

MetaScope corrects this by recentering the sigmoid decision boundary in logit space:

calibrated = σ(logit(raw) + offset)
offset = clipCalibrationLogitOffset = 5.5

The offset was selected so that a typical top-ranked match lands near 0.5.

This places SigLIP2’s output onto the same approximate [0,1] confidence scale as Apple’s classifier.

That makes it possible to use:

• One read-time confidence floor
• One sensitivity control
• One agreement boost
• One comparable merge path

MetaScope applies a conservative write-time floor to the calibrated score:

clipCalibratedWriteFloor = 0.20

This removes obvious non-matches while preserving enough lower-confidence results for the user to reveal more suggestions by changing sensitivity.

The application retains no more than:

clipMaxCandidates = 5

calibrated SigLIP2 candidates per image.

The design is intentionally precision-first.

These constants were originally treated as provisional. A subsequent labelled-set benchmark using Open Images V7 reaffirmed them without changes.

At default sensitivity, the benchmark measured precision at 44.7 and recall at 49.3, with the expected precision and recall trade-off appearing across the available sensitivity floors.

Those numbers describe a calibration operating point. They are not a claim that MetaScope’s overall recognition is “X% accurate,” and we do not present them that way.

Fusing Apple Vision and SigLIP2 results

Once both recognisers operate on a comparable scale, the read-time merge has two jobs.

Improve precision through agreement

When Apple Vision and SigLIP2 identify the same keyword, MetaScope adds an agreement boost:

clipAgreementBonus = 0.05

The combined suggestion remains marked Vision.

Improve recall through complementary results

When a SigLIP2-only result clears the current floor without an Apple match, it appears as a new suggestion marked Enhanced.

The separate badge makes the result’s origin visible to the user.

Both paths evaluate the current read-time confidence floor, so changing Recognition Sensitivity updates existing results without another analysis pass.

Local, typed, and prunable storage

All recognition results are stored in the local SQLite metadata index alongside MetaScope’s search data.

vision_extracted.labelsJSON

Introduced with schema version 14, this column stores the raw pre-threshold signal for both recognition systems.

Each entry includes:

• Label identifier
• Confidence
• Provenance, such as scene, animal, or clip
• Face count

The read-time filtering layer uses this data whenever it produces suggestions.

vision_clip_embeddings

Introduced with schema version 16, this table stores the 1,024-dimensional image embedding as a fixed 2,048-byte Float16 BLOB.

The storage cost is approximately 21 to 22 MB for every 10,000 images. The table is uncapped but can be pruned.

Its name deliberately identifies the embedding space.

A future perceptual-duplicate feature would use a different feature representation and should not be able to store that data in the same table by accident.

Float16 storage across both Mac architectures

One implementation detail illustrates the difference between a model that works locally and one that can ship in a production Mac application.

MetaScope converts the embedding to half precision using Accelerate’s vImageConvert_PlanarFtoPlanar16F, rather than Swift’s Float16 type.

Float16 is marked unavailable on the x86_64 slice. Referencing it directly causes xcodebuild archive to fail because the archive process builds a universal arm64 and x86_64 binary.

The vImage converter performs the necessary bit-pattern conversion on both architectures.

Its round-to-nearest-even output is byte-identical to the Float16 representation produced on arm64. An embedding BLOB written by either architecture is therefore interchangeable.

It is a small implementation decision, but it is necessary for the model to ship in a universal Mac application.

Descriptions begin with a rule engine

MetaScope’s description system is deliberately not an open-ended captioning model.

It begins with a deterministic composer that assembles a sentence in a fixed order:

subject → setting → mood → composition → visible text → factual metadata

The factual metadata can include location, date, and camera information already present in the file.

Every clause must trace back to an available signal. When that signal is missing, the clause is omitted.

The deterministic composer cannot invent a person, place, object, or relationship because it has no generative step.

Optional Foundation Models rewriting

On macOS 26, MetaScope can optionally use Apple’s FoundationModels framework to rewrite the deterministic description into more fluent prose.

The surrounding engineering exists to preserve the same grounding and privacy guarantees.

On-device execution

The implementation references SystemLanguageModel.default and no other model.

A build-time source-grep test prevents the file from referencing any cloud-routed Foundation Models symbol.

There is no network path.

Grounded instructions

The rewrite receives constraints through the instructions channel, which has higher priority than the prompt.

It is instructed to:

• Rewrite the caption as two to four sentences
• Use only facts already present in the source caption
• Add, infer, or invent nothing
• Avoid describing spatial relationships between separate subjects

The final constraint prevents the model from inventing geometry between multiple recognised subjects when the deterministic composer did not provide that relationship.

A fact sheet may be supplied for vocabulary guidance, but it does not authorise the model to introduce new facts.

The rewrite can improve phrasing. It cannot expand the factual content.

It also cannot restore information the deterministic composer deliberately suppressed, such as replacing “several people” with an exact headcount.

Version isolation

Every Foundation Models symbol is contained inside a type marked:

@available(macOS 26, *)

The framework is weak-linked, so launching MetaScope on macOS 15 does not touch its symbol table.

Below macOS 26, the Draft button returns the deterministic description byte-for-byte.

Fail-soft behaviour

Any failure returns nil.

That includes an unavailable model, an oversized request, or a session-level error.

The caller then retains the deterministic description.

The language model can improve the phrasing or get out of the way. It cannot prevent a grounded draft from being produced.

The architecture in one place

The implementation details above reduce to a consistent product posture.

• On-device throughout. Two recognisers and an optional language-model rewrite operate without network calls.
• One decode, away from the hot path. Both recognisers share one source decode. Analysis runs away from the main thread and priority-escalation path, under thermal backpressure, with no more than two images processed concurrently.
• Resumable processing. Analysis can pause, survive an application quit, and continue without repeating completed work.
• Structural isolation. A failure in the bundled model cannot destroy Apple Vision results.
• Comparable calibration. Logit-space recentering places the custom sigmoid head and Apple’s classifier on one confidence scale.
• Live sensitivity. Raw signals are retained so threshold changes can be applied at read time without another analysis pass.
• Reproducible model construction. The bake pipeline, including convert_siglip.py and related tools, is committed.
• Versioned model integrity. A pinned modelVersion and content fingerprint connect the compiled weights, label order, and curated vocabulary. An edit to one component that is not reflected in the others can be detected.
• Opt-in access. Enhanced Recognition is a MetaScope Pro feature and is disabled by default. Foundation Models rewriting is optional and limited to macOS 26 or later.

This is not a wrapper around a remote recognition API.

It is a model selected for its licence and quality, given a classification head built from MetaScope’s own keyword taxonomy, quantised for distribution inside the application, calibrated against Apple’s recognition stack, and integrated into a read layer that produces portable, standard metadata.

All of it runs privately on the user’s Mac.

That is what “configured specifically for MetaScope” means.

Enhanced Recognition and all Image Analysis features require MetaScope Pro. Enhanced Recognition is disabled by default and includes the bundled SigLIP2 model, resulting in a larger initial app download. The optional Foundation Models prose rewrite requires a Mac with Apple Intelligence running macOS 26 or later and also operates entirely on-device. Calibration figures in this article describe operating points from an internal labelled-set benchmark, not accuracy guarantees.