Skip to content
YZOS

Local Inference Stack and Dependency Versions

Full spec: 00-product-spec.md. Modality assignments: 15-modality-models.md.

1. Decision Summary

Section titled “1. Decision Summary”
Layer macOS arm64 (default) macOS Intel Windows (future)
LLM llama-server, Metal, GGUF Q4 llama-server, AVX2, GGUF Q4 llama-server, Vulkan, GGUF Q4
VLM llama-server vision profile same same
Embedding fastembed + ort CoreML EP fastembed, CPU fastembed + ort DirectML EP
OCR Vision.framework Vision.framework WinRT OCR
PII rules + ort NER same rules + ort NER
Audio sherpa-onnx sherpa-onnx sherpa-onnx
PG embedded PostgreSQL + pgvector same same

2. Inference Principles

Section titled “2. Inference Principles”
Principle Description
Preflight first Default capabilities (voice, vlm, screen, ocr, clipboard, llm, embed) must pass install verification, loading, and a smoke test; on failure the capability is SKIPped and the runtime can still enter Observe (degraded mode)
Scheduler-unified dispatch LLM, VLM, OCR, Embed, and Audio all go through InferenceScheduler
Structure source first UI structure, app adapters, AX, and DOM take priority over OCR
OCR fills gaps OCR only covers gaps in layout regions
VLM is low frequency VLM only supplements screen_semantics and visual region labels
Text-only triage The final triage JSON is produced by the text LLM plus grammar
Local-first Raw observation data does not leave the machine by default

3. Structure-First Inference Chain

Section titled “3. Structure-First Inference Chain”
flowchart TD
  AW["Active app / window"] --> AX["App adapter / AX / DOM"]
  AX --> SNAP["ui_snapshots / ui_regions"]
  SNAP --> OCR["ROI OCR text boxes"]
  OCR --> VLM["VLM semantic labels"]
  VLM --> EV["Evidence bundle"]
  EV --> TRI["Text-only LLM triage"]
  TRI --> OUT["OKF / entities / tasks"]

4. Preflight

Section titled “4. Preflight”
Check Acceptance criteria
paths home, models, database, and observations directories are readable and writable
database embedded PG starts, migrations complete
permissions /permissions returns app_path /Applications/YZOS.app
OCR Vision.framework returns a result (or an explicit empty result) for a test frame
LLM llama-server starts, grammar JSON smoke test passes
VLM default capability; returns a short caption for a test image; SKIP if mmproj weights are missing
embedding dummy text returns a vector of the active dimension
PII rules and NER smoke test pass
audio default capability; sherpa pipeline smoke test passes; SKIP if the model is missing

Preflight critical checks (paths, database, permissions, llm) failing puts the runtime into degraded, and no new Observe events may be written. Non-critical checks (VLM, audio) SKIP without stopping the runtime; observing continues with that modality degraded.

5. InferenceScheduler

Section titled “5. InferenceScheduler”
Lane Jobs Concurrency
llm triage, summary, entity body, merge text single slot
vlm keyframe caption, screen_semantics mutually exclusive with LLM, or governed by profile
ocr ROI OCR, keyframe OCR small concurrency
embed OKF chunks, entities, search index small concurrency
audio VAD, ASR, speaker pipeline stream queue

All modules submit an InferenceJob; none may call the sidecar or model runtime directly.

6. OCR and VLM Scheduling

Section titled “6. OCR and VLM Scheduling”

OCR triggers:

  • AX, DOM, or app adapter cannot read the text.
  • The layout region type is canvas, image, browser_content, or unknown.
  • The screen segment needs OCR for audit purposes.
  • The user opens Activity detail and needs visible text evidence filled in.

VLM triggers:

  • OCR character count is below threshold and session duration exceeds threshold.
  • The active app belongs to a visually heavy app category.
  • The layout region contains canvas, image, or design_surface.
  • Triage evidence is missing scene semantics.

VLM output can only enter text-only triage as evidence; it cannot write directly to the final JSON.

7. Acceptance and Verification

Section titled “7. Acceptance and Verification”
Check Command or evidence
workspace tests mise exec -- cargo test --workspace -- --test-threads=1
frontend build pnpm run build
supply ./scripts/verify-supply.sh
goal ./scripts/verify-goal-e2e.sh
status curl -s http://127.0.0.1:17432/status
permissions curl -s http://127.0.0.1:17432/permissions
OCR activity_screen_segments.ocr_status=done and text boxes are traceable
UI state ui_snapshots, ui_regions, ui_behavior_events have samples