Modality Processing and Model Division of Labor

Full product spec: 00-product-spec.md. Inference stack: 13-inference-and-dependencies.md.

1. General Principles

YZOS observes a multimodal world, but it does not route everything through one large model. The processing order is: structured sources first, then region-scoped OCR, then visual semantics, then text-only triage.

Principle	Description
Structure first	App adapters, AX, and DOM take priority over OCR
Specialized tool per task	OCR uses Vision.framework, speech uses sherpa-onnx, text crystallization uses a text-only LLM
ROI OCR	OCR runs on layout regions (ROI), not an undifferentiated full-screen scan
VLM used sparingly	The VLM handles visually heavy, OCR-empty, or canvas scenarios only
Evidence consolidation	Every modality is converted into evidence before triage runs
Preflight readiness	A modality runs once its Preflight check passes; a failed non-critical check yields SKIP but the runtime keeps observing

2. Modality Overview

flowchart TD
  Observe[Observe]
  subgraph Structured[structured sources]
    AILogs[AI tool logs]
    AXDOM[AX / DOM / app adapter]
  end
  subgraph Visual[visual sources]
    Seg[ScreenCaptureKit segments]
    ROI[ROI OCR boxes]
    VLM[VLM labels]
  end
  subgraph Audio[audio sources]
    VAD[VAD / ASR / speaker]
  end
  Observe --> Structured
  Observe --> Visual
  Observe --> Audio
  Structured --> Evidence[evidence bundle]
  Visual --> Evidence
  Audio --> Evidence
  Evidence --> Triage[text-only triage]
  Triage --> Out[entities / OKF / tasks]

3. Structured Signals

Source	Content	Role
AI tool logs	prompt, tool use, cwd, source file	Primary evidence for projects and tasks
AX tree	role, value, label, focus, selection, bounds	Primary evidence for UI layout
DOM / browser	URL, title, selected node, accessibility tree	Primary evidence for web apps
App adapter	Specialized data from Figma, Cursor, Terminal, etc.	High-confidence structured source
Clipboard	copied text, timestamp, target context	Evidence for copy/paste behavior

4. Image and Screen

Subtask	Technique	Output
Screen evidence	ScreenCaptureKit	mp4 segment
Keyframe extraction	ffmpeg	png keyframe
OCR	Vision.framework	text boxes
OCR postprocess	bbox ordering, confidence filtering	paragraphs
Layout labeling	heuristics + AX bounds	ui_regions
Visual semantics	VLM	screen_semantics

Processing order:

flowchart LR
  Regions[ui_regions] --> Crop[crop ROI]
  Crop --> OCR[Vision OCR]
  OCR --> Boxes[text boxes]
  Boxes --> Paras[paragraphs]
  Paras --> Merge[merge evidence]

5. VLM

The VLM is used for:

• Figma, Preview, design tools, and image viewers.
• Sessions with very little OCR text but clear visual work in progress.
• canvas, image, video, and design_surface regions.
• Cases where screen_semantics is needed to support triage.

The VLM does not write tasks or entities directly. Final crystallization is always done by the text-only triage stage.

6. Video

Type	Source	Processing
Screen segment	background screen evidence	extract keyframes -> ROI OCR / VLM
User video file	meeting recording or imported file	ffmpeg frame extraction + audio track separation

There is no default full-video-length pass through a Video-LLM.

7. Audio

Subtask	Technique	Output
VAD	sherpa Silero	speech segments
ASR	sherpa SenseVoice	transcript
Diarization	sherpa pipeline	speaker timeline
Voiceprint	3D-Speaker	person match

Audio evidence is linked to the activity session, the person entity, and the meeting event.

8. Text and Knowledge

Subtask	Model	Output
Triage	Qwen2.5-7B + grammar	JSON decision
Summary	Qwen2.5-7B	OKF body
Entity update	Qwen2.5-7B	summary / links
Embedding	multilingual-e5-small or the active embedder	vector
PII	rules + NER	redacted text

Triage takes the evidence bundle as input, never a raw full-screen screenshot.

9. Phased Delivery

Phase	Modality capability
P0	activity, AI logs, clipboard, OKF, MCP
P1	screen segments, OCR, VLM smoke test, screen_semantics
P2	UI snapshots, layout regions, ROI OCR, behavior events
P3	app adapters, audio, chat, multi-device

10. What We Don’t Do (and Why)

We don’t	Reason
Use full-screen OCR as the primary path for behavior recognition	Too noisy to reliably infer behavior
Run the VLM on every frame	Power and latency cost is unacceptable
Let the VLM produce final JSON directly	It lacks grammar constraints and evidence grounding
Run a Video-LLM over full recordings	Background observation cost is too high
Make a Python ML stack the primary path	Packaging and cross-platform cost is too high