YZOS System Architecture

Full specification: 00-product-spec.md.

1. Overview

flowchart TD
  App["/Applications/YZOS.app"] --> Desktop["yzos-desktop"]
  Desktop --> Runtime["yzos-runtime"]
  Desktop --> PG[("PostgreSQL embedded")]
  Desktop --> Sidecars["sidecars"]
  Desktop --> HTTP["HTTP 127.0.0.1:17432"]
  Runtime --> Preflight["Preflight"]
  Runtime --> Observe["Observe"]
  Runtime --> Crystallize["Crystallize"]
  Runtime --> Index["Index"]
  Runtime --> Supply["Supply"]
  Runtime --> Maintain["Maintain"]
  Sidecars --> LLM["yzos-llm"]
  Sidecars --> FFMPEG["ffmpeg"]
  HTTP --> Status["/status"]
  HTTP --> Perms["/permissions"]
  HTTP --> ApiCall["/api/call"]
  HTTP --> Mcp["/mcp"]

YZOS is a single-bundle, single-runtime architecture. yzos-desktop is the Tauri host, it links yzos-runtime, which carries out observation, crystallization, indexing, supply, and maintenance. When you quit YZOS.app, the runtime and MCP endpoint both go offline.

2. Processes and Components

Component	Form	Responsibility
yzos-desktop	Tauri app binary	app host, menu bar item, frontend window, permission prompts
yzos-runtime	Rust library	HTTP, MCP, observe loop, crystallize loop, maintenance
yzos-core	Rust library	domain logic, entities, OKF, Preflight, inference scheduling, platform capabilities
PostgreSQL	embedded child process	index for activity, entities, jobs
yzos-llm	sidecar	llama-server text and VLM requests
ffmpeg	sidecar	screen segment frame extraction, video processing
yzos-cli	binary	HTTP client
yzos-mcp	adapter binary	compatibility for stdio MCP clients, HTTP MCP is the primary path

3. Startup Lifecycle

flowchart TD
  Launch["App Launch"] --> Config["load config"]
  Config --> Logs["init logs"]
  Logs --> PG["start embedded PostgreSQL"]
  PG --> Migrate["run migrations"]
  Migrate --> HTTP["start HTTP"]
  HTTP --> Preflight["Preflight"]
  Preflight --> Ready["Ready"]
  Ready --> Observe["Observe"]

State	Meaning
booting	app process starting, PostgreSQL and HTTP initializing
preflight	verifying permissions, models, sidecars, and platform capabilities
ready	Preflight passed for default capabilities, observation can begin
observing	observe / crystallize / maintain running normally
degraded	Preflight failed for a default capability (failure downgrades that capability without blocking Observe)
suspended	asleep or a critical dependency is unavailable, Preflight reruns on resume

4. Observe Data Flow

flowchart TD
  Events["macOS platform events"] --> AppTracker["app/window tracker"]
  Events --> UIState["UI state engine"]
  Events --> Recorder["screen segment recorder"]
  Events --> Importer["AI tool importer"]
  Events --> Clipboard["clipboard watcher"]
  Events --> ChatAudio["chat/audio watchers"]
  AppTracker --> Builder["activity session builder"]
  UIState --> Builder
  Recorder --> Builder
  Importer --> Builder
  Clipboard --> Builder
  ChatAudio --> Builder

Observe only collects facts, it does not produce long-lived summaries directly. High-frequency data is written to PostgreSQL, raw files are written under paths.observations.

5. UI State Engine

Accurate behavior recognition depends on the UI State Engine.

flowchart TD
  Active["active app/window"] --> Metadata["metadata"]
  Metadata --> Structure["structure sources"]
  Structure --> Regions["layout regions"]
  Regions --> OCR["ROI OCR"]
  OCR --> VLM["VLM labels"]
  VLM --> Diff["temporal diff"]
  Diff --> Event["behavior event"]
  Metadata -.- Mdetail["bundle id / pid / title / bounds / url / repo / document path"]
  Structure -.- S1["AX tree"]
  Structure -.- S2["DOM / browser adapter"]
  Structure -.- S3["app-specific adapter"]
  Regions -.- Rdetail["editor / terminal / sidebar / toolbar / input / chat / canvas / modal"]

5.1 Structure Source Priority

Priority	Source	Example
1	App adapter	Claude Code jsonl, Cursor workspace, Figma API, browser CDP
2	AX tree	role, label, value, focused element, selection, bounds
3	DOM / accessibility tree	web app, browser tab, URL, selected node
4	ROI OCR	text regions AX/DOM cannot expose
5	VLM	canvas, images, video, design files, regions with no OCR text

OCR only fills gaps within a layout region, it does not run indiscriminately over the full screen.

5.2 Behavior Event Inference

Input	Output
ui_snapshot(t) + ui_snapshot(t+1)	state diff
state diff + clipboard event	copy / paste
state diff + terminal region	run_command
state diff + chat input	send_message
state diff + editor/file path	edit
app/window/url diff	switch_context

Every behavior event carries its evidence, YZOS never stores a bare LLM judgment without it.

6. Screen Capture

ScreenCaptureKit records continuous screen evidence, written by default as mp4 segments:

flowchart TD
  SCStream["SCStream"] --> Writer["segment writer"]
  Writer --> Segments["activity_screen_segments"]
  Segments --> Keyframe["keyframe extraction"]
  Keyframe --> OCRVLM["ROI OCR / VLM"]

Video is not the primary path for behavior recognition. It is used for audit, playback, OCR gap-filling, and VLM visual supplements.

7. Crystallize Data Flow

flowchart TD
  Ended["activity session ended"] --> Collect["collect evidence"]
  Collect --> Triage["LLM triage"]
  Triage --> Signal["entity signal extraction"]
  Signal --> OKF["OKF write/update"]
  OKF --> Index["PostgreSQL index"]
  Collect -.- E1["activity events"]
  Collect -.- E2["UI behavior events"]
  Collect -.- E3["AI tool sessions"]
  Collect -.- E4["clipboard signals"]
  Collect -.- E5["OCR text"]
  Collect -.- E6["screen semantics"]

Crystallize never sends unredacted sensitive text to an external service. Local models can read raw context, remote models only ever receive redacted text.

8. Supply Layer

flowchart LR
  Clients["Claude Code / Codex / Cursor"] --> MCP["MCP HTTP /mcp"]
  MCP --> Runtime["yzos-runtime"]
  Runtime --> Handlers["supply handlers"]
  Handlers --> Store[("PostgreSQL + OKF")]

Supply never talks to a model directly, and never fabricates observation data.

9. Module Boundaries

Module	May	Must Not
Observe	collect facts, write activity tables	generate long-lived conclusions
UI State Engine	structure UI, diff state, produce behavior events	act on the UI on the user’s behalf
OCR	output text boxes for a region of interest	serve as the primary full-screen understanding path
VLM	generate short visual semantic labels	produce the final triage JSON directly
Crystallize	distill knowledge, entities, tasks	fabricate facts without evidence
Supply	query PostgreSQL / OKF	talk to external services directly on the user’s behalf