YZOS Product Specification

This document is the single source of truth for the YZOS product. Topic-specific documents expand on details; when they conflict, this document wins.

Platform: macOS 13 and later (first release) License: MIT, open source

1. What YZOS Is

YZOS is a local observation engine, knowledge crystallization layer, and AI data supply station that runs entirely on your Mac. It passively observes your work across apps, AI tools, chats, voice, and screen, distills these signals locally into people, projects, tasks, knowledge, and events, and supplies that context to upstream AI tools such as Claude Code, Codex, and Cursor through MCP, Skill, and CLI.

YZOS is not a chat product. It does not require you to work inside YZOS, and it does not perform tasks on your behalf.

2. What YZOS Promises

• Zero-overhead observation: keep using the apps you already use, YZOS observes passively in the background.
• Structure-first behavior recognition: read UI structure first, then apply OCR and visual understanding by region.
• Automatic entity profiles: people, projects, and tasks are discovered automatically and kept up to date in OKF.
• Local-first privacy: raw data stays on the device by default, external models only ever see redacted text.
• Portable source of truth: OKF Markdown is the source of truth, PostgreSQL is only an index.
• Supply for upstream AI: Claude Code and Codex query real local data through MCP.
• Reviewable and correctable: a Tauri UI lets you inspect observations, entity candidates, and maintenance logs.

3. What YZOS Does Not Do

• No built-in chatbot: conversation happens in your upstream AI tools, not in YZOS.
• No autonomous task executor: YZOS does not send messages, fill forms, run CI, or open pull requests on your behalf.
• No remote cloud service: sync (when used) relies on your own iCloud, WebDAV, SSH, S3, or Git.
• Not a screenshot-OCR tool: screenshots and video are an evidence layer, not the primary path for behavior recognition.
• Not general desktop RPA: observation and supply are the goal, automated control of your desktop is not.

4. Runtime Architecture

flowchart TD
  LA["LaunchAgent: com.yzos.autostart"] -->|"open -g at login"| APP["/Applications/YZOS.app"]
  APP --> DESK["yzos-desktop (com.yzos.desktop)"]
  DESK --> RT["yzos-runtime"]
  DESK --> PG[("PostgreSQL (embedded)")]
  DESK --> HTTP["HTTP 127.0.0.1:17432"]
  RT --> PRE
  subgraph PIPE["Data plane (single process)"]
    direction LR
    PRE["Preflight"] --> OBS["Observe"] --> CRY["Crystallize"] --> IDX["Index"] --> SUP["Supply"] --> MNT["Maintain"]
  end
  subgraph EP["HTTP endpoints"]
    S["/status"]
    P["/permissions"]
    A["/api/call"]
    M["/mcp"]
  end
  HTTP --> EP

• Single app: /Applications/YZOS.app
• Bundle id: com.yzos.desktop
• Runtime: src-tauri links yzos-runtime
• Data plane: Observe, Crystallize, Index, Supply, and Maintain all run in the same process
• Autostart: com.yzos.autostart only opens the app at login
• MCP: HTTP transport at /mcp
• CLI: yzos is an HTTP client against the same process

5. Core Pipeline

flowchart LR
  Preflight --> Observe --> Crystallize --> Index --> Supply --> Maintain

• Preflight: verifies installation, loads models, runs smoke tests for default capabilities (no per-capability enable switch, failures degrade gracefully without blocking Observe)
• Observe: passively collects app, window, UI structure, AI tool sessions, chat, clipboard, screen, and voice signals
• Crystallize: after a session ends, triages evidence, extracts entity signals, and writes knowledge
• Index: hybrid indexing across OKF, PostgreSQL full-text search, and pgvector
• Supply: MCP, Skill, and CLI return real observed data
• Maintain: scheduled rollups, entity reconciliation, archiving, and reindexing

Observe and Crystallize do not start until Preflight passes.

6. Observation Strategy

YZOS understands the desktop through a structure-first pipeline:

flowchart TD
  AW["Active app / window"] --> META["App metadata"]
  META --> AX["AX tree / DOM / app adapter"]
  AX --> REG["UI layout regions"]
  REG --> OCR["ROI OCR"]
  OCR --> VLM["VLM semantic labels"]
  VLM --> DIFF["Temporal diff"]
  DIFF --> EVT["Behavior event"]

• App / Window: bundle id, PID, title, bounds, URL, repo path, establishes the current work context
• UI Structure: AX nodes, DOM nodes, focused element, selection, role/value, read directly from accessible structure
• Layout Regions: editor, terminal, sidebar, toolbar, input, chat, canvas, modal, partitions the screen for OCR and behavior inference
• ROI OCR: bounding box, confidence, reading order, paragraph, fills in text that AX/DOM cannot expose
• VLM: screen semantics, visual region labels, fills in meaning for Figma, images, video, and canvas content
• Temporal Diff: state differences between consecutive UI snapshots, identifies edits, searches, sends, command runs, and file switches

OCR fills gaps, it is not the primary layer. VLM is a low-frequency semantic supplement, not the final arbiter of behavior.

7. Observation Sources

• Desktop activity: app switches, window focus, screen segments, UI state
• AI tools: Claude Code, Codex, Cursor, Continue, Aider, and other coding-agent sessions
• Clipboard: text history with redaction and search
• Chat apps: sender, message, thread, and entity signals
• Voice: voice activity detection, ASR, speaker count, voiceprint
• OKF: memory, knowledge, and entities

8. Behavior Events

Behavior events come from UI state diffs combined across multiple evidence sources.

• read: active window stable, scrolling, no edit diff
• edit: text change in an editor region, selection, file path
• run_command: terminal prompt plus command plus output diff
• send_message: chat input cleared, message list gains an entry
• copy: clipboard event plus focused region
• paste: clipboard event followed by content change in a target region
• select_file: file-tree selection or tab diff
• switch_context: app/window/URL/repo change
• design_adjust: canvas region change plus Figma/app adapter signal
• review: diff viewer, comment panel, PR URL, selection

9. Entities and Knowledge

Artifact	Source of truth	Index
Person	OKF entity file	PostgreSQL entities / sightings / links
Project	OKF entity file	PostgreSQL entities / repo path / AI sessions
Task	OKF entity file	PostgreSQL entities / status / related events
Knowledge	OKF Markdown	PostgreSQL chunks + full-text search + pgvector
Memory	OKF Markdown	PostgreSQL chunks + full-text search + pgvector
Activity	PostgreSQL event tables	PostgreSQL time-series indexes

If the PostgreSQL index is damaged, it is rebuilt by reconciling from OKF. Raw activity data is retained according to a configurable retention policy.

10. Supply Interfaces

• HTTP MCP /mcp: direct queries from Claude Code, Codex, and other MCP clients
• /api/call: local HTTP API
• CLI yzos: terminal and script queries
• Skill yzos-data: agent-triggered query rules
• Tauri UI: preview, settings, permissions, entity review

Supply interfaces never fabricate data. Queries for activity, entities, knowledge, or clipboard history are always answered from MCP, the HTTP API, or the CLI against real local data.

11. Technology Stack

Component	Choice
App host	Tauri 2 + React
Runtime	Rust yzos-runtime
Core domain	Rust yzos-core
Database	embedded PostgreSQL + pgvector
MCP	HTTP transport at /mcp
LLM	llama.cpp / llama-server sidecar
OCR	macOS Vision framework
UI structure	macOS Accessibility API, CGWindowList, browser/app adapters
Screen capture	ScreenCaptureKit + mp4 segments
Audio	sherpa-onnx
Embedding	multilingual-e5-small