# Architecture overview This is the contributor's map of Sotto: how the pieces fit, what each one owns, and where to read the exact contract when you need interface detail. It covers the four load-bearing subsystems (the transport, the orchestrator, the models, and the speech pipeline) and how a single turn moves through them. This page is the orientation, not the specification. When you implement, the `specs/*.md` contracts are the source of truth for wire formats, schemas, and behavior; where this overview and a spec disagree, the spec wins. The [design page](index.html) explains the reasoning behind each choice; where the design page and a spec disagree, the design page wins and the spec is corrected. ## Two trusted devices, one link Sotto is a fully private voice assistant and chatbot with no third-party AI provider in the path. It is two trusted devices joined by one encrypted link: - **The phone (`app/`)** is the always-available voice client and the hands. It is native Kotlin and Jetpack Compose. It claims the Android assistant role, captures audio, plays audio, and runs tool calls as real device actions through native APIs and the Shizuku bridge. It is a thin, smart client: it holds no models on the hot path and defers reasoning to the Mac. - **The Mac (`server/` + `mac-app/`)** is a stateful inference server on the home LAN. It runs the models, owns the canonical chat, project, and memory store, and does all the reasoning. The `mac-app/` SwiftUI menu-bar app supervises the native services. - **The link** is [iroh](https://www.iroh.computer/): a direct, end-to-end encrypted peer-to-peer QUIC connection the phone opens by dialing the Mac's public key. It is the only network hop between the two devices, and no third party sits in the middle of it. The voice assistant and the text chatbot are the same conversation engine with two front ends. A voice turn is a chat turn whose input arrived as audio and whose reply is spoken, so everything below the front end is shared. ``` Phone (GrapheneOS) Mac mini (home LAN) +---------------------+ iroh QUIC +--------------------------+ | Sotto app | (dial by public key, | orchestrator (FastAPI) | | capture · VAD |===== E2E encrypted, ======| agent loop · store | | playback · tools | direct P2P or relay) | STT · LLM · TTS | +---------------------+ +--------------------------+ ``` ## The transport: iroh The phone reaches the Mac by dialing its public key, never an IP address. The link is app-level, not a system VPN: it runs inside the app over QUIC, so it occupies none of Android's single system-VPN slot and needs no TUN device or elevated networking. Confidentiality, integrity, and authenticity come from iroh's QUIC/TLS 1.3 session plus a `NodeId` allowlist, so every connection is encrypted and admitted only if its public key is on the Mac's list. iroh is owned by **one Rust sidecar daemon** (`sidecar/`) that exposes a local socket. The Python orchestrator, the Kotlin app, and the Swift mac-app all talk to that one sidecar rather than binding their own per-language iroh FFI. The data plane is deliberately dumb: once a peer is trusted and connected, the sidecar tunnels opaque bytes (the orchestrator's existing HTTP and SSE traffic) over a single QUIC connection and adds no application semantics of its own. - **Mac side (listener).** The sidecar binds an iroh endpoint, accepts authenticated connections, and forwards their byte streams to the orchestrator's loopback HTTP server. It opens no internet-facing port. - **Phone side (dialer).** The sidecar dials the Mac by `NodeId` and exposes a local loopback proxy the app's HTTP client targets, so the app just makes ordinary HTTP requests to localhost. By default the Mac exposes **no public attack surface**: it is dialed by key and answers only authenticated peers, with no port to find or forward. The only optional infrastructure is a relay, used when two peers cannot punch a direct path, and it forwards **ciphertext only**. A directly-reachable posture (a single, allowlist-gated iroh port) is an explicit opt-in for a relay-free direct link, not the default. See [`specs/transport.md`](specs/transport.md) for the sidecar socket protocol, node identity, pairing, and the relay contract, and [`specs/sync.md`](specs/sync.md) for how phone and Mac reconcile state across reconnects. ## The orchestrator The orchestrator (`server/orchestrator/`) is the Python FastAPI brain and the contract everything else is built around. It exposes an OpenAI-compatible surface so any standard client can drive it. Its guiding rule is that the HTTP layer stays thin and the real work lives in dedicated modules: - **Routing** is in `main.py` and `rest.py`; streaming responses are assembled in `sse.py`. These stay thin: parse, validate, delegate. - **The agent loop** is `agent.py`: it runs the turn, decides when the model should call a tool, dispatches the call, and feeds the result back to the model until the reply is final. See [`specs/agent-loop.md`](specs/agent-loop.md). - **The model boundary** is the `model/` package, including the router that maps a client-facing alias to a served tier (below). - **Persistence** is behind `store.py` and `sql_store.py`: SQLite by default, a Postgres variant when `DATABASE_URL` selects it, with Alembic `migrations/`. Chats, projects, and memories are the canonical state, and they live here on the Mac, not on the phone. See [`specs/data-and-rest.md`](specs/data-and-rest.md) and [`specs/memory.md`](specs/memory.md). - **Tools** are declared in `tools.py`. They are opt-in and two-tier: only enabled tools reach the model as callable function schemas, while disabled ones are visible as name and one-line description only. Every tool maps to a real Android permission or Shizuku grant, carries a risk tier, and is off by default; high-risk tools require per-use confirmation, and per-project overrides apply. See [`specs/tools.md`](specs/tools.md). - **Configuration** is centralized in `config.py`, and the uniform error envelope is in `errors.py`. See [`specs/config-and-secrets.md`](specs/config-and-secrets.md). The public request and SSE shapes are defined in [`specs/api.md`](specs/api.md), and the layered test tiers (unit, live model-serving, hardware, manual) are in [`specs/testing.md`](specs/testing.md). ## The models The language model is [Gemma 4](https://ollama.com/library/gemma4) (a real, Apache-2.0 family released April 2026). The choice of size is a latency-versus-quality dial, and Sotto settles on three points: - **26B MoE (fast tier)**, the conversational default. Its low active-parameter count keeps tokens fast enough for real-time dialogue. - **31B Dense (quality tier)**, for harder reasoning or tool planning, at a bit more latency. - **E4B (edge tier)**, small enough to live on the phone as the offline safety net when the link or the Mac is unavailable. Clients never send a raw backend tag. They send a stable alias and the `ModelRouter` in `model/` resolves it to a served tier, so deploy-time tags stay out of the request path and cannot leak into clients: - `sotto-fast` maps to the fast tier, `sotto-quality` to the quality tier. - `sotto-auto` lets the router choose per turn and echoes back the concrete alias it used, so the decision is observable. - `sotto-local` names the on-device edge tier; if it reaches the orchestrator it falls back to fast with a warning, since the edge model normally answers on the phone. Serving runs **natively** on the Mac for Metal acceleration (Docker on macOS cannot reach the Apple GPU), behind an Ollama-compatible API, and graduates to MLX on the hot path for higher tokens-per-second on Apple Silicon. The exact served tags live in config and the router, never hardcoded in the loop; see the deploy config and [`specs/config-and-secrets.md`](specs/config-and-secrets.md). ## Speech: transcription and synthesis Speech-to-text and text-to-speech turn a chat turn into a spoken one, and both run on the Mac beside the LLM. - **STT** is NVIDIA Parakeet, the speed leader on Apple Silicon, running native on the Neural Engine. whisper.cpp (Large V3 Turbo, Metal) is the multilingual fallback and switches in when Parakeet is unavailable. Both are reached behind an OpenAI-compatible `/v1/audio/transcriptions` endpoint. - **TTS** is [Kokoro](https://github.com/hexgrad/kokoro), an 82M-parameter model served through Kokoro-FastAPI behind a streaming `/v1/audio/speech` endpoint. Audio is **ephemeral**: it is transcribed, acted on, and discarded, and only the rolling text context is kept. The wire format for both directions is fixed in [`specs/audio.md`](specs/audio.md) so the phone and the Mac agree on framing. The single most important latency trick lives here: **TTS streams**. The system does not wait for the full LLM response before speaking. Tokens pipe into Kokoro sentence by sentence, so the first words are audible while the model is still generating. The model is prompted for plain speech only, no markdown, so nothing pollutes the spoken output. ## A turn, end to end A turn is a loop, and the same loop serves both the voice front end and the text chatbot; only the first and last steps differ for voice. 1. **Wake and capture (phone).** The app wakes on the wake word, captures audio until a voice-activity detector hears you stop, and ships the compressed audio over the iroh link. (For a text turn, this is just an HTTP request.) 2. **Transcribe (Mac).** STT turns the audio into text. 3. **Reason (Mac).** The agent loop in `agent.py` runs the model. If the model calls a tool, the loop dispatches it (a device action runs back on the phone), feeds the result in, and continues until the reply is final. The router picks the tier; the store records the turn. 4. **Speak (Mac to phone).** As the model emits sentences, TTS streams spoken audio back over the link, and the phone plays it while generation continues. Off-network, or with the Mac asleep, step 3 degrades gracefully: the phone answers from the on-device E4B model until the link is back. For the exact timing target at each stage, and how it is measured, see [`specs/latency-budgets.md`](specs/latency-budgets.md). ## Where to read next - Interface contracts, the source of truth: [`specs/`](specs/) (transport, api, agent-loop, data-and-rest, audio, latency-budgets, tools, memory, sync, config-and-secrets, testing). - The design and its reasoning: the [design page](index.html). - Getting a stack running: [`quickstart.md`](quickstart.md) and [`bootstrap.md`](bootstrap.md). - The security posture: [`specs/threat-model.md`](specs/threat-model.md).