# nossa-y/activity-frames architecture

> Architecture diagram of the GitHub repository nossa-y/activity-frames, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nossa-y/activity-frames
- Repository: https://github.com/nossa-y/activity-frames

## Overview

#### Purpose and Entry Points
`activity-frames` is a local Python compiler that turns screen-capture SQLite data into deterministic, evidence-linked activity records, agent context, workflow steps, and repetition signals. Primary user entry points are the `aframes` CLI and Python `ActivityLog`; agents connect through an MCP stdio server. Core package surface: `src/activity_frames/__init__.py`, `src/activity_frames/cli.py`, `src/activity_frames/mcp_server.py`.

#### Capture and Recorder Boundary
The package consumes compatible local SQLite tables (`frames`, `ui_events`, `elements`) rather than owning capture format. `aframes record` manages the external `nocta-recorder` binary, including provisioning/status/stop behavior; users can instead supply a database via `AFRAMES_DB` or CLI options. Capture includes snapshots, accessibility data, and input events; OCR occurs in the external on-device recorder. Paths: `src/activity_frames/capture.py`, `src/activity_frames/db.py`, `src/activity_frames/cli.py`.

#### Read-Only Data Access and Normalization
Database access is a privacy and correctness boundary: compilation reads the capture database without modifying it. The database layer retrieves raw temporal/UI records, while shared time handling normalizes timestamps and date/hour windows for all consumers. Paths: `src/activity_frames/db.py`, `src/activity_frames/_time.py`, `src/activity_frames/frames.py`.

#### Frame Compilation Pipeline
The main compiler converts event instants into bounded activity frames. Sessionization determines dwell periods, inactivity/gaps, and flicker merging; frame construction groups these periods and maintains evidence references to source rows. Enrichment adds deterministic app/site/page metadata and input aggregates, preserving the measured-only contract. Paths: `src/activity_frames/sessionize.py`, `src/activity_frames/frames.py`, `src/activity_frames/enrich.py`.

#### Site Entities, Communications, and Patterns
Entity extraction maps supported website and application surfaces to structured page references, with generic fallback for unknown sites. Communications reports messaging/email surfaces and observed titles only, not message content. Pattern analysis detects repeated workflows and behavioral sequences; text-based detection is gated by explicit content inclusion. Paths: `src/activity_frames/entities.py`, `src/activity_frames/communications.py`, `src/activity_frames/patterns.py`.

#### Workflow Replay Representation
The steps stage expands an activity frame into ordered focus, click, and typing actions, grounded by captured element name, role, URL, and timestamps. This is a deterministic replay representation, not an executor: downstream agents may fill variable values and must halt rather than guess when UI state diverges. Paths: `src/activity_frames/steps.py`, `src/activity_frames/frames.py`, `research/compile_replay.py`.

#### Outputs and Agent Integration
Compiled documents are emitted as context blocks and structured JSON/YAML/Markdown-style output. The CLI exposes day/recent context, apps, patterns, communications, steps, and recording operations. The MCP server wraps these capabilities as six tools over stdio for MCP-compatible clients. PyYAML is optional for YAML serialization; no LLM or network service participates in compilation. Paths: `src/activity_frames/emit.py`, `src/activity_frames/cli.py`, `src/activity_frames/mcp_server.py`.

#### Runtime and Research Context
Runtime is Python, packaged via `pyproject.toml`; macOS is the primary recorder target, while the compiler can operate on Linux with a compatible external recorder. Research scripts benchmark routine-overhead claims and browser replay experiments, but remain separate from the package’s measured compilation path. Paths: `pyproject.toml`, `research/bench_three_arm.py`, `research/replay_agentbrowser.py`.

Main entry points: [`src/activity_frames/db.py`](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/db.py) (Read-only database access), [`src/activity_frames/emit.py`](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/emit.py) (Document emitters), [`src/activity_frames/enrich.py`](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/enrich.py) (Frame enrichment), [`src/activity_frames/frames.py`](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/frames.py) (Activity frames)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["Interfaces"]
  node_cli["aframes CLI<br/>Python CLI<br/>[cli.py]"]
  node_activity_log["ActivityLog<br/>Python API<br/>[__init__.py]"]
  node_mcp{{"MCP stdio server<br/>MCP server<br/>[mcp_server.py]"}}
  node_package["Python package<br/>runtime package<br/>[pyproject.toml]"]
end

subgraph group_capture["Capture Boundary"]
  node_recorder["nocta-recorder<br/>external recorder"]
  node_capture_db[("Capture SQLite DB<br/>SQLite schema")]
  node_capture_control["Recorder control<br/>capture adapter<br/>[capture.py]"]
end

subgraph group_compiler["Deterministic Compiler"]
  node_db[("Read-only database access<br/>data access<br/>[db.py]")]
  node_time["Time normalization<br/>shared utility<br/>[_time.py]"]
  node_sessionize["Sessionization<br/>pipeline stage<br/>[sessionize.py]"]
  node_frames["Activity frames<br/>pipeline stage<br/>[frames.py]"]
  node_enrich["Frame enrichment<br/>pipeline stage<br/>[enrich.py]"]
  node_emit["Document emitters<br/>serialization<br/>[emit.py]"]
end

subgraph group_analysis["Analysis &amp; Replay"]
  node_entities["Site entities<br/>entity extraction<br/>[entities.py]"]
  node_communications["Communications<br/>analysis<br/>[communications.py]"]
  node_patterns["Workflow patterns<br/>analysis<br/>[patterns.py]"]
  node_steps["Replay steps<br/>replay representation<br/>[steps.py]"]
  node_replay_research["Replay research<br/>research script<br/>[compile_replay.py]"]
  node_bench_research["Benchmark research<br/>research script<br/>[bench_three_arm.py]"]
  node_agentbrowser_research["Agent browser research<br/>research script"]
end

node_cli -->|"record commands"| node_capture_control
node_capture_control -->|"manages"| node_recorder
node_recorder -->|"writes capture data"| node_capture_db
node_cli -->|"queries"| node_db
node_activity_log -->|"queries"| node_db
node_mcp -->|"serves tools through"| node_activity_log
node_capture_db -->|"read only"| node_db
node_db -->|"timestamps"| node_time
node_db -->|"temporal/UI records"| node_sessionize
node_time -->|"normalized windows"| node_sessionize
node_sessionize -->|"bounded periods"| node_frames
node_db -->|"evidence rows"| node_frames
node_frames -->|"compiled frames"| node_enrich
node_entities -->|"page references"| node_enrich
node_enrich -->|"observed surfaces"| node_communications
node_enrich -->|"activity sequences"| node_patterns
node_frames -->|"frame evidence"| node_steps
node_enrich -->|"compiled documents"| node_emit
node_communications -->|"communication reports"| node_emit
node_patterns -->|"repetition signals"| node_emit
node_steps -->|"workflow steps"| node_emit
node_cli -->|"formats output"| node_emit
node_activity_log -->|"returns output"| node_emit
node_replay_research -.->|"research only"| node_steps
node_bench_research -.->|"research only"| node_frames
node_agentbrowser_research -.->|"research only"| node_steps
node_package -->|"packages"| node_cli

click node_cli "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/cli.py"
click node_activity_log "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/__init__.py"
click node_mcp "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/mcp_server.py"
click node_capture_control "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/capture.py"
click node_db "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/db.py"
click node_time "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/_time.py"
click node_sessionize "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/sessionize.py"
click node_frames "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/frames.py"
click node_enrich "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/enrich.py"
click node_entities "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/entities.py"
click node_communications "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/communications.py"
click node_patterns "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/patterns.py"
click node_steps "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/steps.py"
click node_emit "https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/emit.py"
click node_replay_research "https://github.com/nossa-y/activity-frames/blob/main/research/compile_replay.py"
click node_bench_research "https://github.com/nossa-y/activity-frames/blob/main/research/bench_three_arm.py"
click node_agentbrowser_research "https://github.com/nossa-y/activity-frames/blob/main/research/replay_agentbrowser.py"
click node_package "https://github.com/nossa-y/activity-frames/blob/main/pyproject.toml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_activity_log,node_mcp,node_package toneBlue
class node_recorder,node_capture_db,node_capture_control toneAmber
class node_db,node_time,node_sessionize,node_frames,node_enrich,node_emit toneMint
class node_entities,node_communications,node_patterns,node_steps,node_replay_research,node_bench_research,node_agentbrowser_research toneRose
```

## Components

### Interfaces

- [aframes CLI](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/cli.py): Python CLI · Exposes compilation queries and recorder operations. · `src/activity_frames/cli.py`
- [ActivityLog](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/__init__.py): Python API · `src/activity_frames/__init__.py`
- [MCP stdio server](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/mcp_server.py): MCP server · Wraps package capabilities as six tools. · `src/activity_frames/mcp_server.py`
- [Python package](https://github.com/nossa-y/activity-frames/blob/main/pyproject.toml): runtime package · Local Python compiler; PyYAML is optional for YAML output. · `pyproject.toml`

### Capture Boundary

- **nocta-recorder**: external recorder · Captures snapshots, accessibility data, OCR, and input events.
- **Capture SQLite DB**: SQLite schema · Compatible frames, ui_events, and elements tables.
- [Recorder control](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/capture.py): capture adapter · Provisions and manages the external recorder. · `src/activity_frames/capture.py`

### Deterministic Compiler

- [Read-only database access](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/db.py): data access · Reads source capture rows without modifying them. · `src/activity_frames/db.py`
- [Time normalization](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/_time.py): shared utility · Normalizes timestamps and date/hour windows. · `src/activity_frames/_time.py`
- [Sessionization](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/sessionize.py): pipeline stage · Computes dwell, gaps, inactivity, and flicker merging. · `src/activity_frames/sessionize.py`
- [Activity frames](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/frames.py): pipeline stage · Builds bounded frames with source-row evidence references. · `src/activity_frames/frames.py`
- [Frame enrichment](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/enrich.py): pipeline stage · Adds deterministic app, page, site, and input aggregates. · `src/activity_frames/enrich.py`
- [Document emitters](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/emit.py): serialization · Emits context blocks and JSON/YAML/Markdown-style output. · `src/activity_frames/emit.py`

### Analysis & Replay

- [Site entities](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/entities.py): entity extraction · Maps supported surfaces to page references with generic fallback. · `src/activity_frames/entities.py`
- [Communications](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/communications.py): analysis · Reports observed messaging/email surfaces and titles only. · `src/activity_frames/communications.py`
- [Workflow patterns](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/patterns.py): analysis · Detects repeated workflows; text detection requires explicit inclusion. · `src/activity_frames/patterns.py`
- [Replay steps](https://github.com/nossa-y/activity-frames/blob/main/src/activity_frames/steps.py): replay representation · Produces ordered grounded focus, click, and typing actions. · `src/activity_frames/steps.py`
- [Replay research](https://github.com/nossa-y/activity-frames/blob/main/research/compile_replay.py): research script · Separate browser replay experiment. · `research/compile_replay.py`
- [Benchmark research](https://github.com/nossa-y/activity-frames/blob/main/research/bench_three_arm.py): research script · Separate routine-overhead benchmark. · `research/bench_three_arm.py`
- [Agent browser research](https://github.com/nossa-y/activity-frames/blob/main/research/replay_agentbrowser.py): research script · Separate browser replay experiment. · `research/replay_agentbrowser.py`

## Connections

- aframes CLI → Recorder control: record commands
- Recorder control → nocta-recorder: manages
- nocta-recorder → Capture SQLite DB: writes capture data
- aframes CLI → Read-only database access: queries
- ActivityLog → Read-only database access: queries
- MCP stdio server → ActivityLog: serves tools through
- Capture SQLite DB → Read-only database access: read only
- Read-only database access → Time normalization: timestamps
- Read-only database access → Sessionization: temporal/UI records
- Time normalization → Sessionization: normalized windows
- Sessionization → Activity frames: bounded periods
- Read-only database access → Activity frames: evidence rows
- Activity frames → Frame enrichment: compiled frames
- Site entities → Frame enrichment: page references
- Frame enrichment → Communications: observed surfaces
- Frame enrichment → Workflow patterns: activity sequences
- Activity frames → Replay steps: frame evidence
- Frame enrichment → Document emitters: compiled documents
- Communications → Document emitters: communication reports
- Workflow patterns → Document emitters: repetition signals
- Replay steps → Document emitters: workflow steps
- aframes CLI → Document emitters: formats output
- ActivityLog → Document emitters: returns output
- Replay research → Replay steps: research only
- Benchmark research → Activity frames: research only
- Agent browser research → Replay steps: research only
- Python package → aframes CLI: packages