nossa-y/activity-frames

Purpose and Entry Points

activity-frames is a local Python compiler that turns screen-capture SQLite data into deterministic, evidence-linked activity records, agent context, workflow steps, and repetition signals. Primary user entry points are the aframes CLI and Python ActivityLog; agents connect through an MCP stdio server. Core package surface: src/activity_frames/__init__.py, src/activity_frames/cli.py, src/activity_frames/mcp_server.py.

Capture and Recorder Boundary

The package consumes compatible local SQLite tables (frames, ui_events, elements) rather than owning capture format. aframes record manages the external nocta-recorder binary, including provisioning/status/stop behavior; users can instead supply a database via AFRAMES_DB or CLI options. Capture includes snapshots, accessibility data, and input events; OCR occurs in the external on-device recorder. Paths: src/activity_frames/capture.py, src/activity_frames/db.py, src/activity_frames/cli.py.

Read-Only Data Access and Normalization

Database access is a privacy and correctness boundary: compilation reads the capture database without modifying it. The database layer retrieves raw temporal/UI records, while shared time handling normalizes timestamps and date/hour windows for all consumers. Paths: src/activity_frames/db.py, src/activity_frames/_time.py, src/activity_frames/frames.py.

Frame Compilation Pipeline

The main compiler converts event instants into bounded activity frames. Sessionization determines dwell periods, inactivity/gaps, and flicker merging; frame construction groups these periods and maintains evidence references to source rows. Enrichment adds deterministic app/site/page metadata and input aggregates, preserving the measured-only contract. Paths: src/activity_frames/sessionize.py, src/activity_frames/frames.py, src/activity_frames/enrich.py.

Site Entities, Communications, and Patterns

Entity extraction maps supported website and application surfaces to structured page references, with generic fallback for unknown sites. Communications reports messaging/email surfaces and observed titles only, not message content. Pattern analysis detects repeated workflows and behavioral sequences; text-based detection is gated by explicit content inclusion. Paths: src/activity_frames/entities.py, src/activity_frames/communications.py, src/activity_frames/patterns.py.

Workflow Replay Representation

The steps stage expands an activity frame into ordered focus, click, and typing actions, grounded by captured element name, role, URL, and timestamps. This is a deterministic replay representation, not an executor: downstream agents may fill variable values and must halt rather than guess when UI state diverges. Paths: src/activity_frames/steps.py, src/activity_frames/frames.py, research/compile_replay.py.

Outputs and Agent Integration

Compiled documents are emitted as context blocks and structured JSON/YAML/Markdown-style output. The CLI exposes day/recent context, apps, patterns, communications, steps, and recording operations. The MCP server wraps these capabilities as six tools over stdio for MCP-compatible clients. PyYAML is optional for YAML serialization; no LLM or network service participates in compilation. Paths: src/activity_frames/emit.py, src/activity_frames/cli.py, src/activity_frames/mcp_server.py.

Runtime and Research Context

Runtime is Python, packaged via pyproject.toml; macOS is the primary recorder target, while the compiler can operate on Linux with a compatible external recorder. Research scripts benchmark routine-overhead claims and browser replay experiments, but remain separate from the package’s measured compilation path. Paths: pyproject.toml, research/bench_three_arm.py, research/replay_agentbrowser.py.