How fatihmakes/mark-xlvii works
1. Repository purpose
- This is a cross-platform local AI assistant, not a single-purpose app. The core runtime combines voice input/output, LLM reasoning, OS control, web access, vision/screen processing, and persistent memory.
2. Main execution paths
- main.py is the primary assistant entrypoint.
- ui.py is likely a secondary interface/bootstrap path.
- dashboard/server.py runs a separate local web dashboard, with static assets in dashboard/static/.
3. Core orchestration layer
- The core/ package is the assistant’s runtime spine.
- llm_client.py is the model gateway, and the README makes Gemini the central LLM backend.
- stt.py and tts.py handle speech-to-text and text-to-speech, forming the voice loop.
- prompt.txt provides the base system prompt/personality for the agent.
- installer.py suggests environment/bootstrap setup for missing dependencies or platform-specific setup.
4. Tool / action subsystem
- The actions/ package is the assistant’s toolbelt, where model decisions become real-world operations.
- It includes computer control, browser control, app launching, file operations, reminders, messaging, weather, web search, YouTube, flight search, and game updating.
- dev_agent.py and code_helper.py imply a more specialized coding/autonomous-workflow path.
- This layer is the main boundary between “reasoning” and “doing.”
5. Perception and device control
- screen_processor.py and browser_control.py indicate visual/screen-aware workflows and browser automation.
- computer_control.py, desktop.py, and computer_settings.py cover OS-level interaction and system tuning.
- system_monitor.py provides background telemetry, matching the README’s CPU/RAM/GPU/temperature monitoring and alerting.
6. Persistent memory
- memory/ holds the assistant’s long-term context.
- memory_manager.py and config_manager.py suggest a configurable store for user preferences, identity, and remembered facts like city/language.
- This memory layer is separate from transient chat state, so the assistant can personalize across boots and sessions.
7. Local dashboard boundary
- dashboard/ is an independent presentation layer for monitoring or interaction.
- app.html and login.html show a minimal web frontend, likely served locally by server.py.
- The presence of config/certs/jarvis.crt and jarvis.key suggests the dashboard or local services may run over HTTPS with bundled certificates.
8. Data flow through the system
- The typical flow is: microphone or UI input → STT/text parsing → LLM reasoning → tool selection in actions/ → OS/web/file side effects → response text → TTS and/or dashboard display.
- For vision-enabled tasks, screen/browser data feeds into the reasoning loop through the processor/control modules before actions are executed.
9. External dependencies and services that matter architecturally
- Gemini is the main external AI service and likely governs both general dialogue and grounded search behavior.
- Web search and browser automation depend on external sites and browser tooling, with fallback behavior noted in the README.
- The assistant is designed around OS-native capabilities, so platform-specific APIs and permissions are important architectural constraints.
10. How to diagram it
- The cleanest diagram is a hub-and-spoke model: a central assistant core calling out to voice I/O, memory, dashboard/UI, and action modules.
- The action modules form the execution boundary to the operating system, browser, files, and external web services.
- Memory sits alongside the core as a persistent context store, while the dashboard is an optional local control/visibility surface.
Main entry points: main.py main.py · llm_client.py LLM client · server.py Dashboard server · screen_processor.py Screen vision
Components 20
Runtime
Primary assistant entrypoints and orchestration
- main.py · Primary assistant process
main.py - ui.py · Secondary UI/bootstrap entry
ui.py - LLM client · Routes prompts to the main AI backend
core/llm_client.py - Base prompt · System prompt and assistant behavior rules
core/prompt.txt - Installer · Dependency and platform setup helper
core/installer.py
Voice I/O
Speech input/output loop
- Speech input · Voice to text transcription
core/stt.py - Speech output · Text to speech response synthesis
core/tts.py
Actions
Tooling and side-effect execution
- Web search · Searches external web sources
actions/web_search.py - Browser control · Controls browser-based workflows
actions/browser_control.py - Screen vision · Processes screen or browser visual context
actions/screen_processor.py - Computer control · Performs desktop and OS interactions
actions/computer_control.py - File ops · File processing and controller boundary
actions/file_controller.py - Comm tools · Messaging, reminders, weather, flights, YouTube, apps, game updates
actions/send_message.py - Dev tools · Coding assistant and autonomous work helpers
actions/dev_agent.py - System monitor · Tracks CPU, RAM, GPU, temperature, and alerts
actions/system_monitor.py
Memory
Persistent context and preferences
- Memory store · Persistent long-term assistant memory
memory/memory_manager.py - Prefs config · Saved user preferences and identity context
memory/config_manager.py
Dashboard
Local web UI and server
- Dashboard server · Serves the local dashboard application
dashboard/server.py - Dashboard UI · Local HTML interface for login and status
dashboard/static/app.html - TLS certs · Bundled certificate and key for local HTTPS
config/certs/jarvis.crt
Connections 22
- main.py → Speech input: capture speech · Primary voice input path
- main.py → LLM client: reason · Sends user context and prompts to the model
- main.py → Speech output: speak · Converts assistant responses to audio
- main.py → Memory store: remember · Reads and updates long-term context
- main.py → Prefs config: personalize · Loads user-specific preferences and identity
- main.py → Base prompt: prime · Uses the base assistant prompt
- main.py → Computer control: act · Delegates OS-level actions
- main.py → Web search: lookup · Queries external information
- main.py → Browser control: browse · Runs browser-based tasks
- main.py → Screen vision: see · Feeds visual context into the loop
- main.py → File ops: files · Handles file-oriented actions
- main.py → Comm tools: assist · Uses reminder, messaging, travel, media, and app tools
- main.py → Dev tools: code · Enables coding and autonomous helper flows
- main.py → System monitor: monitor · Reads system telemetry and alerts
- ui.py → LLM client: bootstrap · Alternative UI path into the assistant runtime
- Dashboard server → Dashboard UI: serve · Hosts the local dashboard frontend
- Dashboard server → TLS certs: https · Uses bundled TLS assets for local secure serving
- LLM client → Memory store: context · Can retrieve persistent memory for grounding
- LLM client → Web search: ground · Can delegate external lookup when needed
- LLM client → Dev tools: plan · Can trigger specialized coding workflows
- Screen vision → Browser control: visual loop · Screen-aware automation feeds browser control
- Computer control → System monitor: state · Desktop control and telemetry inform each other