How daeisbae/open-repo-wiki works
1. What this repository is
- OpenRepoWiki is a monorepo that generates wiki-style documentation for GitHub repositories.
- The system is split into a static frontend, a serverless API layer, and a long-running processing pipeline that analyzes code and produces summaries.
2. Top-level architecture
- The main runtime boundary is AWS.
- User traffic enters through CloudFront, which serves the frontend from S3 and forwards API calls to API Gateway.
- The backend is mostly serverless except for the code-analysis worker, which runs in ECS Fargate.
3. User-facing frontend
- frontend/ is a React + Vite app.
- Its main UI pieces are repo submission, progress tracking, tree browsing, breadcrumbs, and page/document viewing.
- The frontend talks to the API through a small client layer, and requests are signed before being sent to protected endpoints.
4. API boundary
- services/api/ contains Lambda handlers for the public API.
- There are separate handlers for repository operations, job operations, and request authorization.
- API Gateway is the front door, and the Lambda authorizer enforces signed requests on sensitive calls.
5. Job orchestration path
- A repository request becomes a job record and then a workflow execution.
- jobs in the API layer trigger Step Functions, which orchestrate the processing lifecycle.
- This creates a clear split between synchronous request handling and asynchronous repository analysis.
6. Processing worker
- services/processor/ is the ECS Fargate container that performs the heavy lifting.
- It is organized as a staged pipeline: fetch repository tree, fetch file details, filter/collapse the tree, summarize content, then finalize results.
- This is the subsystem that actually turns raw GitHub source into the wiki structure the UI displays.
7. LLM integration
- The processor has an LLM abstraction under services/processor/llm/.
- Provider implementations include DeepSeek and OpenRouter, with prompts, schemas, and code-splitting logic layered around them.
- The LLM is used only inside the processing pipeline, not in the frontend or API routing path.
8. External data sources
- GitHub is the primary source of repository content.
- The github/ and shared/github/ modules encapsulate GitHub client behavior and tree filtering/collapsing logic.
- These modules are reused across the backend so that repository traversal and pruning are consistent.
9. State and artifact storage
- DynamoDB stores durable job/repository state such as progress, metadata, and generated summaries.
- S3 stores static frontend assets and likely generated artifacts or outputs from the processing pipeline.
- The shared/storage/ package and the root storage/ package both wrap these persistence concerns.
10. Infrastructure layer
- infra/terraform/ defines the AWS deployment as reusable modules: CloudFront, API Gateway, Lambda, ECS, Step Functions, DynamoDB, S3, IAM, VPC, WAF, ACM, Route53, Secrets Manager, and ECR.
- WAF and the Lambda authorizer provide the main security boundary for the API.
- Secrets Manager holds runtime secrets such as API keys and signing material.
11. Shared domain model
- models/ and shared/models/ define common entities like repo, branch, job, and tree node.
- This shared model layer keeps the frontend, API, and processor aligned on the same data shape and vocabulary.
12. Testing and operational concerns
- tests/ includes end-to-end, LocalStack-based integration, and property/concurrency tests.
- The presence of LocalStack tests indicates the architecture is designed to be validated against AWS-like services without always hitting real cloud resources.
- docs/ and the service READMEs serve as the formal boundary descriptions for the architecture and API behavior.
Main entry points: dynamodb.py Storage Wrappers · main.py Processor · lambda_handler.py Lambda Entry · client.py GitHub Core
Components 25
API Layer
Serverless request handling and authorization.
- Authorizer · Verifies signed requests for protected API calls.
services/api/handlers/authorizer.py - Repo API · Handles repository submission and repository-level operations.
services/api/handlers/repos.py - Job API · Creates and tracks jobs, and triggers workflow execution.
services/api/handlers/jobs.py - Lambda Entry · Dispatches incoming API Gateway requests to the right handler.
services/api/lambda_handler.py
Processor
ECS-based analysis pipeline that builds wiki content.
- Processor · Long-running worker that runs the analysis pipeline.
services/processor/main.py - Fetch Tree · Loads repository structure from GitHub.
services/processor/stages/fetch_tree.py - Fetch Details · Collects file-level content and metadata for analysis.
services/processor/stages/fetch_details.py - Filter Tree · Prunes and collapses the repository tree for readability.
services/processor/stages/filter_tree.py - Summarize · Uses LLMs to generate wiki-style summaries.
services/processor/stages/summarize.py - Finalize · Persists generated outputs and completes the job.
services/processor/stages/finalize.py - LLM Layer · Provider-agnostic interface for code splitting and summarization.
services/processor/llm/factory.py - DeepSeek · One supported model backend for summarization.
services/processor/llm/providers/deepseek.py - OpenRouter · Alternate model backend for summarization.
services/processor/llm/providers/openrouter.py
Shared Core
Common models, GitHub helpers, and persistence wrappers.
- GitHub Core · Shared client, traversal, filtering, and collapse logic.
shared/github/client.py - Domain Models · Common repo, branch, job, and tree node entities.
shared/models/job.py - Storage Wrappers · Shared DynamoDB and S3 access wrappers.
shared/storage/dynamodb.py
Infrastructure
Terraform modules and deployment plumbing.
- CloudFront · Front door for users; serves static content and forwards API traffic.
infra/terraform/modules/cloudfront/main.tf - Static S3 · Hosts frontend assets and may store generated artifacts.
infra/terraform/modules/s3/main.tf - API Gateway · Public HTTP entry point for repository and job operations.
infra/terraform/modules/apigw/main.tf - Step Functions · Orchestrates asynchronous repository analysis lifecycle.
infra/terraform/modules/sfn/main.tf - DynamoDB · Durable job, progress, metadata, and summary state.
infra/terraform/modules/dynamodb/main.tf - Secrets · Holds API keys and signing material.
infra/terraform/modules/secrets/main.tf - VPC · Private networking boundary for backend workloads.
infra/terraform/modules/vpc/main.tf - WAF · Protects the public API edge with request filtering.
infra/terraform/modules/waf/main.tf
Tests
End-to-end and AWS-emulation validation.
- LocalStack · AWS-emulation tests for infrastructure-backed flows.
tests/localstack/test_localstack_integration.py
Connections 31
- CloudFront → Static S3: serves UI · CloudFront delivers static frontend assets from S3.
- CloudFront → API Gateway: routes API · API requests are forwarded from the edge to API Gateway.
- WAF → API Gateway: protects · WAF filters inbound traffic before the public API.
- API Gateway → Lambda Entry: invokes · Gateway sends requests into the Lambda entry router.
- Lambda Entry → Authorizer: auth · Protected requests are validated by the Lambda authorizer.
- Lambda Entry → Repo API: repo ops · Repository operations are routed to repo handlers.
- Lambda Entry → Job API: job ops · Job and workflow operations are routed to job handlers.
- Job API → Step Functions: start workflow · A job record can trigger Step Functions orchestration.
- Step Functions → Processor: run analysis · Workflow execution drives the processor task.
- Processor → Fetch Tree: stage · Pipeline begins by loading repository structure.
- Fetch Tree → Fetch Details: stage · Tree discovery feeds file detail collection.
- Fetch Details → Filter Tree: stage · Collected data is reduced into a usable tree.
- Filter Tree → Summarize: stage · Filtered repository structure is summarized.
- Summarize → LLM Layer: uses · Summarization runs through the LLM abstraction.
- LLM Layer → DeepSeek: provider · One backend implementation is DeepSeek.
- LLM Layer → OpenRouter: provider · Another backend implementation is OpenRouter.
- Summarize → Finalize: stage · Generated content is finalized and persisted.
- GitHub Core → Fetch Tree: source · GitHub traversal helpers support tree fetching.
- GitHub Core → Fetch Details: source · GitHub client logic supports detail retrieval.
- GitHub Core → Filter Tree: collapse · Shared filtering/collapsing logic is reused in pruning.
- Domain Models → Repo API: shapes · API handlers use shared repository and job schemas.
- Domain Models → Processor: shapes · Processor stages exchange shared domain objects.
- Storage Wrappers → DynamoDB: writes · Shared wrappers persist durable state in DynamoDB.
- Storage Wrappers → Static S3: reads/writes · Shared wrappers access S3 for assets or artifacts.
- Repo API → Storage Wrappers: state · Repository operations read and write durable state.
- Job API → Storage Wrappers: state · Job operations read and update progress and metadata.
- Processor → Storage Wrappers: persist · Processor stores outputs and status through shared storage.
- Secrets → Authorizer: signing · Signing material is consumed by request authorization.
- VPC → Processor: network · Processor runs inside the backend network boundary.
- LocalStack → Step Functions: validates · Integration tests exercise cloud-like orchestration paths.
- LocalStack → Storage Wrappers: validates · Tests cover persistence behavior against AWS-like services.