How kenkoooo/atcoderproblems works
Project type and purpose
• This is a full‑stack web application called “AtCoder Problems” composed of:
• A React/TypeScript front‑end (under atcoder‑problems‑frontend/)
• A Rust‑based back‑end server (under atcoder‑problems‑backend/)
• Serverless pieces (AWS Lambda functions under lambda‑functions/)
• Supporting assets (database schema under config/, documentation under doc/ and guide/)
• Its purpose is to let AtCoder users view problem lists, track submissions, see rankings, manage virtual contests, and more.
File structure
• Top‑level directories:
• atcoder‑problems‑frontend: static UI, API clients, local IndexedDB
• atcoder‑problems‑backend: crawler binaries, server endpoints, SQL‑client crate, atcoder‑client crate
• lambda‑functions: official‑api‑proxy and time‑estimator scripts
• config: database table definitions
• doc/guide: user and developer documentation
• Backend patterns:
• A set of command‑line binaries (in src/bin) for batch crawling and updates
• A layered server: endpoint → services → SQL‑client → database
• Use of AWS S3 (s3.rs) for storing JSON snapshots
• Middleware for GitHub OAuth
• Front‑end patterns:
• React components in src/components and src/pages
• Two API clients: APIClient (public) and InternalAPIClient (authenticated)
• Cypress for E2E tests
• Local cache in IndexedDB (SubmissionsDB.ts) and static_data for initial boot
README for technical stack
• Front‑end: React, TypeScript, Yarn, Cypress
• Back‑end: Rust (Cargo.toml), Dockerized (Dockerfile & docker-compose.yml), Postgres via SQL‑client crate
• Lambda: Python scripts, build.sh for packaging, invoked via API Gateway
• External dependencies: AtCoder official API (proxied), GitHub OAuth, AWS S3
Main components
a. Front‑end (Browser/React app)
b. API Gateway(s) or reverse‑proxy endpoints
c. Back‑end server (Rust)
d. PostgreSQL database
e. AWS S3 bucket for JSON snapshots
f. Batch crawler services (Docker containers or cron jobs)
g. AWS Lambda functions:
• official‑api‑proxy (Python)
• time‑estimator (Python)
h. External services: AtCoder official API, GitHub OAuth
Relationships and data flow
• Browser loads static HTML/CSS/JS from CDN or front‑end host
• Browser → public API endpoints → back‑end server
• Browser → internal API endpoints (authenticated via GitHub OAuth middleware) → back‑end
• Back‑end server → SQL‑client → PostgreSQL (queries for user/submission data)
• Back‑end server → S3 (read/write JSON snapshots)
• Front‑end calls Lambda functions via AWS API Gateway for:
• official API data (to bypass CORS)
• time estimation service
• Batch crawler binaries run on schedule in containers → use atcoder‑client library to scrape official AtCoder → update DB and push snapshots to S3
Architectural patterns and design principles
• Layered architecture in back‑end: CLI crawlers, HTTP server, service layer, data access layer
• Separation of concerns: static UI vs. dynamic APIs vs. crawler jobs vs. serverless functions
• Use of serverless for edge‑needed functionality (proxied API, compute‑on‑demand)
• Strong typing and modular crates in Rust (atcoder‑client, sql‑client)
• Client‑side state and cache (IndexedDB) to reduce server load
Technologies/Frameworks to annotate
• Front‑end: React, TypeScript, Yarn, Cypress
• Back‑end: Rust, tokio/Actix/Rocket (as chosen), Diesel or sql‑client
• Containerization: Docker, docker‑compose
• Database: PostgreSQL
• Cloud: AWS S3, Lambda, API Gateway
• Auth: GitHub OAuth
• Clear, descriptive labels on every component (e.g. “Rust HTTP Server”)
• Arrowed lines for request/response and data flows
• Front‑end vs. back‑end vs. serverless vs. storage vs. external
• Notes on batch jobs vs. real‑time services
• Authentication flow via GitHub OAuth middleware
Details
• Crawler binaries as a separate service converging into the same DB/S3
• Expose that atcoder‑client crate is used by crawlers and sql‑client crate by both services and tests
• Internal vs. public API endpoints in the server (e.g. internal_api/* vs. healthcheck)
• Static_data embed vs. dynamic fetch