How mandatoryprogrammer/tarnish works
Project Type & Purpose
• This is a full-stack, distributed static-analysis tool (“tarnish”) for auditing Chrome extensions.
• It comprises a static JavaScript/CSS/HTML frontend hosted on S3, a Python Tornado web/API backend that enqueues analysis jobs, and horizontally scaling Celery workers that perform the analysis.
File-Structure Analysis
• site/ → All static frontend assets (HTML, CSS, JS, Ace editor modes, highlight.js themes) intended for an S3 (or simple HTTP) static website.
• tarnish-server/ → The Tornado + Celery API server:
• server.py + celerylib.py: HTTP endpoints to submit scans, fetch results, view manifest.
• Dockerfile: container image for Elastic Beanstalk or local Docker.
• docker-compose.yaml.example: local orchestration of server, Redis, and optional worker.
• tarnish-worker/ (and tarnish-server/tarnishworker/) → Celery worker(s):
• tasks.py + celerylib.py: download .crx, unzip, run static scans (fingerprinting, CSP analysis, Retire.js lookup, dangerous-function detection) using the bundled chromium-docs and libs/.
• Dockerfile & .ebextensions/Dockerrun.aws.json: container and EB config for Spot Instances.
• start_workers.sh: bootstraps worker processes.
• tornado-celery/ → Vendored library providing Celery integration for Tornado I/O loop.
• Top-level scripts (start.sh, update-site.sh, beanstalk_conf_gen.sh): helpers to deploy server and upload static site.
a. Main Components
• User’s Browser (UI) – hosts static site from S3
• S3 Static-Site Bucket – serves site/ files
• API Gateway or Load Balancer – routes “tarnish” AJAX calls to the Tornado server
• Tornado Web/API Container (tarnish-server) – receives scan requests, queries Redis, reads/writes S3 result cache
• Redis Broker – message queue broker for Celery tasks (e.g., Elasticache or container)
• Celery Worker Containers (tarnish-worker) – pull jobs, perform static analysis, push results to S3
• S3 Results Bucket – stores scan reports & cached JSON for fast repeat fetches
• External Chrome Web Store API – source of extension CRXs
b. Relationships & Data Flow
1. Browser → (HTTP GET) → S3 Static Bucket → serves index.html and JS/CSS.
2. Browser → (XHR POST) → API Gateway → Tornado API “submit scan” endpoint.
3. Tornado → (PUBLISH) → Redis broker enqueues scan task.
4. Celery Worker → (CONSUME) → Redis → downloads extension CRX from Chrome Web Store.
5. Worker → PROCESS → static-analysis modules (chromium-docs, Retire.js, Python parsers) → generates JSON report.
6. Worker → (PUT) → S3 Results Bucket (cache & report files).
7. Tornado → (GET) → S3 Results Bucket (on subsequent requests) → returns JSON to Browser.
c. Architectural Patterns & Principles
• Producer/Consumer (asynchronous job queue via Celery + Redis)
• Separation of Concerns:
• Frontend (pure static assets)
• API layer (Tornado HTTP + Celery producer logic)
• Worker layer (Celery consumers executing analysis)
• Horizontal scalability (Elastic Beanstalk + Spot Instances for worker fleet)
• Caching (S3-backed results to avoid reprocessing)
d. Key Technologies & Frameworks
• AWS S3 (static hosting & results storage)
• AWS Elastic Beanstalk (server & worker deployment, auto-scaling Spot Instances)
• Docker containers (tarnish-server, tarnish-worker)
• Python Tornado (web server)
• Celery + Redis (task queue)
• chromium-docs data (IDL/JSON specs for API fingerprinting)
• Retire.js (vulnerable-library detection)
• Ace Editor & highlight.js (frontend code viewer)