How reethusugan-cpu/advanced_threat_intelligence_platform works
1. System shape
- This repo is a Python-based threat intelligence ingestion and storage pipeline, not a traditional web app.
- The core runtime talks to external threat feeds, normalizes/scoring them, and persists results in MongoDB for later inspection through a Dockerized DB UI.
2. Primary entrypoint
- main.py is the orchestration layer.
- It likely coordinates feed collection, scoring, and database writes rather than implementing business logic itself.
3. Threat feed adapters
- feeds/alienvault.py, feeds/openphish.py, and feeds/urlhaus.py are source-specific collectors.
- Each module represents a boundary to an external intelligence source and is responsible for fetching raw indicators in that source’s format.
4. Normalization and scoring
- scoring/preliminary_scoring.py is the first analysis stage for incoming indicators.
- The scoring/mitre/ package (mitre_parser.py, technique_mapper.py) maps collected activity or indicators to MITRE ATT&CK technique concepts, creating a standardized enrichment layer above raw feed data.
5. Database layer
- database/mongo_handler.py is the persistence abstraction for MongoDB operations.
- database/init.py likely handles initial setup/seed/schema-like bootstrapping for collections, while database/test_connection.py provides a connectivity sanity check.
- clear_db.py is an administrative utility for wiping stored threat data during resets.
6. Data flow
- External feeds → feed modules → scoring/enrichment modules → MongoDB.
- The repository also includes outputs/latest_urlhaus.json, which suggests at least one feed can be materialized to local JSON before or alongside database ingestion.
7. Infrastructure boundary
- docker-compose.yml defines the local runtime environment.
- The README shows the main infrastructure is MongoDB plus a browser-based DB UI exposed on port 8081, with MongoDB on 27017.
- The app code can connect either via localhost from the host machine or the Docker service name atip-mongodb from inside the container network.
8. Log/ETL boundary
- logstash/pipeline/logstash.conf indicates a downstream ingestion/forwarding path for structured events.
- In architecture terms, this is a secondary integration boundary: Python produces or stages data, and Logstash can route it onward if the stack is extended.
9. Configuration and secrets
- .env.example and the README’s environment variable guidance show the project expects secrets and connection details to be injected, not hardcoded.
- This matters architecturally because feed API keys and MongoDB credentials are treated as deployment-time configuration.
10. Operational dependencies
- The stack is intentionally container-first: Docker and Docker Compose are part of the runtime architecture, not just setup tooling.
- The MongoDB container is stateful via volumes, so the database is the durable system-of-record for collected threat intelligence.
11. Repository intent
- The codebase is organized around a few clear layers: source collection, enrichment/scoring, persistence, and local infra.
- There is no separate application server, message broker, or front-end code in this repo; the DB UI is an operational aid rather than a custom product UI.
12. Diagram focus
- An accurate architecture diagram should emphasize external feed sources on the left, Python ingestion/scoring in the center, MongoDB storage and its UI on the right, and Docker Compose as the environment wrapper around the persistence layer.
Main entry points: main.py Main · mongo_handler.py Mongo handler · docker-compose.yml Compose · urlhaus.py URLHaus
Components 16
Threat feeds
External intelligence sources and their repository adapters.
- AlienVault · Collects indicators from AlienVault feed format.
feeds/alienvault.py - OpenPhish · Collects indicators from OpenPhish feed format.
feeds/openphish.py - URLHaus · Collects indicators from URLHaus feed format.
feeds/urlhaus.py
Python pipeline
Collection, normalization, enrichment, and orchestration code.
- Main · Coordinates feed collection, scoring, and persistence.
main.py - Prelim score · Initial normalization and preliminary scoring of indicators.
scoring/preliminary_scoring.py - MITRE parser · Parses collected activity into ATT&CK-relevant concepts.
scoring/mitre/mitre_parser.py - Technique map · Maps normalized activity to MITRE ATT&CK techniques.
scoring/mitre/technique_mapper.py - URLHaus JSON · Locally materialized feed output used before or alongside ingestion.
outputs/latest_urlhaus.json
Persistence
MongoDB storage and administrative database utilities.
- Mongo handler · Encapsulates MongoDB writes and reads for threat data.
database/mongo_handler.py - DB init · Initializes collections or storage state for the database layer.
database/init.py - Clear DB · Resets stored threat intelligence during maintenance.
clear_db.py - Conn check · Verifies database connectivity and runtime access.
database/test_connection.py
Runtime
Containerized local environment and downstream integration boundary.
- Compose · Defines the containerized local environment and service wiring.
docker-compose.yml - MongoDB · Durable system-of-record for collected threat intelligence.
- DB UI · Browser-based database inspection interface for stored data.
- Logstash · Optional downstream routing for structured events.
logstash/pipeline/logstash.conf
Connections 17
- AlienVault → Main: raw indicators · Feed content enters the orchestration layer.
- OpenPhish → Main: raw indicators · Feed content enters the orchestration layer.
- URLHaus → Main: raw indicators · Feed content enters the orchestration layer.
- Main → Prelim score: normalize · Orchestrates initial scoring and normalization.
- Prelim score → MITRE parser: enrich · Promotes scored indicators into structured activity.
- MITRE parser → Technique map: map techniques · Converts parsed activity into ATT&CK techniques.
- Technique map → Mongo handler: persist · Writes enriched threat records to storage.
- Main → Mongo handler: store · Coordinates database writes for collected intelligence.
- DB init → Mongo handler: bootstrap · Prepares the persistence layer for use.
- Clear DB → Mongo handler: reset · Administrative reset operates through the persistence layer.
- Conn check → Mongo handler: verify · Connectivity checks against MongoDB access.
- Mongo handler → MongoDB: writes · Persists threat intelligence in MongoDB.
- Compose → MongoDB: runs · Container definition provisions the database service.
- Compose → DB UI: exposes · Compose environment includes the browser-based DB UI.
- URLHaus JSON → URLHaus: staged feed · Represents URLHaus output materialized locally.
- Main → Logstash: export · Structured events can be forwarded into Logstash.
- Logstash → MongoDB: forward · Optional downstream integration path for structured records.