# reethusugan-cpu/advanced_threat_intelligence_platform architecture

> Architecture diagram of the GitHub repository reethusugan-cpu/advanced_threat_intelligence_platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/reethusugan-cpu/advanced_threat_intelligence_platform
- Repository: https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform

## Overview

1. **System shape**
   - This repo is a Python-based threat intelligence ingestion and storage pipeline, not a traditional web app.
   - The core runtime talks to external threat feeds, normalizes/scoring them, and persists results in MongoDB for later inspection through a Dockerized DB UI.

2. **Primary entrypoint**
   - `main.py` is the orchestration layer.
   - It likely coordinates feed collection, scoring, and database writes rather than implementing business logic itself.

3. **Threat feed adapters**
   - `feeds/alienvault.py`, `feeds/openphish.py`, and `feeds/urlhaus.py` are source-specific collectors.
   - Each module represents a boundary to an external intelligence source and is responsible for fetching raw indicators in that source’s format.

4. **Normalization and scoring**
   - `scoring/preliminary_scoring.py` is the first analysis stage for incoming indicators.
   - The `scoring/mitre/` package (`mitre_parser.py`, `technique_mapper.py`) maps collected activity or indicators to MITRE ATT&CK technique concepts, creating a standardized enrichment layer above raw feed data.

5. **Database layer**
   - `database/mongo_handler.py` is the persistence abstraction for MongoDB operations.
   - `database/init.py` likely handles initial setup/seed/schema-like bootstrapping for collections, while `database/test_connection.py` provides a connectivity sanity check.
   - `clear_db.py` is an administrative utility for wiping stored threat data during resets.

6. **Data flow**
   - External feeds → feed modules → scoring/enrichment modules → MongoDB.
   - The repository also includes `outputs/latest_urlhaus.json`, which suggests at least one feed can be materialized to local JSON before or alongside database ingestion.

7. **Infrastructure boundary**
   - `docker-compose.yml` defines the local runtime environment.
   - The README shows the main infrastructure is MongoDB plus a browser-based DB UI exposed on port `8081`, with MongoDB on `27017`.
   - The app code can connect either via `localhost` from the host machine or the Docker service name `atip-mongodb` from inside the container network.

8. **Log/ETL boundary**
   - `logstash/pipeline/logstash.conf` indicates a downstream ingestion/forwarding path for structured events.
   - In architecture terms, this is a secondary integration boundary: Python produces or stages data, and Logstash can route it onward if the stack is extended.

9. **Configuration and secrets**
   - `.env.example` and the README’s environment variable guidance show the project expects secrets and connection details to be injected, not hardcoded.
   - This matters architecturally because feed API keys and MongoDB credentials are treated as deployment-time configuration.

10. **Operational dependencies**
   - The stack is intentionally container-first: Docker and Docker Compose are part of the runtime architecture, not just setup tooling.
   - The MongoDB container is stateful via volumes, so the database is the durable system-of-record for collected threat intelligence.

11. **Repository intent**
   - The codebase is organized around a few clear layers: source collection, enrichment/scoring, persistence, and local infra.
   - There is no separate application server, message broker, or front-end code in this repo; the DB UI is an operational aid rather than a custom product UI.

12. **Diagram focus**
   - An accurate architecture diagram should emphasize external feed sources on the left, Python ingestion/scoring in the center, MongoDB storage and its UI on the right, and Docker Compose as the environment wrapper around the persistence layer.

Main entry points: [`main.py`](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/main.py) (Main), [`database/mongo_handler.py`](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/database/mongo_handler.py) (Mongo handler), [`docker-compose.yml`](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/docker-compose.yml) (Compose), [`feeds/urlhaus.py`](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/feeds/urlhaus.py) (URLHaus)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["Threat feeds"]
  node_alienvault["AlienVault<br/>feed adapter<br/>[alienvault.py]"]
  node_openphish["OpenPhish<br/>feed adapter<br/>[openphish.py]"]
  node_urlhaus["URLHaus<br/>feed adapter<br/>[urlhaus.py]"]
end

subgraph group_pipeline["Python pipeline"]
  node_main["Main<br/>orchestrator<br/>[main.py]"]
  node_prelim_score(("Prelim score<br/>scoring stage"))
  node_mitre_parser["MITRE parser<br/>enrichment<br/>[mitre_parser.py]"]
  node_technique_mapper["Technique map<br/>enrichment"]
  node_latest_urlhaus["URLHaus JSON<br/>staged output"]
end

subgraph group_storage["Persistence"]
  node_mongo_handler[("Mongo handler<br/>persistence adapter<br/>[mongo_handler.py]")]
  node_db_init["DB init<br/>bootstrap<br/>[init.py]"]
  node_clear_db["Clear DB<br/>admin utility<br/>[clear_db.py]"]
  node_db_check["Conn check<br/>sanity check<br/>[test_connection.py]"]
end

subgraph group_infra["Runtime"]
  node_docker_compose["Compose<br/>runtime config<br/>[docker-compose.yml]"]
  node_mongodb[("MongoDB<br/>datastore")]
  node_mongo_ui["DB UI<br/>ops interface"]
  node_logstash_conf["Logstash<br/>integration pipeline<br/>[logstash.conf]"]
end

node_alienvault -->|"raw indicators"| node_main
node_openphish -->|"raw indicators"| node_main
node_urlhaus -->|"raw indicators"| node_main
node_main -->|"normalize"| node_prelim_score
node_prelim_score -->|"enrich"| node_mitre_parser
node_mitre_parser -->|"map techniques"| node_technique_mapper
node_technique_mapper -->|"persist"| node_mongo_handler
node_main -->|"store"| node_mongo_handler
node_db_init -->|"bootstrap"| node_mongo_handler
node_clear_db -->|"reset"| node_mongo_handler
node_db_check -->|"verify"| node_mongo_handler
node_mongo_handler -->|"writes"| node_mongodb
node_docker_compose -->|"runs"| node_mongodb
node_docker_compose -->|"exposes"| node_mongo_ui
node_latest_urlhaus -.->|"staged feed"| node_urlhaus
node_main -.->|"export"| node_logstash_conf
node_logstash_conf -.->|"forward"| node_mongodb

click node_main "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/main.py"
click node_alienvault "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/feeds/alienvault.py"
click node_openphish "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/feeds/openphish.py"
click node_urlhaus "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/feeds/urlhaus.py"
click node_prelim_score "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/scoring/preliminary_scoring.py"
click node_mitre_parser "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/scoring/mitre/mitre_parser.py"
click node_technique_mapper "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/scoring/mitre/technique_mapper.py"
click node_mongo_handler "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/database/mongo_handler.py"
click node_db_init "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/database/init.py"
click node_clear_db "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/clear_db.py"
click node_db_check "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/database/test_connection.py"
click node_latest_urlhaus "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/outputs/latest_urlhaus.json"
click node_docker_compose "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/docker-compose.yml"
click node_logstash_conf "https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/logstash/pipeline/logstash.conf"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_alienvault,node_openphish,node_urlhaus toneBlue
class node_main,node_prelim_score,node_mitre_parser,node_technique_mapper,node_latest_urlhaus toneAmber
class node_mongo_handler,node_db_init,node_clear_db,node_db_check toneMint
class node_docker_compose,node_mongodb,node_mongo_ui,node_logstash_conf toneRose
```

## Components

### Threat feeds

External intelligence sources and their repository adapters.

- [AlienVault](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/feeds/alienvault.py): feed adapter · Collects indicators from AlienVault feed format. · `feeds/alienvault.py`
- [OpenPhish](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/feeds/openphish.py): feed adapter · Collects indicators from OpenPhish feed format. · `feeds/openphish.py`
- [URLHaus](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/feeds/urlhaus.py): feed adapter · Collects indicators from URLHaus feed format. · `feeds/urlhaus.py`

### Python pipeline

Collection, normalization, enrichment, and orchestration code.

- [Main](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/main.py): orchestrator · Coordinates feed collection, scoring, and persistence. · `main.py`
- [Prelim score](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/scoring/preliminary_scoring.py): scoring stage · Initial normalization and preliminary scoring of indicators. · `scoring/preliminary_scoring.py`
- [MITRE parser](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/scoring/mitre/mitre_parser.py): enrichment · Parses collected activity into ATT&CK-relevant concepts. · `scoring/mitre/mitre_parser.py`
- [Technique map](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/scoring/mitre/technique_mapper.py): enrichment · Maps normalized activity to MITRE ATT&CK techniques. · `scoring/mitre/technique_mapper.py`
- [URLHaus JSON](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/outputs/latest_urlhaus.json): staged output · Locally materialized feed output used before or alongside ingestion. · `outputs/latest_urlhaus.json`

### Persistence

MongoDB storage and administrative database utilities.

- [Mongo handler](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/database/mongo_handler.py): persistence adapter · Encapsulates MongoDB writes and reads for threat data. · `database/mongo_handler.py`
- [DB init](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/database/init.py): bootstrap · Initializes collections or storage state for the database layer. · `database/init.py`
- [Clear DB](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/clear_db.py): admin utility · Resets stored threat intelligence during maintenance. · `clear_db.py`
- [Conn check](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/database/test_connection.py): sanity check · Verifies database connectivity and runtime access. · `database/test_connection.py`

### Runtime

Containerized local environment and downstream integration boundary.

- [Compose](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/docker-compose.yml): runtime config · Defines the containerized local environment and service wiring. · `docker-compose.yml`
- **MongoDB**: datastore · Durable system-of-record for collected threat intelligence.
- **DB UI**: ops interface · Browser-based database inspection interface for stored data.
- [Logstash](https://github.com/reethusugan-cpu/advanced_threat_intelligence_platform/blob/main/logstash/pipeline/logstash.conf): integration pipeline · Optional downstream routing for structured events. · `logstash/pipeline/logstash.conf`

## Connections

- AlienVault → Main: raw indicators · Feed content enters the orchestration layer.
- OpenPhish → Main: raw indicators · Feed content enters the orchestration layer.
- URLHaus → Main: raw indicators · Feed content enters the orchestration layer.
- Main → Prelim score: normalize · Orchestrates initial scoring and normalization.
- Prelim score → MITRE parser: enrich · Promotes scored indicators into structured activity.
- MITRE parser → Technique map: map techniques · Converts parsed activity into ATT&CK techniques.
- Technique map → Mongo handler: persist · Writes enriched threat records to storage.
- Main → Mongo handler: store · Coordinates database writes for collected intelligence.
- DB init → Mongo handler: bootstrap · Prepares the persistence layer for use.
- Clear DB → Mongo handler: reset · Administrative reset operates through the persistence layer.
- Conn check → Mongo handler: verify · Connectivity checks against MongoDB access.
- Mongo handler → MongoDB: writes · Persists threat intelligence in MongoDB.
- Compose → MongoDB: runs · Container definition provisions the database service.
- Compose → DB UI: exposes · Compose environment includes the browser-based DB UI.
- URLHaus JSON → URLHaus: staged feed · Represents URLHaus output materialized locally.
- Main → Logstash: export · Structured events can be forwarded into Logstash.
- Logstash → MongoDB: forward · Optional downstream integration path for structured records.
