# sahilnegi4444/aml-transaction-intelligence-de- architecture

> Architecture diagram of the GitHub repository sahilnegi4444/aml-transaction-intelligence-de-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sahilnegi4444/aml-transaction-intelligence-de-
- Repository: https://github.com/sahilnegi4444/aml-transaction-intelligence-de-

## Overview

1. **What this repo is**
   - A local, containerized AML analytics system centered on **batch transaction processing**, **rule-based alerting**, and **LLM-backed alert explanations**.
   - The repo is organized around a data pipeline plus a lightweight serving layer, not a traditional monolithic app.

2. **Primary runtime boundary**
   - Most moving parts run under **Docker Compose**: PostgreSQL, Spark, Redis, Ollama, MLflow, and the FastAPI service.
   - The dashboard is a separate static frontend served independently, so the UI is not coupled to the API server process.

3. **Data ingestion and cleanup path**
   - The raw input is a large CSV transaction dataset.
   - `scripts/clean_and_store.py` is the ETL entrypoint: it uses **PySpark + JDBC** to clean transactions at scale and load them into PostgreSQL.
   - This is the main batch-processing boundary between raw data and queryable operational storage.

4. **Detection and alert generation**
   - `scripts/spark_detect.py` and the SQL schema/init script drive the anomaly detection layer.
   - The README describes a **rule-based detection rule**: transaction count > 2 and total amount > $10,000, with dynamic risk scoring layered on top.
   - The output of this stage is the alerts table, which the rest of the system consumes.

5. **Persistence layer**
   - **PostgreSQL 16** is the system of record for transactions, alerts, and likely regulatory/reference content.
   - **pgvector** extends PostgreSQL for semantic search over regulations and related text chunks, enabling retrieval for explanations.
   - `scripts/init_db.sql` defines the schema/bootstrap needed before analytics and API queries work.

6. **RAG explanation path**
   - `api/main.py` is the FastAPI service that exposes the main operational API.
   - It serves:
     - `/health`
     - `/alerts?limit=N`
     - `/explain_case/{id}`
   - The explanation endpoint is the key AI flow: it retrieves relevant context from vector search, then calls **Ollama** with **llama3.1** to generate a regulatory-style explanation.

7. **AI and retrieval dependencies**
   - The model stack is local and privacy-preserving: **Ollama** for inference, **nomic-embed-text** for embeddings, and **pgvector** for similarity retrieval.
   - The README indicates **LangChain** is used to assemble the retrieval/generation pipeline.
   - **Redis** sits alongside this path as a low-latency cache for repeated case lookups.

8. **User-facing dashboard**
   - `dashboard/index.html` is a static web UI that consumes the API.
   - Its main functions are alert search, sorting, risk filtering, charting, and triggering AI explanations.
   - This is the analyst-facing presentation layer over the alert dataset, not a separate source of truth.

9. **Experimentation and model tuning**
   - `scripts/fine_tune_unsloth.py` supports **LoRA fine-tuning** with Unsloth.
   - `scripts/interactive_analysis.py` appears to be an offline comparison/analysis tool rather than production serving code.
   - **MLflow** is used to track prompt versions and latency/experiment metrics, so model and prompt iteration are part of the architecture.

10. **Operational/analytics tooling**
   - `docs/setup_powerbi.md` and the `output/plots` directory suggest downstream BI/reporting and generated artifacts.
   - Spark UI, MLflow UI, and Power BI are supporting observability/analysis surfaces rather than core runtime dependencies.

11. **Main end-to-end flow to diagram**
   - Raw CSV → PySpark cleaning/loading → PostgreSQL tables → SQL/PySpark detection → alerts stored in PostgreSQL.
   - Analyst dashboard or API client → FastAPI → alerts query and/or RAG explanation path → PostgreSQL/pgvector + Redis + Ollama → returned explanation.

12. **Key architectural boundaries**
   - **Batch processing boundary:** Spark jobs and SQL detection operate offline on large datasets.
   - **Serving boundary:** FastAPI exposes read/explain endpoints over precomputed alerts.
   - **AI boundary:** Retrieval and generation are local, decoupled from ETL, and rely on vectorized regulatory content.

Main entry points: [`api/main.py`](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/api/main.py) (FastAPI), [`docker-compose.yaml`](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/docker-compose.yaml) (Compose stack), [`scripts/clean_and_store.py`](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/clean_and_store.py) (Clean & load), [`requirements.txt`](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/requirements.txt) (Dependencies)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_batch["Batch processing"]
  node_csv_input["Raw CSV<br/>dataset"]
  node_clean_store["Clean & load<br/>spark etl<br/>[clean_and_store.py]"]
  node_detect_rules["Detect rules<br/>spark/sql job<br/>[spark_detect.py]"]
  node_init_db["Schema init<br/>sql bootstrap<br/>[init_db.sql]"]
  node_postgres[("PostgreSQL<br/>system of record")]
end

subgraph group_serving["Serving layer"]
  node_api["FastAPI<br/>serving api<br/>[main.py]"]
  node_dashboard["Dashboard<br/>static ui<br/>[index.html]"]
end

subgraph group_ai["AI retrieval"]
  node_pgvector[("pgvector<br/>vector extension")]
  node_redis[("Redis<br/>cache")]
  node_ollama{{"Ollama<br/>llm runtime"}}
  node_embed_model{{"Embeddings<br/>nomic-embed-text"}}
  node_langchain["Retrieval chain<br/>rag orchestration"]
  node_tune["LoRA tuning<br/>unsloth script"]
  node_analysis["Analysis tool<br/>offline analytics"]
end

subgraph group_ops["Ops and artifacts"]
  node_mlflow[("MLflow<br/>experiment tracking")]
  node_compose["Compose stack<br/>deployment"]
  node_dockerfile["API image<br/>container build<br/>[Dockerfile.api]"]
  node_reqs["Dependencies<br/>python requirements<br/>[requirements.txt]"]
  node_powerbi["Power BI setup<br/>reporting docs<br/>[setup_powerbi.md]"]
  node_plots["Generated plots<br/>artifacts"]
end

node_csv_input -->|"raw data"| node_clean_store
node_clean_store -->|"load"| node_postgres
node_init_db -->|"bootstrap"| node_postgres
node_postgres -->|"query"| node_detect_rules
node_detect_rules -->|"alerts"| node_postgres
node_postgres -->|"read"| node_api
node_api -->|"serve"| node_dashboard
node_api -->|"cache"| node_redis
node_api -->|"explain"| node_langchain
node_langchain -->|"retrieve"| node_pgvector
node_langchain -->|"embed"| node_embed_model
node_langchain -->|"generate"| node_ollama
node_pgvector -->|"stored in"| node_postgres
node_tune -->|"track"| node_mlflow
node_analysis -->|"compare"| node_mlflow
node_compose -->|"runs"| node_postgres
node_compose -->|"runs"| node_ollama
node_compose -->|"runs"| node_redis
node_compose -->|"runs"| node_mlflow
node_compose -->|"runs"| node_api
node_dockerfile -->|"builds"| node_api
node_reqs -->|"deps"| node_clean_store
node_reqs -->|"deps"| node_api
node_powerbi -->|"consumes"| node_plots

click node_clean_store "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/clean_and_store.py"
click node_detect_rules "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/spark_detect.py"
click node_init_db "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/init_db.sql"
click node_api "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/api/main.py"
click node_dashboard "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/dashboard/index.html"
click node_tune "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/fine_tune_unsloth.py"
click node_analysis "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/interactive_analysis.py"
click node_compose "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/docker-compose.yaml"
click node_dockerfile "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/Dockerfile.api"
click node_reqs "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/requirements.txt"
click node_powerbi "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/docs/setup_powerbi.md"
click node_plots "https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/tree/main/output/plots"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_csv_input,node_clean_store,node_detect_rules,node_init_db,node_postgres toneBlue
class node_api,node_dashboard toneAmber
class node_pgvector,node_redis,node_ollama,node_embed_model,node_langchain,node_tune,node_analysis toneMint
class node_mlflow,node_compose,node_dockerfile,node_reqs,node_powerbi,node_plots toneRose
```

## Components

### Batch processing

Offline ETL and detection pipeline for transaction data.

- **Raw CSV**: dataset · Large transaction input dataset processed in batch.
- [Clean & load](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/clean_and_store.py): spark etl · PySpark/JDBC ETL entrypoint that cleans transactions and loads PostgreSQL. · `scripts/clean_and_store.py`
- [Detect rules](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/spark_detect.py): spark/sql job · Batch anomaly detection and alert generation logic. · `scripts/spark_detect.py`
- [Schema init](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/init_db.sql): sql bootstrap · Database schema and startup setup for transactions, alerts, and vector search. · `scripts/init_db.sql`
- **PostgreSQL**: system of record · Primary store for transactions, alerts, and reference text.

### Serving layer

FastAPI API and static dashboard for analysts.

- [FastAPI](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/api/main.py): serving api · Operational API exposing health, alerts, and case explanations. · `api/main.py`
- [Dashboard](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/dashboard/index.html): static ui · Analyst-facing frontend for alert browsing and explanations. · `dashboard/index.html`

### AI retrieval

Local RAG and model tuning flow for case explanations.

- **pgvector**: vector extension · Semantic retrieval over regulatory and explanatory text chunks.
- **Redis**: cache · Low-latency cache for repeated case lookups.
- **Ollama**: llm runtime · Local inference runtime used to generate regulatory-style explanations.
- **Embeddings**: nomic-embed-text · Local embedding model used for vectorization.
- **Retrieval chain**: rag orchestration · Assembles retrieval, prompting, and generation for explanations.
- [LoRA tuning](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/fine_tune_unsloth.py): unsloth script · Offline fine-tuning workflow for experimentation and prompt/model iteration. · `scripts/fine_tune_unsloth.py`
- [Analysis tool](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/scripts/interactive_analysis.py): offline analytics · Interactive comparison and exploratory analysis utility. · `scripts/interactive_analysis.py`

### Ops and artifacts

Containerization, experiment tracking, and downstream reporting.

- **MLflow**: experiment tracking · Tracks prompt versions and latency/experiment metrics.
- [Compose stack](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/docker-compose.yaml): deployment · Container boundary for PostgreSQL, Spark, Redis, Ollama, MLflow, and the API. · `docker-compose.yaml`
- [API image](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/Dockerfile.api): container build · Build definition for the FastAPI service container. · `Dockerfile.api`
- [Dependencies](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/requirements.txt): python requirements · Runtime and library dependencies for the repository. · `requirements.txt`
- [Power BI setup](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/blob/main/docs/setup_powerbi.md): reporting docs · Downstream BI/reporting integration notes. · `docs/setup_powerbi.md`
- [Generated plots](https://github.com/sahilnegi4444/aml-transaction-intelligence-de-/tree/main/output/plots): artifacts · Saved analytical outputs and visual artifacts. · `output/plots`

## Connections

- Raw CSV → Clean & load: raw data · Batch ingestion begins with the transaction CSV.
- Clean & load → PostgreSQL: load · Cleansed transactions are written to PostgreSQL.
- Schema init → PostgreSQL: bootstrap · Initial schema and tables are created in PostgreSQL.
- PostgreSQL → Detect rules: query · Detection job reads operational transaction data.
- Detect rules → PostgreSQL: alerts · Detected alerts are persisted back to PostgreSQL.
- PostgreSQL → FastAPI: read · API serves alert data and health status from the database.
- FastAPI → Dashboard: serve · Dashboard consumes the API for search, filters, and explanations.
- FastAPI → Redis: cache · Repeated case lookups are cached for low latency.
- FastAPI → Retrieval chain: explain · Case explanation endpoint delegates to the RAG pipeline.
- Retrieval chain → pgvector: retrieve · Relevant regulation and context chunks are fetched via vector search.
- Retrieval chain → Embeddings: embed · Queries and documents are embedded locally.
- Retrieval chain → Ollama: generate · LLM generates the final explanation text.
- pgvector → PostgreSQL: stored in · Vector data lives inside PostgreSQL via pgvector.
- LoRA tuning → MLflow: track · Fine-tuning runs and metrics are recorded.
- Analysis tool → MLflow: compare · Offline analysis can inspect experiment outputs and latency metrics.
- Compose stack → PostgreSQL: runs · Compose orchestrates the database service.
- Compose stack → Ollama: runs · Compose orchestrates the local model runtime.
- Compose stack → Redis: runs · Compose orchestrates the cache service.
- Compose stack → MLflow: runs · Compose orchestrates experiment tracking services.
- Compose stack → FastAPI: runs · Compose orchestrates the serving API container.
- API image → FastAPI: builds · Defines the API container image.
- Dependencies → Clean & load: deps · Shared Python dependencies support ETL and runtime code.
- Dependencies → FastAPI: deps · Shared Python dependencies support the API.
- Power BI setup → Generated plots: consumes · Downstream reporting can use generated analytics artifacts.
