How sahilnegi4444/aml-transaction-intelligence-de- works
1. What this repo is
- A local, containerized AML analytics system centered on batch transaction processing, rule-based alerting, and LLM-backed alert explanations.
- The repo is organized around a data pipeline plus a lightweight serving layer, not a traditional monolithic app.
2. Primary runtime boundary
- Most moving parts run under Docker Compose: PostgreSQL, Spark, Redis, Ollama, MLflow, and the FastAPI service.
- The dashboard is a separate static frontend served independently, so the UI is not coupled to the API server process.
3. Data ingestion and cleanup path
- The raw input is a large CSV transaction dataset.
- scripts/clean_and_store.py is the ETL entrypoint: it uses PySpark + JDBC to clean transactions at scale and load them into PostgreSQL.
- This is the main batch-processing boundary between raw data and queryable operational storage.
4. Detection and alert generation
- scripts/spark_detect.py and the SQL schema/init script drive the anomaly detection layer.
- The README describes a rule-based detection rule: transaction count > 2 and total amount > $10,000, with dynamic risk scoring layered on top.
- The output of this stage is the alerts table, which the rest of the system consumes.
5. Persistence layer
- PostgreSQL 16 is the system of record for transactions, alerts, and likely regulatory/reference content.
- pgvector extends PostgreSQL for semantic search over regulations and related text chunks, enabling retrieval for explanations.
- scripts/init_db.sql defines the schema/bootstrap needed before analytics and API queries work.
6. RAG explanation path
- api/main.py is the FastAPI service that exposes the main operational API.
- It serves:
- /health
- /alerts?limit=N
- /explain_case/{id}
- The explanation endpoint is the key AI flow: it retrieves relevant context from vector search, then calls Ollama with llama3.1 to generate a regulatory-style explanation.
7. AI and retrieval dependencies
- The model stack is local and privacy-preserving: Ollama for inference, nomic-embed-text for embeddings, and pgvector for similarity retrieval.
- The README indicates LangChain is used to assemble the retrieval/generation pipeline.
- Redis sits alongside this path as a low-latency cache for repeated case lookups.
8. User-facing dashboard
- dashboard/index.html is a static web UI that consumes the API.
- Its main functions are alert search, sorting, risk filtering, charting, and triggering AI explanations.
- This is the analyst-facing presentation layer over the alert dataset, not a separate source of truth.
9. Experimentation and model tuning
- scripts/fine_tune_unsloth.py supports LoRA fine-tuning with Unsloth.
- scripts/interactive_analysis.py appears to be an offline comparison/analysis tool rather than production serving code.
- MLflow is used to track prompt versions and latency/experiment metrics, so model and prompt iteration are part of the architecture.
10. Operational/analytics tooling
- docs/setup_powerbi.md and the output/plots directory suggest downstream BI/reporting and generated artifacts.
- Spark UI, MLflow UI, and Power BI are supporting observability/analysis surfaces rather than core runtime dependencies.
11. Main end-to-end flow to diagram
- Raw CSV → PySpark cleaning/loading → PostgreSQL tables → SQL/PySpark detection → alerts stored in PostgreSQL.
- Analyst dashboard or API client → FastAPI → alerts query and/or RAG explanation path → PostgreSQL/pgvector + Redis + Ollama → returned explanation.
12. Key architectural boundaries
- Batch processing boundary: Spark jobs and SQL detection operate offline on large datasets.
- Serving boundary: FastAPI exposes read/explain endpoints over precomputed alerts.
- AI boundary: Retrieval and generation are local, decoupled from ETL, and rely on vectorized regulatory content.
Main entry points: main.py FastAPI · docker-compose.yaml Compose stack · clean_and_store.py Clean & load · requirements.txt Dependencies
Components 20
Batch processing
Offline ETL and detection pipeline for transaction data.
- Raw CSV · Large transaction input dataset processed in batch.
- Clean & load · PySpark/JDBC ETL entrypoint that cleans transactions and loads PostgreSQL.
scripts/clean_and_store.py - Detect rules · Batch anomaly detection and alert generation logic.
scripts/spark_detect.py - Schema init · Database schema and startup setup for transactions, alerts, and vector search.
scripts/init_db.sql - PostgreSQL · Primary store for transactions, alerts, and reference text.
Serving layer
FastAPI API and static dashboard for analysts.
AI retrieval
Local RAG and model tuning flow for case explanations.
- pgvector · Semantic retrieval over regulatory and explanatory text chunks.
- Redis · Low-latency cache for repeated case lookups.
- Ollama · Local inference runtime used to generate regulatory-style explanations.
- Embeddings · Local embedding model used for vectorization.
- Retrieval chain · Assembles retrieval, prompting, and generation for explanations.
- LoRA tuning · Offline fine-tuning workflow for experimentation and prompt/model iteration.
scripts/fine_tune_unsloth.py - Analysis tool · Interactive comparison and exploratory analysis utility.
scripts/interactive_analysis.py
Ops and artifacts
Containerization, experiment tracking, and downstream reporting.
- MLflow · Tracks prompt versions and latency/experiment metrics.
- Compose stack · Container boundary for PostgreSQL, Spark, Redis, Ollama, MLflow, and the API.
docker-compose.yaml - API image · Build definition for the FastAPI service container.
Dockerfile.api - Dependencies · Runtime and library dependencies for the repository.
requirements.txt - Power BI setup · Downstream BI/reporting integration notes.
docs/setup_powerbi.md - Generated plots · Saved analytical outputs and visual artifacts.
output/plots
Connections 24
- Raw CSV → Clean & load: raw data · Batch ingestion begins with the transaction CSV.
- Clean & load → PostgreSQL: load · Cleansed transactions are written to PostgreSQL.
- Schema init → PostgreSQL: bootstrap · Initial schema and tables are created in PostgreSQL.
- PostgreSQL → Detect rules: query · Detection job reads operational transaction data.
- Detect rules → PostgreSQL: alerts · Detected alerts are persisted back to PostgreSQL.
- PostgreSQL → FastAPI: read · API serves alert data and health status from the database.
- FastAPI → Dashboard: serve · Dashboard consumes the API for search, filters, and explanations.
- FastAPI → Redis: cache · Repeated case lookups are cached for low latency.
- FastAPI → Retrieval chain: explain · Case explanation endpoint delegates to the RAG pipeline.
- Retrieval chain → pgvector: retrieve · Relevant regulation and context chunks are fetched via vector search.
- Retrieval chain → Embeddings: embed · Queries and documents are embedded locally.
- Retrieval chain → Ollama: generate · LLM generates the final explanation text.
- pgvector → PostgreSQL: stored in · Vector data lives inside PostgreSQL via pgvector.
- LoRA tuning → MLflow: track · Fine-tuning runs and metrics are recorded.
- Analysis tool → MLflow: compare · Offline analysis can inspect experiment outputs and latency metrics.
- Compose stack → PostgreSQL: runs · Compose orchestrates the database service.
- Compose stack → Ollama: runs · Compose orchestrates the local model runtime.
- Compose stack → Redis: runs · Compose orchestrates the cache service.
- Compose stack → MLflow: runs · Compose orchestrates experiment tracking services.
- Compose stack → FastAPI: runs · Compose orchestrates the serving API container.
- API image → FastAPI: builds · Defines the API container image.
- Dependencies → Clean & load: deps · Shared Python dependencies support ETL and runtime code.
- Dependencies → FastAPI: deps · Shared Python dependencies support the API.
- Power BI setup → Generated plots: consumes · Downstream reporting can use generated analytics artifacts.