# haaroonafroz/legal_compliance_agent architecture

> Architecture diagram of the GitHub repository haaroonafroz/legal_compliance_agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haaroonafroz/legal_compliance_agent
- Repository: https://github.com/haaroonafroz/legal_compliance_agent

## Overview

1. **Overall shape**
   - `legal-agent` is a Python 3.11+ compliance pipeline with two major stages: regulatory ingestion and compliance analysis.
   - The system is organized around a local CLI, a Qdrant-backed document store, a Scrapy crawler, and a LlamaIndex event-driven workflow.

2. **Primary runtime entrypoint**
   - `src/legal_agent/cli.py` is the main control surface.
   - It exposes operational commands such as database initialization, policy ingestion, scraping, workflow execution, and status checks.
   - `config.py` centralizes environment-backed settings, so runtime behavior is mostly driven by `.env` and external service credentials.

3. **Persistent data boundary**
   - Qdrant is the key persistence layer and retrieval index.
   - The system uses two collections:
     - `regulatory_updates` for scraped regulation chunks plus processing state
     - `internal_policies` for chunked internal policy text used during retrieval
   - `db/client.py` and `db/schemas.py` define the Qdrant connection and collection/index schema.

4. **Regulatory harvesting subsystem**
   - `src/legal_agent/scraping/` contains a Scrapy-based crawler.
   - `spiders/regulatory_spider.py` reads crawl targets from `data/sources/targets.json`.
   - The crawl path is designed to collect both HTML and PDF sources, then normalize them into chunked, metadata-rich records for storage.

5. **Document ingestion and chunking**
   - The scraping pipeline uses items and pipelines to move extracted content through chunking, enrichment, and upsert.
   - PDFs are converted through Docling before chunking, which matters because the architecture treats PDFs and HTML as separate extraction problems but a common downstream storage format.
   - `utils/loader.py` and `utils/models.py` support document parsing and structured representation.

6. **Policy ingestion path**
   - Internal policies are ingested separately from regulatory scraping, typically from PDFs in `data/policies/`.
   - This path produces embeddings and stores chunks in the `internal_policies` Qdrant collection.
   - Keeping policy ingestion separate from regulatory ingestion creates a clean boundary between “system of record” policy content and external regulatory updates.

7. **Compliance workflow subsystem**
   - `src/legal_agent/workflow/` implements the analysis phase as an explicit LlamaIndex workflow rather than a free-form agent loop.
   - `workflow.py` contains the main `ComplianceWorkflow`, while `events.py` defines typed step-to-step events.
   - The workflow steps are:
     - Horizon Scanner
     - Librarian
     - Relevance Check
     - Analyst
     - Redliner
     - Auditor
   - This stepwise design is the core architectural choice for traceability and evaluability.

8. **Retrieval and reasoning flow**
   - The workflow begins by grouping staged regulatory chunks back into complete documents.
   - It then retrieves relevant policy chunks from Qdrant using hybrid retrieval, combining semantic and lexical signals.
   - Retrieved context feeds the comparison step, which generates a gap analysis, then a redline-style policy draft, and finally an audit pass that can reject weak or unsupported output.

9. **Model routing and external AI services**
   - `workflow/llm_provider.py` and `workflow/prompts.py` encapsulate model selection and prompt design.
   - The repo supports OpenAI by default, with optional Gemini for workflow steps and VoyageAI for reranking.
   - This is architecturally important because different steps have different cost/quality requirements: cheaper models can be used for filtering, while stronger models are reserved for analysis, drafting, and auditing.

10. **Observability and evaluation**
   - `instrumentation/setup.py` wires trace/monitoring integrations such as Langfuse, Phoenix, and LangWatch.
   - The architecture is intentionally inspectable: intermediate artifacts are stored, workflow steps are explicit, and traces can be correlated with saved reports.
   - This makes the repo suitable for compliance-style evaluation where you need to audit retrieval quality, reasoning quality, and final output quality separately.

11. **Outputs and artifacts**
   - The workflow writes report artifacts to `data/reports/` as Markdown and JSON.
   - The final output is not just a generated answer; it is an auditable package containing the analysis, draft amendments, and auditor verdict.
   - That output boundary is important for diagramming: the pipeline ends in persisted reports, not in a live service response.

12. **Tests and validation scope**
   - `tests/test_chunking.py` and `tests/test_events.py` indicate lightweight validation around document segmentation and workflow event construction.
   - The test suite supports sanity checks, but the architecture relies more heavily on runtime traces and generated artifacts for meaningful evaluation.

13. **Key architectural flow to diagram**
   - Sources in `data/sources/targets.json` feed the Scrapy crawler.
   - Crawled content is extracted, converted, chunked, and stored in Qdrant.
   - The workflow pulls unprocessed regulatory documents, retrieves internal policy evidence, generates analysis and redlines, audits the result, and saves the report.
   - The main boundary crossings are: filesystem/config → crawler → Qdrant → workflow/LLM services → report artifacts.

Main entry points: [`src/legal_agent/workflow/workflow.py`](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/workflow.py) (Compliance workflow), [`src/legal_agent/cli.py`](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/cli.py) (CLI), [`src/legal_agent/db/client.py`](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/db/client.py) (Qdrant client), [`src/legal_agent/scraping/spiders/regulatory_spider.py`](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/scraping/spiders/regulatory_spider.py) (Regulatory spider)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control & config"]
  node_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_config["Config<br/>settings<br/>[config.py]"]
end

subgraph group_data["Data layer"]
  node_db_client[("Qdrant client<br/>persistence<br/>[client.py]")]
  node_db_schemas["Schemas<br/>index schema<br/>[schemas.py]"]
  node_reg_updates[("Regulatory store<br/>qdrant collection")]
  node_policy_store[("Policy store<br/>qdrant collection")]
end

subgraph group_ingest["Ingestion"]
  node_targets["Targets<br/>source list<br/>[targets.json]"]
  node_spider["Regulatory spider<br/>scrapy crawler"]
  node_scrape_pipe["Scrape pipeline<br/>scrapy pipeline<br/>[pipelines.py]"]
  node_scrape_items["Scrape items<br/>scrapy models<br/>[items.py]"]
  node_loader["Document loader<br/>document parsing<br/>[loader.py]"]
  node_models["Doc models<br/>document model<br/>[models.py]"]
end

subgraph group_workflow["Compliance workflow"]
  node_workflow_core["Compliance workflow<br/>llamaindex workflow<br/>[workflow.py]"]
  node_workflow_events["Workflow events<br/>typed events<br/>[events.py]"]
  node_llm_provider["Model router<br/>llm selection<br/>[llm_provider.py]"]
  node_prompts["Prompts<br/>prompt library<br/>[prompts.py]"]
end

subgraph group_ops["Observability & artifacts"]
  node_report["Reports<br/>artifact writer<br/>[report.py]"]
  node_instrumentation["Tracing<br/>observability setup<br/>[setup.py]"]
end

node_cli -->|"loads"| node_config
node_cli -->|"initializes"| node_db_client
node_cli -->|"runs"| node_spider
node_cli -->|"executes"| node_workflow_core
node_cli -->|"enables"| node_instrumentation
node_db_client -->|"uses"| node_db_schemas
node_db_client -->|"stores"| node_reg_updates
node_db_client -->|"stores"| node_policy_store
node_targets -->|"feeds"| node_spider
node_spider -->|"emits"| node_scrape_items
node_spider -->|"extracts"| node_loader
node_loader -->|"normalizes"| node_models
node_scrape_items -->|"processed by"| node_scrape_pipe
node_loader -->|"chunks"| node_scrape_pipe
node_scrape_pipe -->|"upserts"| node_reg_updates
node_workflow_core -->|"drives"| node_workflow_events
node_workflow_core -->|"routes models"| node_llm_provider
node_workflow_core -->|"uses"| node_prompts
node_workflow_core -->|"reads staged docs"| node_reg_updates
node_workflow_core -->|"retrieves evidence"| node_policy_store
node_workflow_core -->|"writes"| node_report
node_workflow_core -->|"traces"| node_instrumentation

click node_cli "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/cli.py"
click node_config "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/config.py"
click node_db_client "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/db/client.py"
click node_db_schemas "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/db/schemas.py"
click node_targets "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/data/sources/targets.json"
click node_spider "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/scraping/spiders/regulatory_spider.py"
click node_scrape_pipe "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/scraping/pipelines.py"
click node_scrape_items "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/scraping/items.py"
click node_loader "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/utils/loader.py"
click node_models "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/utils/models.py"
click node_workflow_core "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/workflow.py"
click node_workflow_events "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/events.py"
click node_llm_provider "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/llm_provider.py"
click node_prompts "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/prompts.py"
click node_report "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/utils/report.py"
click node_instrumentation "https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/instrumentation/setup.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config toneBlue
class node_db_client,node_db_schemas,node_reg_updates,node_policy_store toneAmber
class node_targets,node_spider,node_scrape_pipe,node_scrape_items,node_loader,node_models toneMint
class node_workflow_core,node_workflow_events,node_llm_provider,node_prompts toneRose
class node_report,node_instrumentation toneIndigo
```

## Components

### Control & config

CLI entrypoint and runtime settings.

- [CLI](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/cli.py): entrypoint · Main operational command surface. · `src/legal_agent/cli.py`
- [Config](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/config.py): settings · Environment-backed runtime settings. · `src/legal_agent/config.py`

### Data layer

Persistent collections and schema definitions.

- [Qdrant client](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/db/client.py): persistence · Connection and collection access layer. · `src/legal_agent/db/client.py`
- [Schemas](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/db/schemas.py): index schema · Collection and indexing definitions. · `src/legal_agent/db/schemas.py`
- **Regulatory store**: qdrant collection · Scraped regulation chunks and processing state.
- **Policy store**: qdrant collection · Chunked internal policy evidence for retrieval.

### Ingestion

Regulatory crawling and policy import paths.

- [Targets](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/data/sources/targets.json): source list · Crawl targets for regulatory harvesting. · `data/sources/targets.json`
- [Regulatory spider](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/scraping/spiders/regulatory_spider.py): scrapy crawler · Harvests HTML and PDF regulatory sources. · `src/legal_agent/scraping/spiders/regulatory_spider.py`
- [Scrape pipeline](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/scraping/pipelines.py): scrapy pipeline · Normalizes, chunks, enriches, and upserts crawl output. · `src/legal_agent/scraping/pipelines.py`
- [Scrape items](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/scraping/items.py): scrapy models · Structured crawl item payloads. · `src/legal_agent/scraping/items.py`
- [Document loader](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/utils/loader.py): document parsing · Helpers for parsing and extraction, including PDF handling. · `src/legal_agent/utils/loader.py`
- [Doc models](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/utils/models.py): document model · Structured document representations. · `src/legal_agent/utils/models.py`

### Compliance workflow

Event-driven analysis and drafting pipeline.

- [Compliance workflow](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/workflow.py): llamaindex workflow · Main event-driven analysis pipeline. · `src/legal_agent/workflow/workflow.py`
- [Workflow events](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/events.py): typed events · Step-to-step event definitions. · `src/legal_agent/workflow/events.py`
- [Model router](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/llm_provider.py): llm selection · Selects OpenAI, Gemini, and reranking services. · `src/legal_agent/workflow/llm_provider.py`
- [Prompts](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/workflow/prompts.py): prompt library · Step-specific prompts for scanning, analysis, and auditing. · `src/legal_agent/workflow/prompts.py`

### Observability & artifacts

Tracing and persisted reports.

- [Reports](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/utils/report.py): artifact writer · Persists Markdown and JSON audit artifacts. · `src/legal_agent/utils/report.py`
- [Tracing](https://github.com/haaroonafroz/legal_compliance_agent/blob/main/src/legal_agent/instrumentation/setup.py): observability setup · Langfuse, Phoenix, and LangWatch wiring. · `src/legal_agent/instrumentation/setup.py`

## Connections

- CLI → Config: loads · CLI uses environment-backed settings.
- CLI → Qdrant client: initializes · CLI manages database and persistence operations.
- CLI → Regulatory spider: runs · CLI triggers regulatory scraping commands.
- CLI → Compliance workflow: executes · CLI launches the compliance analysis workflow.
- CLI → Tracing: enables · CLI/runtime wiring includes tracing setup.
- Qdrant client → Schemas: uses · Client relies on collection and schema definitions.
- Qdrant client → Regulatory store: stores · Regulatory updates are written to Qdrant.
- Qdrant client → Policy store: stores · Internal policy chunks are written to Qdrant.
- Targets → Regulatory spider: feeds · Spider reads crawl targets from the source list.
- Regulatory spider → Scrape items: emits · Crawler emits structured extraction items.
- Regulatory spider → Document loader: extracts · Loader supports HTML/PDF document extraction.
- Document loader → Doc models: normalizes · Parsed content is shaped into document models.
- Scrape items → Scrape pipeline: processed by · Pipeline consumes crawl items for downstream handling.
- Document loader → Scrape pipeline: chunks · Pipeline uses parsing helpers for chunking and conversion.
- Scrape pipeline → Regulatory store: upserts · Chunked regulatory records are stored in the regulatory collection.
- Compliance workflow → Workflow events: drives · Workflow steps communicate through typed events.
- Compliance workflow → Model router: routes models · Workflow chooses models per step and cost tier.
- Compliance workflow → Prompts: uses · Workflow step behavior is prompt-driven.
- Compliance workflow → Regulatory store: reads staged docs · Workflow pulls unprocessed regulatory documents for analysis.
- Compliance workflow → Policy store: retrieves evidence · Workflow performs hybrid retrieval over policy chunks.
- Compliance workflow → Reports: writes · Final analysis, redlines, and verdict are persisted as reports.
- Compliance workflow → Tracing: traces · Workflow execution is instrumented for inspection.
