# 3gpp-chatbox/p2_backend architecture

> Architecture diagram of the GitHub repository 3gpp-chatbox/p2_backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/3gpp-chatbox/p2_backend
- Repository: https://github.com/3gpp-chatbox/p2_backend

## Overview

##### Project type and purpose
- This is a backend‐only processing pipeline and RESTful API service (“3GPP Document Processor”) implemented in Python.
- It ingests .docx 3GPP specification files, preprocesses them, extracts procedure state machines, stores structured data in PostgreSQL, and exposes the results via FastAPI endpoints.

##### File structure
- Top-level directories under src/:
- API/          – FastAPI app, Pydantic models, REST routes
- retrieval/    – Document ingestion, chunking, context retrieval
- extraction/   – Multi-step state-machine extraction logic
- accuracy/     – Dataset comparison and validation modules
- db/           – Database handlers, schema init, migrations, models
- lib/          – Shared helpers (file utilities, logging, token counting)
- schemas/      – Pydantic/typing definitions for data validation
- data/ holds raw inputs, intermediate markdown outputs, mock data
- Configuration files: .env.example, pyproject.toml, requirements.txt, pytest.ini
- docs/ contains architecture notes and API instructions
- tests/ cover unit/integration tests per component

##### Key architectural insights from README
- Four main functional domains:
A. Document ingestion/preprocessing
B. Procedure extraction (4-step prompt chain)
C. Database persistence and query
D. API layer for external access
- External dependency: Google Gemini API for NLP
- Uses PostgreSQL for storage
- FastAPI + uvicorn for serving endpoints

1. Client / CLI (user triggers processing or calls API)
2. Document Processor (src/retrieval)
3. Prompt Chain Manager (src/extraction + src/lib/jsontomermaid)
4. Accuracy Validator (src/accuracy)
5. Database Service (PostgreSQL + src/db handlers)
6. API Server (FastAPI in src/API)
7. External Services: Google Gemini API
8. Data Storage: raw files, markdown outputs

- Client → Document Processor: submits .docx path
- Document Processor → Local File System: reads .docx, writes markdown to data/markdown
- Document Processor → Prompt Chain Manager: sends filtered content
- Prompt Chain Manager ↔ Google Gemini API: sends prompts, receives responses
- Prompt Chain Manager → Accuracy Validator: hands off extracted graphs and metrics
- Accuracy Validator → Database Service: writes validation results
- Prompt Chain Manager → Database Service: persists procedure graphs and metadata
- API Server → Database Service: read/write procedure data on GET/PUT
- Client (REST consumer) ↔ API Server: fetches or updates procedures

c. Architectural patterns and design principles
- Layered architecture: Ingestion layer, Extraction layer, Persistence layer, API layer
- Separation of concerns: retrieval, extraction, validation, API are decoupled
- Single Responsibility: each module (e.g., chunker, store_graphs, db_handler) has a narrow focus
- Configurable pipelines: prompt_manager orchestrates versions via src/prompts
- Test coverage: each layer has dedicated tests (tests/retrieval, tests/db, etc.)

d. Key technologies, frameworks, libraries
- Python 3.8+, FastAPI (uvicorn), Pydantic for data models
- PostgreSQL for storage, psycopg or SQLAlchemy in src/db
- Google Gemini API for NLP calls
- pytest for testing, pre-commit for linting

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Client
    Client["Client / CLI"]:::external

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        DP["Document Processor"]:::ingest
        PP["Pre Processor"]:::ingest
        CH["Chunker"]:::ingest
        SC["Store Chunks"]:::ingest
        GC["Get Context"]:::ingest
        SCR["Sections Content Retrieval"]:::ingest
        TR["TOC Retrieval"]:::ingest
    end

    %% Extraction Layer
    subgraph "Extraction Layer"
        direction TB
        PCM["Prompt Chain Manager"]:::extract
        EP["Extract Procedure"]:::extract
        SG["Store Graphs"]:::extract
        PM["Prompt Manager"]:::extract
        PA["Active Prompts"]:::extract
        PD["Deprecated Prompts"]:::extract
    end

    %% Validation Layer
    subgraph "Validation Layer"
        direction TB
        CV["Compare Datasets"]:::validate
        SB["SBERT Simple"]:::validate
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        direction TB
        DB["PostgreSQL Database"]:::db
        DH["DB Handler"]:::db
        DA["Async DB Handler"]:::db
        DM["Document Model"]:::db
        IDB["Init DB & Migrations"]:::db
        DD["Dummy Data"]:::db
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        API["FastAPI Server"]:::api
        PMOD["Pydantic Models"]:::api
        FR["Fetch Routes"]:::api
        IR["Insert Routes"]:::api
    end

    %% Shared Libraries
    subgraph "Shared Libraries"
        direction TB
        FU["File Utils"]:::lib
        JM["jsontomermaid"]:::lib
        LG["Logger"]:::lib
        TC["Token Counter"]:::lib
    end

    %% Schemas
    subgraph "Schemas & Models"
        direction TB
        ST["Extraction Types"]:::schema
        SG2["Procedure Graph"]:::schema
        SV["Schema Validation"]:::schema
        CT["Context Types"]:::schema
        DT["Document Types"]:::schema
    end

    %% Data Storage
    subgraph "Data Storage"
        direction TB
        RAW["Raw Files"]:::storage
        MD["Markdown Outputs"]:::storage
        MOCK["Mock Pydantic Data"]:::storage
    end

    %% External Services
    subgraph "External Services"
        direction TB
        GCP["Google Gemini API"]:::external
    end

    %% Data Flows
    Client -->|"submit .docx path"| DP
    DP -->|"writes markdown"| MD
    DP -->|"reads .docx"| RAW
    DP -->|"filtered content"| PCM
    PCM -->|"prompts"| GCP
    GCP -->|"responses"| PCM
    PCM -->|"graphs & metrics"| CV
    CV -->|"validation results"| DH
    PCM -->|"procedure graphs & metadata"| DH
    API -->|"read/write data"| DH
    Client <-->|"REST API calls"| API

    %% Click Events
    click DP "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/retrieval/document_processor.py"
    click PP "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/retrieval/pre_processor.py"
    click CH "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/retrieval/chunker.py"
    click SC "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/retrieval/store_chunks.py"
    click GC "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/retrieval/get_context.py"
    click SCR "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/retrieval/sections_content_retrieval.py"
    click TR "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/retrieval/toc_retrieval.py"

    click PCM "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/extraction/prompt_chain.py"
    click EP "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/extraction/extract_procedure.py"
    click SG "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/extraction/store_graphs.py"
    click PM "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/prompts/prompt_manager.py"
    click PA "https://github.com/3gpp-chatbox/p2_backend/tree/main/src/prompts/active/"
    click PD "https://github.com/3gpp-chatbox/p2_backend/tree/main/src/prompts/deprecated/"

    click CV "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/accuracy/compare_datasets.py"
    click SB "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/accuracy/sbert_simple.py"

    click DH "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/db/db_handler.py"
    click DA "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/db/db_ahandler.py"
    click DM "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/db/document.py"
    click IDB "https://github.com/3gpp-chatbox/p2_backend/tree/main/src/db/migrations/"
    click DD "https://github.com/3gpp-chatbox/p2_backend/tree/main/src/db/dummy_data/"

    click API "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/API/main.py"
    click PMOD "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/API/pydantic_models.py"
    click FR "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/API/routes/fetch_routes.py"
    click IR "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/API/routes/insert_routes.py"

    click FU "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/lib/file_utils.py"
    click JM "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/lib/jsontomermaid.py"
    click LG "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/lib/logger.py"
    click TC "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/lib/token_counter.py"

    click ST "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/schemas/extraction_types.py"
    click SG2 "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/schemas/procedure_graph.py"
    click SV "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/schemas/schema_validation.py"
    click CT "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/schemas/types/context.py"
    click DT "https://github.com/3gpp-chatbox/p2_backend/blob/main/src/schemas/types/document.py"

    %% Styles
    classDef ingest fill:#d4f8d4,stroke:#2d862d,color:#000
    classDef extract fill:#ffe4b3,stroke:#b36b00,color:#000
    classDef validate fill:#fff2b3,stroke:#b38f00,color:#000
    classDef db fill:#b3d9ff,stroke:#0066b3,color:#000
    classDef api fill:#e3b3ff,stroke:#8000b3,color:#000
    classDef lib fill:#cccccc,stroke:#666666,color:#000
    classDef schema fill:#e6e6e6,stroke:#8c8c8c,color:#000
    classDef storage fill:#f2f2f2,stroke:#999999,color:#000
    classDef external fill:#ffb3b3,stroke:#b30000,color:#000
```
