How 3gpp-chatbox/p2_backend works
Project type and purpose
• This is a backend‐only processing pipeline and RESTful API service (“3GPP Document Processor”) implemented in Python.
• It ingests .docx 3GPP specification files, preprocesses them, extracts procedure state machines, stores structured data in PostgreSQL, and exposes the results via FastAPI endpoints.
File structure
• Top-level directories under src/:
• API/ – FastAPI app, Pydantic models, REST routes
• retrieval/ – Document ingestion, chunking, context retrieval
• extraction/ – Multi-step state-machine extraction logic
• accuracy/ – Dataset comparison and validation modules
• db/ – Database handlers, schema init, migrations, models
• lib/ – Shared helpers (file utilities, logging, token counting)
• schemas/ – Pydantic/typing definitions for data validation
• data/ holds raw inputs, intermediate markdown outputs, mock data
• Configuration files: .env.example, pyproject.toml, requirements.txt, pytest.ini
• docs/ contains architecture notes and API instructions
• tests/ cover unit/integration tests per component
Key architectural insights from README
• Four main functional domains:
A. Document ingestion/preprocessing
B. Procedure extraction (4-step prompt chain)
C. Database persistence and query
D. API layer for external access
• External dependency: Google Gemini API for NLP
• Uses PostgreSQL for storage
• FastAPI + uvicorn for serving endpoints
1. Client / CLI (user triggers processing or calls API)
2. Document Processor (src/retrieval)
3. Prompt Chain Manager (src/extraction + src/lib/jsontomermaid)
4. Accuracy Validator (src/accuracy)
5. Database Service (PostgreSQL + src/db handlers)
6. API Server (FastAPI in src/API)
7. External Services: Google Gemini API
8. Data Storage: raw files, markdown outputs
• Client → Document Processor: submits .docx path
• Document Processor → Local File System: reads .docx, writes markdown to data/markdown
• Document Processor → Prompt Chain Manager: sends filtered content
• Prompt Chain Manager ↔ Google Gemini API: sends prompts, receives responses
• Prompt Chain Manager → Accuracy Validator: hands off extracted graphs and metrics
• Accuracy Validator → Database Service: writes validation results
• Prompt Chain Manager → Database Service: persists procedure graphs and metadata
• API Server → Database Service: read/write procedure data on GET/PUT
• Client (REST consumer) ↔ API Server: fetches or updates procedures
c. Architectural patterns and design principles
• Layered architecture: Ingestion layer, Extraction layer, Persistence layer, API layer
• Separation of concerns: retrieval, extraction, validation, API are decoupled
• Single Responsibility: each module (e.g., chunker, store_graphs, db_handler) has a narrow focus
• Configurable pipelines: prompt_manager orchestrates versions via src/prompts
• Test coverage: each layer has dedicated tests (tests/retrieval, tests/db, etc.)
d. Key technologies, frameworks, libraries
• Python 3.8+, FastAPI (uvicorn), Pydantic for data models
• PostgreSQL for storage, psycopg or SQLAlchemy in src/db
• Google Gemini API for NLP calls
• pytest for testing, pre-commit for linting