How asselk1/ows-assignment works
This repository implements a domain-specific procurement AI system with two main faces:
1. an online serving layer exposed as an API and chat client, and
2. an offline data pipeline that ingests procurement data, validates it, stores it, and builds retrieval/analytics assets.
The architecture is centered on a relational database, a vector store for retrieval, and an LLM-backed agent that combines analytics, classification, and explanation generation.
Core subsystems
• API layer: src/api
- server.py is the FastAPI application entrypoint.
- endpoints.py defines the HTTP routes.
- models.py defines request/response schemas.
- ows_client.py is the outbound integration point to the external procurement data source or service.
- This layer is the public boundary for both programmatic clients and the chat UI.
• Chat client: chat.py
- A local CLI client that talks to the API.
- It is a thin presentation layer; it does not contain business logic.
• Agent layer: src/agent
- classifier.py likely routes or classifies the user query into intent/domain buckets.
- response_generator.py produces the final user-facing answer.
- explainability.py adds rationale, provenance, or why-an-answer framing.
- This layer sits between the API and the underlying analytics/RAG/LLM components.
• Analytics layer: src/analytics
- statistics.py and queries.py support database-driven analytics.
- anomaly_price.py, anomaly_volume.py, and anomaly_fairness.py implement domain-specific anomaly detection.
- This layer is where procurement-specific insights are computed from stored data.
- It is the main structured-data reasoning subsystem.
• RAG layer: src/rag
- vector_store.py manages the embedding-backed index.
- retriever.py performs retrieval over indexed content.
- This layer supports semantic lookup over contract-related material and is distinct from the SQL analytics path.
• LLM layer: src/llm
- client.py wraps access to the language model provider.
- The rest of the code likely treats this as an abstraction so the agent can remain model-agnostic.
• ETL layer: src/etl
- etl_plans.py, etl_subjects.py, etl_announcements.py, etl_lots.py, etl_contracts.py, and etl_contract_acts.py correspond to domain entities in the procurement lifecycle.
- load_ref_data.py handles reference/master data.
- This layer is the batch ingestion boundary from the external procurement system into the local database.
• Data quality layer: src/quality
- data_quality.py likely validates completeness, consistency, and other dataset integrity checks before data is used for analytics or retrieval.
- This is a guardrail between ingestion and downstream consumption.
• Scheduler / refresh layer: src/scheduler
- refresh.py suggests periodic rebuilds or synchronization jobs.
- This is the automation boundary that keeps the database and vector store current.
• Database layer:
- src/database/create_tables.sql defines the relational schema.
- initialize_db.py bootstraps the database.
- The repo structure indicates SQL-backed persistence, with the ETL and analytics code built around it.
• Maintenance / bootstrapping scripts:
- run_etl.py likely orchestrates the full ingestion pipeline.
- index_all_contracts.py likely rebuilds the retrieval index from stored contract data.
- initialize_db.py creates or prepares the database schema.
- These are offline operational entrypoints rather than user-facing features.
Main data flow
1. External procurement data source
- The system pulls data from an external procurement API/service through src/api/ows_client.py.
- The ETL code knows the domain entities and transforms raw source records into normalized local tables.
2. Ingestion and storage
- run_etl.py or the ETL modules populate the relational database defined by create_tables.sql.
- Reference data is loaded first, then transactional entities such as plans, announcements, lots, contracts, and contract acts.
3. Data validation
- src/quality/data_quality.py checks the loaded data.
- This is an important boundary because analytics and retrieval depend on the quality of the stored records.
4. Derived assets
- Analytics modules read from the database to compute statistics and anomaly signals.
- The RAG subsystem builds or refreshes a vector store from contract-related text/content using index_all_contracts.py and the retriever/vector store modules.
- These are separate derived representations of the same domain data: one structured, one semantic.
5. Online query handling
- A user query enters through chat.py or an HTTP client into the FastAPI server.
- The API forwards the request into the agent layer.
- The classifier decides whether the request should be answered using analytics, retrieval, general LLM generation, or a combination.
- The response generator composes the final answer, possibly enriched with explainability details and retrieved evidence.
Important boundaries
• Online vs offline
- Offline: ETL, DB initialization, index building, refresh jobs, data quality.
- Online: API server, agent orchestration, chat client, query-time retrieval and analytics.
- This separation is central to the design.
• Structured analytics vs semantic retrieval
- Analytics modules use SQL/data statistics for precise procurement metrics and anomaly detection.
- RAG uses embeddings/vector search for unstructured or semi-structured contract knowledge.
- The agent can choose between them depending on the query.
• Domain integration vs presentation
- ows_client.py and the ETL modules interface with external procurement data.
- chat.py and src/api/server.py are presentation/serving interfaces.
- The agent layer bridges these worlds.
Likely technologies and operational shape
• Python is the primary runtime.
• FastAPI/Uvicorn is used for the API server.
• SQL is used for persistence and analytics access.
• A vector store is used for retrieval-augmented generation.
• An external LLM is accessed through a wrapper client.
• Docker is supported through docker-compose.yml and docker/Dockerfile.api, indicating containerized deployment of the API stack.
• config.yaml likely centralizes environment configuration such as endpoints, credentials, database settings, and model/vector store parameters.
How to think about the architecture diagram
A good diagram for this repo would show:
• On the left, an external procurement data source feeding the ETL pipeline.
• In the middle, a relational database as the system of record.
• Nearby, a data-quality gate and a scheduler/refresh process that keep the database and retrieval index current.
• A separate vector store fed from contract data for semantic retrieval.
• On the right, the API server receiving requests from the chat client or other consumers.
• Inside the API, an agent layer that routes queries to:
- analytics modules for statistics and anomaly detection,
- retriever/vector store for semantic lookup,
- the LLM client for generation and explanation.
• The final response returns through the API to the chat client.
In short, this is not just a chatbot repo. It is a procurement intelligence system with a clear split between ingestion, validation, structured analytics, retrieval, and LLM-driven response generation.
Main entry points: create_tables.sql Relational DB · classifier.py Classifier · run_etl.py ETL pipeline · endpoints.py API routes
Components 22
Online serving
API, chat, and query-time orchestration.
- Chat client · Thin local client for sending user queries to the API.
chat.py - API server · HTTP serving entrypoint for programmatic and chat-based queries.
src/api/server.py - API routes · Defines request handling and public endpoints.
src/api/endpoints.py - API models · Request and response shapes for the public API.
src/api/models.py - OWS client · Outbound integration to the external procurement source.
src/api/ows_client.py - Classifier · Routes queries to analytics, retrieval, or generation paths.
src/agent/classifier.py - Response gen · Builds the final user-facing response.
src/agent/response_generator.py - Explainability · Adds provenance and explanation framing to answers.
src/agent/explainability.py
Offline pipeline
Ingestion, validation, indexing, and refresh jobs.
- ETL pipeline · Transforms and loads procurement entities into the database.
run_etl.py - Domain ETL · Loads plans, subjects, announcements, lots, contracts, and acts.
src/etl/etl_contracts.py - Ref data · Loads reference and lookup data needed by downstream processing.
src/etl/load_ref_data.py - Data quality · Checks completeness and consistency before analytics and retrieval.
src/quality/data_quality.py - Refresh job · Keeps database and derived assets current.
src/scheduler/refresh.py - Index build · Rebuilds the retrieval index from stored contract data.
index_all_contracts.py - DB bootstrap · Initializes or prepares the database schema.
initialize_db.py
Core systems
Shared persistence, retrieval, and model access.
- LLM client · Abstracts access to the external language model provider.
src/llm/client.py - Statistics · Database-driven procurement metrics and queries.
src/analytics/statistics.py - Anomalies · Price, volume, and fairness anomaly detection.
src/analytics/anomaly_price.py - Retriever · Fetches relevant contract content from the vector index.
src/rag/retriever.py - Vector store · Backs semantic retrieval over contract-related text.
src/rag/vector_store.py - Relational DB · Primary structured store for normalized procurement data.
src/database/create_tables.sql
Other components
- Procurement source · Upstream procurement API or service providing raw domain records.
Connections 25
- Chat client → API server: queries · Chat UI sends user requests to the API.
- API server → API routes: dispatches · Server exposes the HTTP routes.
- API routes → API models: validates · Routes use request and response schemas.
- API routes → Classifier: hands off · Incoming requests are forwarded to the agent.
- Classifier → Statistics: routes to · Structured questions are answered with SQL analytics.
- Classifier → Anomalies: routes to · Anomaly-oriented questions use domain detectors.
- Classifier → Retriever: routes to · Semantic lookup requests go to retrieval.
- Classifier → Response gen: routes to · General or combined answers are composed downstream.
- Response gen → Explainability: enriches · Final answers can include rationale and provenance.
- Response gen → LLM client: generates with · The agent uses the language model for text generation.
- Retriever → Vector store: reads · Retriever queries the embedding-backed index.
- Statistics → Relational DB: reads · Analytics run against the relational store.
- Anomalies → Relational DB: reads · Anomaly detection uses structured stored data.
- ETL pipeline → Ref data: starts with · Reference data is loaded before transactional entities.
- ETL pipeline → Domain ETL: loads · Domain entity loaders populate procurement tables.
- Domain ETL → Relational DB: writes · Normalized records are persisted to the database.
- Ref data → Relational DB: writes · Master data is stored in the database.
- Data quality → Relational DB: checks · Validation runs over loaded data.
- Refresh job → Relational DB: syncs · Refresh jobs keep the database current.
- Refresh job → Vector store: rebuilds · Refresh jobs also update retrieval assets.
- Index build → Vector store: builds · Offline indexing populates the vector store.
- Index build → Relational DB: reads · Indexing pulls source content from stored contracts.
- DB bootstrap → Relational DB: initializes · Bootstrap creates or prepares the schema.
- OWS client → Procurement source: connects to · Outbound adapter reaches the upstream procurement source.
- ETL pipeline → OWS client: ingests from · ETL retrieves raw source data through the adapter.