How youcef3939/godspeed works
1. Overall shape
- Godspeed is a layered retrieval pipeline: local text files are ingested, normalized and chunked, embedded into vectors, stored in a FAISS index, then queried through a retriever and exposed via a FastAPI service.
- The architecture is intentionally split so the same core pipeline can later swap in remote services or distributed infrastructure.
2. Primary runtime boundary
- The current runtime is a Python application.
- The API entrypoint is godspeed/api/main.py, which serves the query surface.
- Core search logic is not embedded in the API; it sits behind a SearchPipeline orchestration layer.
3. Input / ingestion subsystem
- godspeed/ingestion/text_ingestor.py reads a local corpus of .txt files from a directory.
- This is the repository’s current source boundary: raw text files come in, and document metadata such as source path is preserved for citation later.
4. Processing subsystem
- godspeed/processing/chunker.py is responsible for transforming ingested text into smaller semantic or fixed-size chunks.
- This stage is where text becomes indexable units, and chunk identity becomes important for retrieval and citations.
5. Embedding subsystem
- godspeed/embedding/sentence_transformer.py wraps a Sentence Transformers model.
- The README indicates the model is all-MiniLM-L6-v2, so the current implementation depends on local embedding generation rather than an external embedding API.
6. Storage / indexing subsystem
- godspeed/storage/faiss_store.py provides the vector index layer using FAISS.
- This is a local vector store, meaning retrieval is backed by in-process or local filesystem–style indexing rather than a managed database.
- The store is the key persistence/search boundary between offline indexing and online querying.
7. Retrieval subsystem
- godspeed/retrieval/retriever.py performs the query-time similarity search and ranking.
- It likely consumes the query embedding, searches FAISS, and returns scored chunk hits with associated metadata and citations.
- This is the main “answer selection” boundary in the system.
8. Orchestration layer
- godspeed/pipeline.py ties ingestion, chunking, embedding, storage, and retrieval into one end-to-end flow.
- This is the central coordination point and the best place to draw the system’s main data path: corpus → chunks → embeddings/index → query → ranked hits.
9. API layer
- godspeed/api/main.py exposes the pipeline over HTTP with at least GET /health and POST /query.
- The API returns structured, machine-readable responses with ranks, scores, chunk IDs, document IDs, and source citations.
- The API is a thin boundary: it should not own indexing logic, only request/response handling.
10. Domain and contracts
- godspeed/domain/models.py defines the core entities used across layers, such as documents, chunks, hits, and query/result shapes.
- godspeed/interfaces/contracts.py defines abstract contracts so ingestion, embedding, storage, and retrieval can be swapped without rewriting the pipeline.
- These two modules are the main abstraction boundary between “what the system means” and “how the current implementation works.”
11. Core utilities
- godspeed/core/logging.py centralizes logging behavior.
- This is infrastructure support rather than business logic, but it matters because the pipeline has multiple stages and benefits from consistent traceability.
12. Testing and validation
- tests/test_pipeline.py suggests the main verification focus is the integrated pipeline rather than isolated components only.
- That fits the architecture: the important system behavior is end-to-end retrieval quality and correct wiring between ingestion, indexing, and response formatting.
13. Key architectural constraint
- The repo currently favors a local-first implementation: local corpus, local embedding model, local FAISS index.
- The interfaces package shows the design already anticipates replacing those pieces with distributed or managed equivalents later, without changing the API or orchestration contract.
14. Diagram-friendly data flow
- A good architecture diagram would show one left-to-right path: local text corpus enters ingestion, moves through chunking and embedding, is indexed in FAISS, then query traffic enters the API, passes through the pipeline and retriever, and exits as ranked citation-backed results.
Main entry points: pipeline.py Pipeline · retriever.py Retriever · main.py API · sentence_transformer.py Embedder
Components 15
Runtime
Python application and HTTP entrypoint
- API · Serves health and query endpoints over HTTP
godspeed/api/main.py - Query · User search input handled by the API
- Results · Ranked hits with scores, chunk IDs, document IDs, and citations
Pipeline
End-to-end retrieval flow from corpus to ranked results
- Pipeline · Coordinates ingestion, embedding, storage, and retrieval
godspeed/pipeline.py - Ingestor · Reads local .txt corpus and preserves source metadata
godspeed/ingestion/text_ingestor.py - Chunker · Splits documents into retrieval-ready chunks
godspeed/processing/chunker.py - Embedder · Converts chunks and queries into vectors using a local model
godspeed/embedding/sentence_transformer.py - Vector store · Stores and searches dense vectors locally
godspeed/storage/faiss_store.py - Retriever · Ranks nearest chunks and returns scored hits with citations
godspeed/retrieval/retriever.py - Local corpus · Source directory of raw .txt documents
- Chunks · Chunked document segments ready for embedding and citation
Abstractions
Shared domain models and contracts
Connections 17
- Local corpus → Ingestor: load · Read local text documents and retain source metadata
- Ingestor → Chunker: normalize · Pass ingested documents into chunk preparation
- Chunker → Chunks: produce · Create chunked retrieval units
- Chunks → Embedder: encode · Turn chunks into vector embeddings
- Embedder → Vector store: index · Persist embeddings into the local FAISS store
- API → Pipeline: delegate · Forward query handling to the orchestration layer
- Pipeline → Retriever: query · Run retrieval through the search component
- Query → API: submit · Incoming HTTP query request
- Retriever → Embedder: embed query · Convert the search text into a query vector
- Retriever → Vector store: search · Perform nearest-neighbor lookup over the vector index
- Retriever → Models: shape hits · Materialize scored result objects using domain schemas
- Pipeline → Models: use · Share domain entities across stages
- Pipeline → Contracts: depend on · Keep pipeline components swappable behind abstractions
- API → Models: serialize · Return structured results and health responses
- Logging → Pipeline: trace · Provide cross-stage execution logging
- Tests → Pipeline: verify · Exercise the end-to-end orchestration
- Tests → API: verify · Check exposed query behavior