How chrys/yasrl works
Project type and purpose
• yasrl is an open-source Python library (not a standalone web app) for building Retrieval-Augmented Generation (RAG) pipelines.
• It provides opinionated defaults, a configuration system, pluggable providers for embeddings/LLM/vector stores, pipelines for indexing and querying, and evaluation hooks.
File structure
• Top level:
• docs/ – design docs, configuration guides, project specs
• src/yasrl/ – core package
• tests/ – unit tests covering every module
• src/yasrl subpackages:
• config/ – manager.py (loads .env/config), models.py (pydantic models)
• loaders.py – ingestion of raw data
• text_processor.py – cleanup, tokenization, chunking
• vector_store.py – interface to in-memory or external (Postgres) stores
• providers/ – embeddings.py, llm.py (pluggable provider abstractions)
• query_processor.py – builds retrieval + generation workflow
• pipeline.py – orchestrates index vs. ask vs. complete pipelines
• evaluation/ – base.py (metrics), cli.py (evaluate via CLI)
• exceptions.py, models.py – domain models & errors
• config files: .env.example, pyproject.toml, requirements.txt
README for additional insights
• Confirms design: core RAG pipeline, optional extras for OpenAI/Gemini/Ollama, PostgreSQL, dev/test/docs.
a. Main components
1. Configuration Layer
– Config Manager (reads .env, YAML/JSON configs via pydantic models)
2. Data Ingestion & Processing
– Loader(s)
– Text Processor (preprocessing, chunking)
3. Vector Store
– In-memory (default) or external (Postgres)
4. Embeddings Provider
– Abstract interface; concrete clients for OpenAI, Gemini, Ollama
5. LLM Provider
– Abstract interface; concrete clients for OpenAI, Gemini, Ollama
6. Query Processor & Pipeline
– Retrieval + Generation orchestration
– Pipeline module selects index vs. ask vs. complete
7. Evaluation
– Base metrics & CLI runner
8. CLI / API (if used) for end-user interactions
9. External Services
– LLM APIs
– PostgreSQL (vector store)
– Environment (AWS SSM, Vault, etc. if used)
b. Relationships & interactions
1. Config Manager → provides settings (API keys, endpoints, model names)
2. Loader → Text Processor → Vector Store (indexing pipeline)
3. Query Processor ↔ Vector Store (retrieval)
4. Query Processor → Embeddings Provider (to embed queries/doc chunks)
5. Query Processor → LLM Provider (to complete or answer)
6. Pipeline module drives Data Ingestion or Query workflows end-to-end
7. Evaluation CLI invokes Pipeline & Query Processor, collects metrics
c. Architectural patterns & principles
• Pipeline pattern for ETL and inference
• Provider (Strategy) pattern for swapping embeddings/LLM/vector store backends
• Dependency injection via Config Manager and pydantic models
• Modular layered design: config → ingestion → processing → storage → inference → evaluation
d. Key technologies & libraries
• Python 3.8+
• pydantic (config models)
• AnyIO / click (CLI in evaluation/cli.py)
• HTTP clients (requests or aiohttp) for LLM providers
• SQLAlchemy or psycopg2 for Postgres vector store (optional)
• pytest for tests