How dermatologist/pyomop works
Main Components
• Library Core: Represented by the modules under src/pyomop. This includes components like:
• Engine Factory (engine_factory.py): Responsible for database connections.
• CDM Models & Tables (cdm6_tables.py): These define the structure of the OMOP CDM.
• Query Components: Such as CdmVocabulary, CdmVector, llm_engine, and llm_query, which handle vocabulary processing, conversion to pandas dataframes, and LLM-driven query processing.
• Main Entrypoints (e.g., __main__.py, main.py): Offering both command-line execution and script-based usage.
• Database Interaction Layer: Highlight that the project utilizes SQLAlchemy for ORM, which manages sessions and connections across supported databases (SQLite, PostgreSQL, MySQL, etc.).
• Data Conversion & Query Handling: This includes the conversion of query results to pandas dataframes as well as support for custom SQL queries (via sqldict.py).
• LLM Integration: Include the newer module components (llm_engine.py and llm_query.py) for processing natural language queries through an LLM. Indicate that these are optional extensions enabled by installing additional extras.
Out the Relationships and Interactions
• That customers or end users interact primarily with a command-line interface (CLI usage via main.py and pyomop command-line help) or by importing the library in their own scripts.
• Flow:
• The user instantiates a CdmEngineFactory that sets up the SQLAlchemy engine.
• They then create a session (possibly asynchronously) to interact with the database via ORM models defined in the project.
• Queries are executed either synchronously (for older versions) or asynchronously (for version 4.0.0 and later), and results can be wrapped or converted using modules like CdmVector.
• For vocabulary and domain-specific querying logic, the library uses components from CdmVocabulary and sqldict for predefined query templates.
• Optionally, if LLM capabilities are enabled and installed, the llm_engine and llm_query modules interact with external LLM services to transform natural language queries into SQL queries.
• How error handling, connection disposal, and session management are integral parts of the Engine Factory’s workflow.
Architectural Patterns and Design Principles
• Layered Architecture: Clearly separate layers such as the API/client layer (CLI/examples), the domain logic (engine and model definitions), and the data access layer (SQLAlchemy ORM).
• Modularity and Extensibility: The separation of concerns in modules like vocabulary creation and vector conversion shows a modular design, which is further extended by LLM-based query capabilities.
• Support for Asynchronous and Synchronous Operations: Emphasize the dual-mode operation allowing the system to handle both blocking and non-blocking DB operations.
Relevant Technologies
• Python, SQLAlchemy, and Pandas – these are core to data processing and ORM functionality.
• AsyncIO – used for asynchronous operations.
• LLM integration components like llama-index (as mentioned in the README) for natural language processing.
• Testing frameworks (pytest, tox, Travis CI) and documentation tools (Sphinx via docs/conf.py) as part of the development and deployment pipeline.
Essential Elements
• Annotations where necessary to represent asynchronous flows versus synchronous ones.