How thyripian/core works
Project type and purpose
• This is a full-stack reporting engine (“CORE”) written in Python with a React frontend.
• It extracts data from various file types, processes and indexes it in Elasticsearch and relational stores (Postgres/SQLite), and exposes search/update APIs to the React UI.
File structure and architectural layers
• Top-level layers:
• config/ – environment‐specific settings and credentials
• initialization/ – configuration loading and service bootstrapping
• core/ – entry point and high-level orchestration
• file_handling/ – I/O, directory traversal, file reads
• data_extraction/ – text and image extraction from files
• data_processing/ – cleanup, NLP, info structuring
• database_operations/ – multi-store support (Elasticsearch, Postgres, SQLite) via factory pattern
• search_functionality/ – query construction against ES
• update_functionality/ – triggering re-processing or new data ingestion
• user_interface/ui/ – React single-page application
• utilities/ – shared helpers (logging, configuration, dialogues)
• tests/ – unit, integration, synthetic data
README insights
• Dependencies: Python ≥3.8, Node.js/npm, Elasticsearch 8.x, Postgres ≥12, (optional) SQLite
• Run sequence:
1. Load sys_config.json and settings.json
2. Initialize logging and NLP models
3. Kick off core/run_app.py or update/run_update_process.py
4. Extract → Process → Store in ES/Postgres/SQLite
5. User queries via search_functionality → ES
6. React frontend calls backend APIs on startup and user actions
Main components
a. Presentation Layer
• React Frontend (user_interface/ui): components, scripts, assets
• HTTP/API interface exposed by Python backend
b. Application Layer (Backend)
• initialization (config_manager, init_app, initialize_elasticsearch)
• core/run_app (or run_update_process) orchestrates flows
• modules: file_handling → data_extraction → data_processing → database_operations
• search_functionality for query endpoints
c. Data Layer
• Elasticsearch cluster (indexing & search)
• PostgreSQL (relational storage, fallback)
• SQLite (local fallback storage)
d. Infrastructure/Utilities
• utilities (logging, config loading, dialogues)
• config files (sys_config.json, settings.json)
• external services (Elasticsearch, Postgres, Node.js server hosting React)
Relationships and data flow
1. React UI → HTTP calls → Python API endpoints
2. API endpoints invoke core orchestration
3. Core calls file_handling to discover and read files
4. Raw files → data_extraction modules → extracted content
5. Extracted content → data_processing modules → normalized records
6. Processed records → database_operations via db_factory → ES index, Postgres tables or SQLite file
7. Search requests → search_functionality.elastic_query → ES → return results → UI
8. Update workflow analogous: change detection → re‐ingestion pipelines
Architectural patterns and design principles
• Layered architecture: clear separation (Presentation / Application / Domain / Data)
• Factory pattern: db_factory abstracts multi-store support
• Modular single‐responsibility modules (extraction vs processing vs storage)
• Configuration‐driven: sys_config.json and settings.json drive behavior without code changes
• Extensibility points: new extractors, new database drivers