# thyripian/core architecture

> Architecture diagram of the GitHub repository thyripian/core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thyripian/core
- Repository: https://github.com/thyripian/core

## Overview

##### Project type and purpose
- This is a full-stack reporting engine (“CORE”) written in Python with a React frontend.
- It extracts data from various file types, processes and indexes it in Elasticsearch and relational stores (Postgres/SQLite), and exposes search/update APIs to the React UI.

##### File structure and architectural layers
- Top-level layers:
- config/ – environment‐specific settings and credentials
- initialization/ – configuration loading and service bootstrapping
- core/ – entry point and high-level orchestration
- file_handling/ – I/O, directory traversal, file reads
- data_extraction/ – text and image extraction from files
- data_processing/ – cleanup, NLP, info structuring
- database_operations/ – multi-store support (Elasticsearch, Postgres, SQLite) via factory pattern
- search_functionality/ – query construction against ES
- update_functionality/ – triggering re-processing or new data ingestion
- user_interface/ui/ – React single-page application
- utilities/ – shared helpers (logging, configuration, dialogues)
- tests/ – unit, integration, synthetic data

##### README insights
- Dependencies: Python ≥3.8, Node.js/npm, Elasticsearch 8.x, Postgres ≥12, (optional) SQLite
- Run sequence:
1. Load sys_config.json and settings.json
2. Initialize logging and NLP models
3. Kick off core/run_app.py or update/run_update_process.py
4. Extract → Process → Store in ES/Postgres/SQLite
5. User queries via search_functionality → ES
6. React frontend calls backend APIs on startup and user actions

##### Main components
a. Presentation Layer
- React Frontend (user_interface/ui): components, scripts, assets
- HTTP/API interface exposed by Python backend
b. Application Layer (Backend)
- initialization (config_manager, init_app, initialize_elasticsearch)
- core/run_app (or run_update_process) orchestrates flows
- modules: file_handling → data_extraction → data_processing → database_operations
- search_functionality for query endpoints
c. Data Layer
- Elasticsearch cluster (indexing & search)
- PostgreSQL (relational storage, fallback)
- SQLite (local fallback storage)
d. Infrastructure/Utilities
- utilities (logging, config loading, dialogues)
- config files (sys_config.json, settings.json)
- external services (Elasticsearch, Postgres, Node.js server hosting React)

##### Relationships and data flow
1. React UI → HTTP calls → Python API endpoints
2. API endpoints invoke core orchestration
3. Core calls file_handling to discover and read files
4. Raw files → data_extraction modules → extracted content
5. Extracted content → data_processing modules → normalized records
6. Processed records → database_operations via db_factory → ES index, Postgres tables or SQLite file
7. Search requests → search_functionality.elastic_query → ES → return results → UI
8. Update workflow analogous: change detection → re‐ingestion pipelines

##### Architectural patterns and design principles
- Layered architecture: clear separation (Presentation / Application / Domain / Data)
- Factory pattern: db_factory abstracts multi-store support
- Modular single‐responsibility modules (extraction vs processing vs storage)
- Configuration‐driven: sys_config.json and settings.json drive behavior without code changes
- Extensibility points: new extractors, new database drivers

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Presentation Layer"
        UI["React Frontend"]:::frontend
        Components["Components (src/components)"]:::frontend
    end
    UI --> Components
    UI -->|HTTP/API calls| API["HTTP/API Interface"]:::backend

    subgraph "Application Layer"
        subgraph "Initialization"
            ConfigManager["config_manager"]:::backend
            InitApp["init_app"]:::backend
            InitES["initialize_elasticsearch"]:::backend
            ConfigManager --> InitApp --> InitES
        end
        subgraph "Core"
            GenericCore["generic.py"]:::backend
            RunApp["run_app.py"]:::backend
            GenericCore --> RunApp
        end
        subgraph "File Handling"
            DirTrav["directory_traversal"]:::backend
            ExtractFiles["extract_files"]:::backend
            FileIO["file_io"]:::backend
            DirTrav --> ExtractFiles --> FileIO
        end
        subgraph "Data Extraction"
            ImgExt["image_extractors"]:::backend
            TextExt["text_extractors"]:::backend
            ImgExt --> TextExt
        end
        subgraph "Data Processing"
            DPI["data_processing_manager"]:::backend
            InfoProc["info_processing"]:::backend
            TextProc["text_processing"]:::backend
            DPI --> InfoProc --> TextProc
        end
        subgraph "Database Operations"
            DBFactory["db_factory"]:::backend
            BaseDB["base_database"]:::backend
            DBManager["db_manager"]:::backend
            ESOps["elasticsearch_operations"]:::backend
            PGOps["postgres_operations"]:::backend
            SQLOps["sqlite_operations"]:::backend
            FallbackOps["fallback_operations"]:::backend
            HashCheck["hash_checker"]:::backend
            DBFactory --> BaseDB --> DBManager
            DBManager --> ESOps
            DBManager --> PGOps
            DBManager --> SQLOps
            DBManager --> FallbackOps
            DBFactory --> HashCheck
        end
        subgraph "Search & Update"
            Search["elastic_query"]:::backend
            CheckUpd["check_updates"]:::backend
            RunUpd["run_update_process"]:::backend
            API --> Search
            API --> RunApp
            API --> RunUpd
            Search --> ESCluster
            CheckUpd --> RunUpd
        end

        RunApp --> DirTrav --> ImgExt --> DPI --> DBFactory
        FileIO --> ImgExt
        ExtractFiles --> DPI
    end

    subgraph "Data Layer"
        ESCluster["Elasticsearch"]:::datastore
        Postgres["PostgreSQL"]:::datastore
        SQLite["SQLite"]:::datastore
        ESCluster --> Postgres
        Postgres --> SQLite
    end

    subgraph "Configurations"
        Settings["settings.json"]:::config
        SysConfig["sys_config.json"]:::config
        DefaultKW["default_keywords.json"]:::config
        Settings --> ConfigManager
        SysConfig --> ConfigManager
        DefaultKW --> ConfigManager
    end

    subgraph "Utilities"
        UtilsConfig["configs.py"]:::util
        DBConfig["database_config.py"]:::util
        DataPrep["data_preparation.py"]:::util
        Keywords["keywords.json"]:::util
        ProgWin["progress_window.py"]:::util
        SelDiag["selection_dialogues.py"]:::util
        KWLoad["keyword_loading.py"]:::util
        DevLog["dev_logging.py"]:::util
        LogUtil["logging_utilities.py"]:::util
        ResMgmt["resource_management.py"]:::util
        UtilsConfig --> RunApp
        DBConfig --> DBFactory
        DataPrep --> DPI
    end

    click UI "https://github.com/thyripian/core/tree/main/user_interface/ui"
    click Components "https://github.com/thyripian/core/tree/main/user_interface/ui/src/components/"
    click API "https://github.com/thyripian/core/blob/main/core/run_app.py"
    click RunUpd "https://github.com/thyripian/core/blob/main/update_functionality/run_update_process.py"
    click ConfigManager "https://github.com/thyripian/core/blob/main/initialization/config_manager.py"
    click InitApp "https://github.com/thyripian/core/blob/main/initialization/init_app.py"
    click InitES "https://github.com/thyripian/core/blob/main/initialization/initialize_elasticsearch.py"
    click GenericCore "https://github.com/thyripian/core/blob/main/core/generic.py"
    click RunApp "https://github.com/thyripian/core/blob/main/core/run_app.py"
    click DirTrav "https://github.com/thyripian/core/blob/main/file_handling/directory_traversal.py"
    click ExtractFiles "https://github.com/thyripian/core/blob/main/file_handling/extract_files.py"
    click FileIO "https://github.com/thyripian/core/blob/main/file_handling/file_io.py"
    click ImgExt "https://github.com/thyripian/core/blob/main/data_extraction/image_extractors.py"
    click TextExt "https://github.com/thyripian/core/blob/main/data_extraction/text_extractors.py"
    click DPI "https://github.com/thyripian/core/blob/main/data_processing/data_processing_manager.py"
    click InfoProc "https://github.com/thyripian/core/blob/main/data_processing/info_processing.py"
    click TextProc "https://github.com/thyripian/core/blob/main/data_processing/text_processing.py"
    click DBFactory "https://github.com/thyripian/core/blob/main/database_operations/db_factory.py"
    click BaseDB "https://github.com/thyripian/core/blob/main/database_operations/base_database.py"
    click DBManager "https://github.com/thyripian/core/blob/main/database_operations/db_manager.py"
    click ESOps "https://github.com/thyripian/core/blob/main/database_operations/elasticsearch_operations.py"
    click PGOps "https://github.com/thyripian/core/blob/main/database_operations/postgres_operations.py"
    click SQLOps "https://github.com/thyripian/core/blob/main/database_operations/sqlite_operations.py"
    click FallbackOps "https://github.com/thyripian/core/blob/main/database_operations/fallback_operations.py"
    click HashCheck "https://github.com/thyripian/core/blob/main/database_operations/hash_checker.py"
    click Search "https://github.com/thyripian/core/blob/main/search_functionality/elastic_query.py"
    click CheckUpd "https://github.com/thyripian/core/blob/main/update_functionality/check_updates.py"
    click Settings "https://github.com/thyripian/core/blob/main/config/settings.json"
    click SysConfig "https://github.com/thyripian/core/blob/main/config/sys_config.json"
    click DefaultKW "https://github.com/thyripian/core/blob/main/config/default_keywords.json"
    click UtilsConfig "https://github.com/thyripian/core/blob/main/utilities/configurations/configs.py"
    click DBConfig "https://github.com/thyripian/core/blob/main/utilities/configurations/database_config.py"
    click DataPrep "https://github.com/thyripian/core/blob/main/utilities/data_preparation.py"
    click Keywords "https://github.com/thyripian/core/blob/main/utilities/default_keywords/keywords.json"
    click ProgWin "https://github.com/thyripian/core/blob/main/utilities/dialogues/progress_window.py"
    click SelDiag "https://github.com/thyripian/core/blob/main/utilities/dialogues/selection_dialogues.py"
    click KWLoad "https://github.com/thyripian/core/blob/main/utilities/keyword_loading.py"
    click DevLog "https://github.com/thyripian/core/blob/main/utilities/logging/dev/dev_logging.py"
    click LogUtil "https://github.com/thyripian/core/blob/main/utilities/logging/logging_utilities.py"
    click ResMgmt "https://github.com/thyripian/core/blob/main/utilities/resource_management.py"

    classDef frontend fill:#ACE1AF,stroke:#333,stroke-width:1px
    classDef backend fill:#B2DFDB,stroke:#333,stroke-width:1px
    classDef datastore fill:#FFD54F,stroke:#333,stroke-width:1px
    classDef config fill:#FFECB3,stroke:#333,stroke-width:1px
    classDef util fill:#E1BEE7,stroke:#333,stroke-width:1px
```
