How cmu-hgc-mac/hgc_db_postgres works
Main Components Identification
• Database Server – This is the PostgreSQL-15 database which is the core of the system. The README and numerous configuration files (e.g., dbase_info/conn_example.yaml) indicate that accurate connection details (ports, hostnames, user credentials) are crucial.
• Database Setup Modules – The “create” directory contains scripts (create_database.py, create_tables.py) to initially create the database and configure tables.
• Database Information & Configuration – The “dbase_info” directory holds connection examples, tables definitions (YAML and CSV files) for various parts (e.g., baseplate, sensor, module, etc.), showing the schema design.
• Data Export Functionality – The “export_data” folder includes scripts and various utility modules for exporting data (especially generating XMLs and CSV files). There are clearly defined utilities for module assembly, sensor test data, and protomodule exports.
• Data Import Functionality – The “import_data” directory includes scripts to retrieve parts data and sensor IV data from external sources (including a reference to CERN parts information), implementing the data ingestion pipeline.
• Table Modification & Housekeeping – The “modify” and “housekeeping” directories include scripts for updating table hierarchies, verifying received components, and updating foreign keys. These are critical for ongoing data integrity and DB maintenance.
• MAC Public Information – The “mac_public” folder contains scripts to extract and process data (e.g., get_macs_data.py) that are likely used for public reporting or dashboard purposes.
• Control Panel Interface – The script “postgres_control_panel.py” serves as a central GUI-based tool to trigger operations like table modifications and refreshes, integrating the various database operations.
Relationships and Interactions
• The Control Panel (postgres_control_panel.py) is the user-facing layer that triggers the database setup, modification, and data import/export operations.
• The Setup modules (create and dbase_info) interact directly with the PostgreSQL server to create/modify the database schema.
• Import (import_data) and Export (export_data) components perform bidirectional data flows: data is imported from and exported to external sources (e.g., CERN’s API and file systems), ensuring the database stays synchronized.
• Housekeeping and Modify scripts perform periodic verification and update tasks to maintain consistency between various tables and relationships within the database.
• There is also an integration point with a browser-based tool (AdminerEvo) that is accessed via PHP installations, emphasizing external interaction for data viewing and editing.
Architectural Patterns & Technologies
• Modular design is evident – the project separates functionality into distinct folders (create, import_data, export_data, modify, etc.). This modularity supports maintainability and clear separation of concerns.
• Configuration-Driven Setup – YAML files and CSV templates define database connection parameters and table structures.
• While this is a backend database management system, the GUI elements (via the control panel) indicate an integrated application layer aimed at both automation and user-triggered interventions.