# georgetown-cset/parat architecture

> Architecture diagram of the GitHub repository georgetown-cset/parat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/georgetown-cset/parat
- Repository: https://github.com/georgetown-cset/parat

## Overview

##### company_linkage – the “backend” data engine
- This component is responsible for aggregating, cleaning, and processing data from diverse sources.
- It contains configuration files (for Airtable and SQLFluff), Docker support (Dockerfile), numerous SQL scripts and queries for generating reports (e.g., ai_company_papers.sql, visualization_data.sql, etc.), and JSON schema definitions.
- Also present are Python scripts (e.g., parat_data_dag.py, various “parat_scripts”) that likely orchestrate data ingestion, possibly by triggering DAGs in Airflow (as suggested by push_to_airflow.sh).

##### web – the “frontend” website
- This part contains two subprojects (gui-v2 and parat) that appear to be built with a static site generator (Gatsby, as deduced by files like gatsby-config.js, gatsby-browser.js, etc.).
- The frontend code is organized into components, pages, static data, and tests.
- It fetches and displays the processed data produced by company_linkage, thereby rendering the visual part of PARAT.

1. Data Processing / Aggregation (company_linkage)
- Sub-components:
- Data Extraction & Configuration: airtable_configs, airtable_queries, and airtable_schemas.
- Data Transformation: SQL scripts (in the sql directory) and Python scripts (parat_data_dag.py, parat_scripts). This is where raw data is processed and aggregated.
- Data Storage / Intermediate Files: schemas and sequences (CSV files, etc.) that capture the system’s state.
- Containerization: Dockerfile and related configuration (enabling reproducible environments).
- CI/CD: GitHub workflows (deploy.yml, pythonapp.yml, etc.) that automate testing and deployment.

2. Web Frontend (web)
- Sub-components:
- GUI-v2 and parat projects, which contain similar files but may target different audiences or modes. They both use Gatsby, with separate configurations and components.
- Static assets: pages, components, images, and raw_data that are incorporated into the build.
- Testing infrastructure: a dedicated tests directory and test files under components and util.
- Scripts: additional deployment or build scripts (push_to_production.sh, etc.).

B. Illustrate the relationship and data flow:
- GitHub Actions (CI/CD pipelines) as a supporting process that touches both the backend and frontend, ensuring tests, linting, and deployment.

C. Highlight important architectural patterns/design principles:
- Separation of concerns: one part of the system dedicates itself to processing and generating data, while the other is focused on user interface and presentation.
- Use of containerization (Docker) for environment management and reproducibility.
- That both parts have independent testing suites and CI/CD workflows, indicating robust practices.

D. Include references to relevant technologies/frameworks:
- Docker for containerization.
- Gatsby (React-based) for the frontend.
- Python for data orchestration and SQL for data querying.
- GitHub Actions for automating workflows.

- Clearly separate the backend processing unit (company_linkage) from the frontend serving unit (web).

- Within each block, break out sub-components (for instance, inside company_linkage: separate blocks for Airtable configs, SQL queries, Python scripts, etc.).
- Additional components like CI/CD pipelines (GitHub workflows) that support the entire system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Processing & Aggregation (company_linkage)
    subgraph "Data Processing & Aggregation (company_linkage)"
        direction TB
        A1["Airtable Configs"]
        A2["Airtable Queries"]
        A3["Airtable Schemas"]
        B1["SQL Scripts"]
        P1["parat_data_dag.py"]
        P2["parat_scripts"]
        DS["Schemas"]
        DSeq["Sequences"]
        DCont["Docker Support"]
        CI["GitHub Workflows (CI/CD)"]
        PD["Processed Data"]
    end

    %% Web Frontend (web)
    subgraph "Web Frontend (web)"
        direction TB
        FE_G["GUI-v2 (Gatsby)"]
        FE_G_Pages["GUI-v2 Pages & Assets"]
        FE_G_Test["GUI Testing"]
        FE_P["parat (Gatsby)"]
        FE_P_Pages["parat Pages & Assets"]
        FE_P_Test["parat Testing"]
    end

    %% External Services
    AW["Airflow Service"]

    %% Data Flow - Backend processing to Frontend rendering
    A1 --> PD
    A2 --> PD
    A3 --> PD
    B1 --> PD
    P1 --> PD
    P2 --> PD
    DS --> PD
    DSeq --> PD
    DCont --> PD
    PD -->|"feeds"| FE_G_Pages
    PD -->|"feeds"| FE_P_Pages

    %% CI/CD deployment flows
    CI -->|"deploys"| FE_G
    CI -->|"deploys"| FE_P

    %% Python orchestration triggers external Airflow
    P1 -->|"triggers"| AW

    %% Internal Frontend structure connections
    FE_G --> FE_G_Pages
    FE_G --> FE_G_Test
    FE_P --> FE_P_Pages
    FE_P --> FE_P_Test

    %% Click Events for Data Processing & Aggregation
    click A1 "https://github.com/georgetown-cset/parat/tree/master/company_linkage/airtable_configs"
    click A2 "https://github.com/georgetown-cset/parat/tree/master/company_linkage/airtable_queries"
    click A3 "https://github.com/georgetown-cset/parat/tree/master/company_linkage/airtable_schemas"
    click B1 "https://github.com/georgetown-cset/parat/tree/master/company_linkage/sql"
    click P1 "https://github.com/georgetown-cset/parat/blob/master/company_linkage/parat_data_dag.py"
    click P2 "https://github.com/georgetown-cset/parat/tree/master/company_linkage/parat_scripts"
    click DS "https://github.com/georgetown-cset/parat/tree/master/company_linkage/schemas"
    click DSeq "https://github.com/georgetown-cset/parat/tree/master/company_linkage/sequences"
    click DCont "https://github.com/georgetown-cset/parat/tree/master/company_linkage/Dockerfile"
    click CI "https://github.com/georgetown-cset/parat/tree/master/.github/workflows"
    click PD "https://github.com/georgetown-cset/parat/tree/master/company_linkage"

    %% Click Events for Web Frontend
    click FE_G "https://github.com/georgetown-cset/parat/tree/master/web/gui-v2"
    click FE_G_Pages "https://github.com/georgetown-cset/parat/tree/master/web/gui-v2/src/pages"
    click FE_G_Test "https://github.com/georgetown-cset/parat/tree/master/web/gui-v2/src/test"
    click FE_P "https://github.com/georgetown-cset/parat/tree/master/web/parat"
    click FE_P_Pages "https://github.com/georgetown-cset/parat/tree/master/web/parat/src/pages"
    click FE_P_Test "https://github.com/georgetown-cset/parat/tree/master/web/parat/src/components/__tests__"

    %% Styles
    classDef backend fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef frontend fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef external fill:#F4A460,stroke:#000,stroke-width:1px;

    class A1,A2,A3,B1,P1,P2,DS,DSeq,DCont,CI,PD backend;
    class FE_G,FE_G_Pages,FE_G_Test,FE_P,FE_P_Pages,FE_P_Test frontend;
    class AW external;
```
