How georgetown-cset/parat works
company_linkage – the “backend” data engine
• This component is responsible for aggregating, cleaning, and processing data from diverse sources.
• It contains configuration files (for Airtable and SQLFluff), Docker support (Dockerfile), numerous SQL scripts and queries for generating reports (e.g., ai_company_papers.sql, visualization_data.sql, etc.), and JSON schema definitions.
• Also present are Python scripts (e.g., parat_data_dag.py, various “parat_scripts”) that likely orchestrate data ingestion, possibly by triggering DAGs in Airflow (as suggested by push_to_airflow.sh).
web – the “frontend” website
• This part contains two subprojects (gui-v2 and parat) that appear to be built with a static site generator (Gatsby, as deduced by files like gatsby-config.js, gatsby-browser.js, etc.).
• The frontend code is organized into components, pages, static data, and tests.
• It fetches and displays the processed data produced by company_linkage, thereby rendering the visual part of PARAT.
1. Data Processing / Aggregation (company_linkage)
• Sub-components:
• Data Extraction & Configuration: airtable_configs, airtable_queries, and airtable_schemas.
• Data Transformation: SQL scripts (in the sql directory) and Python scripts (parat_data_dag.py, parat_scripts). This is where raw data is processed and aggregated.
• Data Storage / Intermediate Files: schemas and sequences (CSV files, etc.) that capture the system’s state.
• Containerization: Dockerfile and related configuration (enabling reproducible environments).
• CI/CD: GitHub workflows (deploy.yml, pythonapp.yml, etc.) that automate testing and deployment.
2. Web Frontend (web)
• Sub-components:
• GUI-v2 and parat projects, which contain similar files but may target different audiences or modes. They both use Gatsby, with separate configurations and components.
• Static assets: pages, components, images, and raw_data that are incorporated into the build.
• Testing infrastructure: a dedicated tests directory and test files under components and util.
• Scripts: additional deployment or build scripts (push_to_production.sh, etc.).
B. Illustrate the relationship and data flow:
• GitHub Actions (CI/CD pipelines) as a supporting process that touches both the backend and frontend, ensuring tests, linting, and deployment.
C. Highlight important architectural patterns/design principles:
• Separation of concerns: one part of the system dedicates itself to processing and generating data, while the other is focused on user interface and presentation.
• Use of containerization (Docker) for environment management and reproducibility.
• That both parts have independent testing suites and CI/CD workflows, indicating robust practices.
D. Include references to relevant technologies/frameworks:
• Docker for containerization.
• Gatsby (React-based) for the frontend.
• Python for data orchestration and SQL for data querying.
• GitHub Actions for automating workflows.
• Clearly separate the backend processing unit (company_linkage) from the frontend serving unit (web).
• Within each block, break out sub-components (for instance, inside company_linkage: separate blocks for Airtable configs, SQL queries, Python scripts, etc.).
• Additional components like CI/CD pipelines (GitHub workflows) that support the entire system.