# julianprester/calls-for-papers architecture

> Architecture diagram of the GitHub repository julianprester/calls-for-papers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/julianprester/calls-for-papers
- Repository: https://github.com/julianprester/calls-for-papers

## Overview

##### Main Components
- Scrapers: This portion of the system (located in the “scrapers” folder) uses Node.js along with Puppeteer and other custom modules (like llmParser.mjs, pageController.mjs, etc.) to gather information on calls for papers from various academic journals. Notice the existence of dedicated subfolders (like “journals”) which group journal-specific scraper scripts.
- Data Aggregation/Preparation: Modules like dataPreparation.mjs, fileParser.mjs, and diffChecker.mjs suggest that after scraping, data is processed and possibly transformed (e.g., feed generation via feedgen.mjs) before being used by the website.
- Website/Frontend: The “www” folder contains the static site structure (HTML templates, layout chunks, and static assets) which is built using 11ty. There’s also evidence of using Tailwind CSS (tailwind.config.mjs), Alpine.js, and other libraries mentioned in the README.
- Build and Configuration: Files such as eleventy.config.mjs, package.json, and tailwind.config.mjs indicate configuration settings for the build process, static site generation, and styling. GitHub Actions workflows (inside .github/workflows) handle automation tasks (like scraping tasks or site deployments) and contribute to continuous integration (CI).

##### Relationships and Data Flow
- Data Flow: The scrapers run periodically (or upon triggering) to collect calls for papers data. This data is processed (via diffChecker, fileParser, etc.) and stored into data sources (for example, JSON files within www/_data).
- Website Build: The static site generator (11ty) then uses the processed data to dynamically generate pages (such as index.html, journal-pages.njk, call-pages.njk, etc.) that are later deployed to a hosting platform like Netlify.
- Configuration and Build Pipelines: GitHub workflows manage automated tasks such as scraping, builds, and possibly deployments.
- External Libraries/Services: Puppeteer, Node.js, and other packages assist in execution behind the scenes. The README specifies integration with external platforms like Netlify for deployment.

##### Architectural Patterns & Design Principles
- Separation of Concerns: One set of modules focuses on data collection (scrapers, data preparation) while another is responsible for rendering a polished, static website.
- Modular Design: The presence of multiple, specialized scraper files and clearly segmented “www” directory hints at an architecture designed for extensibility and maintainability.
- Automation and CI: With GitHub workflows, the system promotes continuous integration practices that keep the data fresh and the deployment processes automated.

## Diagram (Mermaid)

```mermaid
graph TD
    %% CI/CD Layer
    M["CI/CD Pipeline (GitHub Workflows)"]:::ci

    %% Scraping Layer
    subgraph "Scraping"
        A["Scraper Engine"]:::scraper
        B["Browser-Based Scraper"]:::scraper
        C["Journal-Specific Scrapers"]:::scraper
        D1["LLM Parser"]:::scraper
        D2["Page Control"]:::scraper
    end

    %% Data Aggregation Layer
    subgraph "Data Aggregation/Preparation Modules"
        E["Data Preparation"]:::data
        F["File Parser"]:::data
        G["Diff Checker"]:::data
        H["Feed Generator"]:::data
        I["Aggregated Data (JSON)"]:::datastore
    end

    %% Static Site Build Layer
    subgraph "Static Site Build"
        J["Static Site Generator (11ty) & Frontend"]:::frontend
        subgraph "Build & Configuration"
            K["eleventy.config.mjs"]:::config
            L["tailwind.config.mjs"]:::config
            O["package.json"]:::config
        end
    end

    %% Deployment Layer
    P["Website Deployment (Netlify)"]:::deploy

    %% Connections from CI/CD to other layers
    M -->|"triggers"| A
    M -->|"configures"| K
    M -->|"deploys"| P

    %% Internal connections within Scraping Layer
    A --> B
    A --> C
    A --> D1
    A --> D2

    %% Connections from Scraping to Data Aggregation
    B --> E
    C --> F
    D1 --> G
    D2 --> H

    %% Aggregation modules consolidate data
    E --> I
    F --> I
    G --> I
    H --> I

    %% Data flows to Static Site Build
    I --> J

    %% Build & Configuration support Static Site Generation
    K --> J
    L --> J
    O --> J

    %% Static Site Generation results in Deployment
    J --> P

    %% Styling Classes
    classDef scraper fill:#FFC0CB,stroke:#FF69B4,stroke-width:2px;
    classDef data fill:#ADD8E6,stroke:#00008B,stroke-width:2px;
    classDef datastore fill:#E6E6FA,stroke:#8A2BE2,stroke-width:2px;
    classDef frontend fill:#90EE90,stroke:#006400,stroke-width:2px;
    classDef config fill:#FFFF99,stroke:#CCCC00,stroke-width:2px;
    classDef ci fill:#DDA0DD,stroke:#9932CC,stroke-width:2px;
    classDef deploy fill:#FFA500,stroke:#FF4500,stroke-width:2px;

    %% Click Events for Scraping Layer
    click A "https://github.com/julianprester/calls-for-papers/tree/main/scrapers/"
    click B "https://github.com/julianprester/calls-for-papers/blob/main/scrapers/browser.mjs"
    click C "https://github.com/julianprester/calls-for-papers/tree/main/scrapers/journals/"
    click D1 "https://github.com/julianprester/calls-for-papers/blob/main/scrapers/llmParser.mjs"
    click D2 "https://github.com/julianprester/calls-for-papers/blob/main/scrapers/pageController.mjs"

    %% Click Events for Data Aggregation Modules
    click E "https://github.com/julianprester/calls-for-papers/blob/main/scrapers/dataPreparation.mjs"
    click F "https://github.com/julianprester/calls-for-papers/blob/main/scrapers/fileParser.mjs"
    click G "https://github.com/julianprester/calls-for-papers/blob/main/scrapers/diffChecker.mjs"
    click H "https://github.com/julianprester/calls-for-papers/blob/main/scrapers/feedgen.mjs"

    %% Click Event for Static Site Generator & Frontend
    click J "https://github.com/julianprester/calls-for-papers/tree/main/www/"

    %% Click Events for Build & Configuration
    click K "https://github.com/julianprester/calls-for-papers/blob/main/eleventy.config.mjs"
    click L "https://github.com/julianprester/calls-for-papers/blob/main/tailwind.config.mjs"
    click O "https://github.com/julianprester/calls-for-papers/blob/main/package.json"

    %% Click Event for CI/CD Pipeline
    click M "https://github.com/julianprester/calls-for-papers/tree/main/.github/workflows/"
```
