How julianprester/calls-for-papers works
Main Components
• Scrapers: This portion of the system (located in the “scrapers” folder) uses Node.js along with Puppeteer and other custom modules (like llmParser.mjs, pageController.mjs, etc.) to gather information on calls for papers from various academic journals. Notice the existence of dedicated subfolders (like “journals”) which group journal-specific scraper scripts.
• Data Aggregation/Preparation: Modules like dataPreparation.mjs, fileParser.mjs, and diffChecker.mjs suggest that after scraping, data is processed and possibly transformed (e.g., feed generation via feedgen.mjs) before being used by the website.
• Website/Frontend: The “www” folder contains the static site structure (HTML templates, layout chunks, and static assets) which is built using 11ty. There’s also evidence of using Tailwind CSS (tailwind.config.mjs), Alpine.js, and other libraries mentioned in the README.
• Build and Configuration: Files such as eleventy.config.mjs, package.json, and tailwind.config.mjs indicate configuration settings for the build process, static site generation, and styling. GitHub Actions workflows (inside .github/workflows) handle automation tasks (like scraping tasks or site deployments) and contribute to continuous integration (CI).
Relationships and Data Flow
• Data Flow: The scrapers run periodically (or upon triggering) to collect calls for papers data. This data is processed (via diffChecker, fileParser, etc.) and stored into data sources (for example, JSON files within www/_data).
• Website Build: The static site generator (11ty) then uses the processed data to dynamically generate pages (such as index.html, journal-pages.njk, call-pages.njk, etc.) that are later deployed to a hosting platform like Netlify.
• Configuration and Build Pipelines: GitHub workflows manage automated tasks such as scraping, builds, and possibly deployments.
• External Libraries/Services: Puppeteer, Node.js, and other packages assist in execution behind the scenes. The README specifies integration with external platforms like Netlify for deployment.
Architectural Patterns & Design Principles
• Separation of Concerns: One set of modules focuses on data collection (scrapers, data preparation) while another is responsible for rendering a polished, static website.
• Modular Design: The presence of multiple, specialized scraper files and clearly segmented “www” directory hints at an architecture designed for extensibility and maintainability.
• Automation and CI: With GitHub workflows, the system promotes continuous integration practices that keep the data fresh and the deployment processes automated.