How blackcursive/civilservicenyc works
Project Type & Purpose
• This is a full-stack web application (albeit monolithic in structure) that provides public access to civil service exam information.
• It includes a data ingestion pipeline (ETL) that periodically extracts and transforms external datasets before loading them into an internal database.
• While the project uses Django for server-side rendering and business logic management, it also leverages Pandas for data handling and analysis.
File Structure Analysis
• Even though the file tree only shows a README.md, the README content outlines the entire technology stack and process.
• Architectural separation between the ETL process and the web application layer built with Django.
• Recognize that configuration settings, models, views, and templates (typical in Django) would exist even if not explicitly shown in the file tree.
Main Components
a. External Data Sources:
• NYC OpenData portals (Civil Service List, Civil Service Titles) and DCAS Past Exam Archive.
• As external systems or databases from which data originates.
b. ETL Process:
• Extraction: Data is downloaded in CSV/Excel formats from the external data sources.
• Transformation: Processed using Python, Pandas, and sometimes reviewed in Excel or via a Jupyter Notebook.
• Loading: Data is inserted into Django Models, populating an SQLite database.
c. Web Application:
• Django Backend: Handles requests, enforces business logic, renders views/templates;
• Database (SQLite): Stores processed data, including applicant and exam information.
• (Optional) Frontend Layer: Although implicitly handled by Django templates, it provides an intuitive and user-friendly experience.
d. Developer Tools:
• Visual Studio Code and any other development and debugging utilities that support the development pipeline.
Relationships and Interactions
• From External Data Sources → ETL Process (Extraction and Transformation)
• From ETL Process → SQLite Database (Loading phase)
• From Django Web Application → SQLite Database (Querying and rendering data for the UI)
• That the ETL process functions periodically or on-demand to update the central database that the Django application uses.
Architectural Patterns & Design Principles
• Separation of concerns: The ETL pipeline is distinct from the web serving components.
• Use of Django’s MVC (or MVT) pattern, where Models represent the data layer, Views/Controllers manage business logic, and Templates render the UI.
• Data flow is modular so that each component (data ingestion, transformation, storage, and presentation) is clearly delineated.