# blackcursive/civilservicenyc architecture

> Architecture diagram of the GitHub repository blackcursive/civilservicenyc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blackcursive/civilservicenyc
- Repository: https://github.com/blackcursive/civilservicenyc

## Overview

##### Project Type & Purpose
- This is a full-stack web application (albeit monolithic in structure) that provides public access to civil service exam information.
- It includes a data ingestion pipeline (ETL) that periodically extracts and transforms external datasets before loading them into an internal database.
- While the project uses Django for server-side rendering and business logic management, it also leverages Pandas for data handling and analysis.

##### File Structure Analysis
- Even though the file tree only shows a README.md, the README content outlines the entire technology stack and process.
- Architectural separation between the ETL process and the web application layer built with Django.
- Recognize that configuration settings, models, views, and templates (typical in Django) would exist even if not explicitly shown in the file tree.

##### Main Components
a. External Data Sources:
- NYC OpenData portals (Civil Service List, Civil Service Titles) and DCAS Past Exam Archive.
- As external systems or databases from which data originates.
b. ETL Process:
- Extraction: Data is downloaded in CSV/Excel formats from the external data sources.
- Transformation: Processed using Python, Pandas, and sometimes reviewed in Excel or via a Jupyter Notebook.
- Loading: Data is inserted into Django Models, populating an SQLite database.
c. Web Application:
- Django Backend: Handles requests, enforces business logic, renders views/templates;
- Database (SQLite): Stores processed data, including applicant and exam information.
- (Optional) Frontend Layer: Although implicitly handled by Django templates, it provides an intuitive and user-friendly experience.
d. Developer Tools:
- Visual Studio Code and any other development and debugging utilities that support the development pipeline.

##### Relationships and Interactions
- From External Data Sources → ETL Process (Extraction and Transformation)
- From ETL Process → SQLite Database (Loading phase)
- From Django Web Application → SQLite Database (Querying and rendering data for the UI)
- That the ETL process functions periodically or on-demand to update the central database that the Django application uses.

##### Architectural Patterns & Design Principles
- Separation of concerns: The ETL pipeline is distinct from the web serving components.
- Use of Django’s MVC (or MVT) pattern, where Models represent the data layer, Views/Controllers manage business logic, and Templates render the UI.
- Data flow is modular so that each component (data ingestion, transformation, storage, and presentation) is clearly delineated.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Pipeline"
        extData["External Data Sources (NYC OpenData & DCAS Archive)"]:::external
        etlProc["ETL Process (Extract,Transform,Load with Python & Pandas)"]:::etl
        sqliteDB["SQLite Database (Django ORM Storage)"]:::database
        extData -->|"data_extraction"| etlProc
        etlProc -->|"data_loading"| sqliteDB
    end

    subgraph "Web Application"
        djangoApp["Django Web Application (Business Logic)"]:::backend
        userInterface["User Interface (Django Templates)"]:::ui
        djangoApp -->|"renders"| userInterface
        djangoApp -->|"queries"| sqliteDB
    end

    devTools["Developer Tools (VS Code)"]:::tools
    devTools -.-> djangoApp

    click etlProc "https://github.com/blackcursive/civilservicenyc/blob/master/README.md"
    click djangoApp "https://github.com/blackcursive/civilservicenyc/blob/master/README.md"
    click sqliteDB "https://github.com/blackcursive/civilservicenyc/blob/master/README.md"

    classDef external fill:#f96,stroke:#333,stroke-width:2px;
    classDef etl fill:#ff0,stroke:#333,stroke-width:2px;
    classDef backend fill:#ccf,stroke:#333,stroke-width:2px;
    classDef database fill:#cfc,stroke:#333,stroke-width:2px;
    classDef ui fill:#fcf,stroke:#333,stroke-width:2px;
    classDef tools fill:#ddd,stroke:#333,stroke-width:2px;
```
