# rendello/canada-census-2021-sqlite architecture

> Architecture diagram of the GitHub repository rendello/canada-census-2021-sqlite, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rendello/canada-census-2021-sqlite
- Repository: https://github.com/rendello/canada-census-2021-sqlite

## Overview

##### Data Sources Layer
- Sub-components for the six regional TAB files:
- Atlantic
- British Columbia
- Ontario
- Prairies
- Quebec
- Territories

##### Processing Layer
- task.py (likely the main processing script)
- to_db.py (database creation/population script)

##### Storage Layer
- Main view 'cview' as a sub-component
- Data schema structure with key columns as listed in the README

Design Guidelines:

- [Data Sources] (Top)
- ↓
- [Processing Scripts] (Middle)
- ↓
- [SQLite Database] (Bottom)

##### Data Transformation Details
- Encoding transformation (ISO-8859-1 → UTF-8)
- Column mapping process
- Consolidation of multiple regional files

##### Technical Specifications
- File format conversions (TAB → SQLite)
- View creation process
- Data quality flags and symbols handling

Specific Elements to Highlight:

- Arrowheads to show direction of data movement

##### Component Labels
- Use clear, concise labels
- Brief descriptions of each component's purpose
- Version or date information where relevant

##### Database Schema
- Main view 'cview' structure
- Columns and their relationships
- Data types and constraints

##### Processing Details
- Transformation steps
- Data validation or quality checks
- Error handling processes

- Notes about excluded data (French data, confidence intervals, etc.)
- Document the geographic levels included

Remember to maintain a balance between technical detail and clarity, as this project's main purpose is to make census data more accessible and queryable through SQLite.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Sources Layer
    subgraph DataSources["Statistics Canada Census Data Sources"]
        Atlantic["Atlantic Region TAB"]:::source
        BC["British Columbia TAB"]:::source
        Ontario["Ontario TAB"]:::source
        Prairies["Prairies TAB"]:::source
        Quebec["Quebec TAB"]:::source
        Territories["Territories TAB"]:::source
    end

    %% Processing Layer
    subgraph Processing["Data Processing Layer"]
        TaskPy["task.py<br/>Main Processing Script"]:::processing
        ToDBPy["to_db.py<br/>Database Creation"]:::processing
        
        subgraph Transformations["Data Transformations"]
            Encoding["ISO-8859-1 → UTF-8<br/>Encoding Conversion"]:::transform
            ColumnMap["Column Mapping<br/>& Standardization"]:::transform
            DataValidation["Data Validation<br/>& Quality Checks"]:::transform
        end
    end

    %% Storage Layer
    subgraph Database["SQLite Database"]
        SQLite["Census SQLite DB"]:::database
        CView["cview<br/>Main Database View"]:::database
        Schema["Schema Structure:<br/>- Geographic Identifiers<br/>- Census Variables<br/>- Data Quality Flags"]:::schema
    end

    %% Documentation
    Doc["Project Documentation"]:::docs
    Release["Release Documentation"]:::docs

    %% Relationships
    Atlantic & BC & Ontario & Prairies & Quebec & Territories --> TaskPy
    TaskPy --> Encoding
    Encoding --> ColumnMap
    ColumnMap --> DataValidation
    DataValidation --> ToDBPy
    ToDBPy --> SQLite
    SQLite --> CView

    %% Click Events
    click TaskPy "https://github.com/rendello/canada-census-2021-sqlite/blob/master/task.py"
    click ToDBPy "https://github.com/rendello/canada-census-2021-sqlite/blob/master/to_db.py"
    click Doc "https://github.com/rendello/canada-census-2021-sqlite/blob/master/.github/README.md"
    click Release "https://github.com/rendello/canada-census-2021-sqlite/blob/master/release_template.md"

    %% Styling
    classDef source fill:#a8d1f0,stroke:#333,stroke-width:2px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:2px
    classDef transform fill:#FFE4B5,stroke:#333,stroke-width:2px
    classDef database fill:#FFA07A,stroke:#333,stroke-width:2px
    classDef schema fill:#F0E68C,stroke:#333,stroke-width:2px
    classDef docs fill:#D3D3D3,stroke:#333,stroke-width:2px

    %% Legend
    subgraph Legend["Legend"]
        L1["Data Sources"]:::source
        L2["Processing Scripts"]:::processing
        L3["Transformations"]:::transform
        L4["Database Components"]:::database
        L5["Schema Details"]:::schema
        L6["Documentation"]:::docs
    end
```
