# camiloandresdg/crime_prediction_la architecture

> Architecture diagram of the GitHub repository camiloandresdg/crime_prediction_la, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/camiloandresdg/crime_prediction_la
- Repository: https://github.com/camiloandresdg/crime_prediction_la

## Overview

##### Project Type and Purpose
- This is a batch‑oriented machine‑learning pipeline built in Python with PySpark.
- Purpose: Ingest LA crime data, process it, train and run forecasting models, and produce rolling 8‑day predictions by zone.

##### Main Architectural Components
a. External Data Source
- LA Official Crime Database (e.g., REST API or flat files)
b. Data Ingestion Layer (`src/data/data_ingestion.py`, `scheduler.py`)
- Pulls raw data on a scheduled basis
- Writes to raw data storage
c. Storage
- Raw Data Store (e.g., HDFS/S3/local `data/raw/`)
- Processed Data Store (temporary staging)
- Predictions Store (`predictions/` directory)
d. Orchestrator (`src/pipeline.py`)
- Coordinates stages in sequence: Ingest → Process → Analyze → Model → Export
e. Data Processing Layer
- PySpark Processor (`src/processing/spark_processor.py`)
- Data Processor (`src/processing/data_processor.py`) for feature engineering
- Descriptive Analytics (`src/analysis/descriptive_analysis.py`)
f. Modeling Layer (`src/models/crime_predictor.py`)
- Crime classification and count regression models
- Zone‑specific model instantiation
g. Scheduler (`scheduler.py`)
- Cron‑style or Airflow‑style trigger for `pipeline.py` every 8 days
h. External Clients/Consumers
- BI dashboard or API reading from `predictions/`

##### Relationships & Data Flow
- Scheduler → triggers → Data Ingestion
- Data Ingestion → writes → Raw Data Store
- Orchestrator reads Raw Data → passes to → Spark Processor → writes → Processed Data
- Processed Data → fed into → Descriptive Analysis & Feature Engineering
- Engineered Features → input → Crime Predictor → outputs → Predictions Store
- Predictions → consumed by → External Clients

##### Architectural Patterns & Principles
- Modular ETL Pipeline: clear layering of ingestion, processing, modeling
- Separation of Concerns: each Python module does one job
- Scalability: PySpark cluster for large data volumes
- Scheduling: decoupled from processing logic

##### Technologies & Frameworks to Annotate
- Python 3.x
- PySpark (Spark Cluster Master & Executors)
- File‑based storage or HDFS/S3
- Virtual Environment / `requirements.txt` dependencies
- Local scheduler (cron/scheduler.py) or external orchestration

## Diagram (Mermaid)

```mermaid
flowchart TB
    ExternalData["External Data Source"]:::external
    Scheduler["Scheduler\n(every 8 days)"]:::compute
    DataIngestion["Data Ingestion\n(data_ingestion.py)"]:::compute
    RawStore["Raw Data Store\n(data/raw/)"]:::storage
    Orchestrator["Orchestrator\n(pipeline.py)"]:::compute

    subgraph "Processing Layer"
        SparkProcessor["Spark Processor\n(spark_processor.py)"]:::compute
        DataProcessor["Data Processor\n(data_processor.py)"]:::compute
        DescriptiveAnalysis["Descriptive Analysis\n(descriptive_analysis.py)"]:::compute
    end

    ProcessedStore["Processed Data Store"]:::storage
    CrimePredictor["Crime Predictor\n(crime_predictor.py)"]:::compute
    PredictionsStore["Predictions Store\n(predictions/)"]:::storage
    ExternalClient["External Clients\n(BI Dashboard/API)"]:::external
    Requirements["Requirements\n(requirements.txt)"]:::external
    Docs["Project Docs\n(README.md)"]:::external

    ExternalData -->|"raw data"| DataIngestion
    Scheduler -->|"triggers"| DataIngestion
    DataIngestion -->|"writes Raw CSV"| RawStore
    RawStore -->|"reads raw data"| Orchestrator
    Orchestrator -->|"passes data"| SparkProcessor
    SparkProcessor -->|"writes processed data"| ProcessedStore
    ProcessedStore -->|"feeds"| DataProcessor
    ProcessedStore -->|"feeds"| DescriptiveAnalysis
    DataProcessor -->|"features"| CrimePredictor
    DescriptiveAnalysis -->|"analytics output"| CrimePredictor
    CrimePredictor -->|"JSON Predictions"| PredictionsStore
    PredictionsStore -->|"consumed by"| ExternalClient

    click DataIngestion "https://github.com/camiloandresdg/crime_prediction_la/blob/main/src/data/data_ingestion.py"
    click Scheduler "https://github.com/camiloandresdg/crime_prediction_la/blob/main/scheduler.py"
    click Orchestrator "https://github.com/camiloandresdg/crime_prediction_la/blob/main/src/pipeline.py"
    click SparkProcessor "https://github.com/camiloandresdg/crime_prediction_la/blob/main/src/processing/spark_processor.py"
    click DataProcessor "https://github.com/camiloandresdg/crime_prediction_la/blob/main/src/processing/data_processor.py"
    click DescriptiveAnalysis "https://github.com/camiloandresdg/crime_prediction_la/blob/main/src/analysis/descriptive_analysis.py"
    click CrimePredictor "https://github.com/camiloandresdg/crime_prediction_la/blob/main/src/models/crime_predictor.py"
    click RawStore "https://github.com/camiloandresdg/crime_prediction_la/tree/main/data/raw/"
    click PredictionsStore "https://github.com/camiloandresdg/crime_prediction_la/tree/main/predictions/"
    click Requirements "https://github.com/camiloandresdg/crime_prediction_la/blob/main/requirements.txt"
    click Docs "https://github.com/camiloandresdg/crime_prediction_la/blob/main/README.md"

    classDef storage fill:#E0F7FA,stroke:#006064,stroke-width:1px
    classDef compute fill:#E8F5E9,stroke:#1B5E20,stroke-width:1px
    classDef external fill:#FFF3E0,stroke-dasharray: 5 5,stroke:#E65100,stroke-width:1px
```
