How camiloandresdg/crime_prediction_la works
Project Type and Purpose
• This is a batch‑oriented machine‑learning pipeline built in Python with PySpark.
• Purpose: Ingest LA crime data, process it, train and run forecasting models, and produce rolling 8‑day predictions by zone.
Main Architectural Components
a. External Data Source
• LA Official Crime Database (e.g., REST API or flat files)
b. Data Ingestion Layer (src/data/data_ingestion.py, scheduler.py)
• Pulls raw data on a scheduled basis
• Writes to raw data storage
c. Storage
• Raw Data Store (e.g., HDFS/S3/local data/raw/)
• Processed Data Store (temporary staging)
• Predictions Store (predictions/ directory)
d. Orchestrator (src/pipeline.py)
• Coordinates stages in sequence: Ingest → Process → Analyze → Model → Export
e. Data Processing Layer
• PySpark Processor (src/processing/spark_processor.py)
• Data Processor (src/processing/data_processor.py) for feature engineering
• Descriptive Analytics (src/analysis/descriptive_analysis.py)
f. Modeling Layer (src/models/crime_predictor.py)
• Crime classification and count regression models
• Zone‑specific model instantiation
g. Scheduler (scheduler.py)
• Cron‑style or Airflow‑style trigger for pipeline.py every 8 days
h. External Clients/Consumers
• BI dashboard or API reading from predictions/
Relationships & Data Flow
• Scheduler → triggers → Data Ingestion
• Data Ingestion → writes → Raw Data Store
• Orchestrator reads Raw Data → passes to → Spark Processor → writes → Processed Data
• Processed Data → fed into → Descriptive Analysis & Feature Engineering
• Engineered Features → input → Crime Predictor → outputs → Predictions Store
• Predictions → consumed by → External Clients
Architectural Patterns & Principles
• Modular ETL Pipeline: clear layering of ingestion, processing, modeling
• Separation of Concerns: each Python module does one job
• Scalability: PySpark cluster for large data volumes
• Scheduling: decoupled from processing logic
Technologies & Frameworks to Annotate
• Python 3.x
• PySpark (Spark Cluster Master & Executors)
• File‑based storage or HDFS/S3
• Virtual Environment / requirements.txt dependencies
• Local scheduler (cron/scheduler.py) or external orchestration