How smohith2105/crime-analysis-buffalo works
Project Type & Purpose
• This is a full-stack data analysis and forecasting system centered on crime analysis in Buffalo. It integrates crime risk prediction, forecasting using Facebook Prophet, threat score assessment with XGBoost, and neighborhood trend analysis using SVM.
• The project leverages various data sources (CSV, GeoJSON, pickle files for models and scalers), processes this data, stores it (including in an SQLite database), and then visualizes the outcomes through an interactive Streamlit dashboard.
Main Components
a. Data Sources & Storage:
• Data Files: Represent the raw and preprocessed data files located in the "data/" directory. This includes subdirectories for forecast, risk, threat, and trends. Include key file types such as CSVs, GeoJSON, and pickle files.
• Database: Although not directly seen in the file tree, the README mentions a "db" folder with an SQLite database (incidents.db). Show this as a central repository for processed crime data.
b. Data Ingestion & Processing:
• This layer processes raw data inputs (from CSVs, etc.) and prepares them for analysis, including cleaning and transformation.
c. Experimentation & Model Development:
• Notebooks: The "exp/" and "src/" directories host Jupyter notebooks that support experimentation with different models:
• Forecasting notebook using Facebook Prophet (exp/forecast.ipynb)
• Risk prediction notebook using Random Forest (exp/risk.ipynb)
• Threat assessment notebook using XGBoost (exp/threat.ipynb)
• Trend analysis notebook using SVM (exp/trends.ipynb)
• That these notebooks represent the research and model training phase that produces the trained predictive models.
d. Model Artifacts:
• Model Files: Pickle files (stored in data/risk, data/threat, and data/trends) imply storage of pre-trained models (Random Forest, XGBoost, and SVM) and scalers. Represent these as “Model Artifacts” which are loaded for prediction.
e. Application / Presentation Layer (Streamlit Web Interface):
• Main App: The "app.py" acts as the Streamlit dashboard which integrates all functionalities. Show how it ties together the predictive models, visualization (using Plotly, Pydeck, etc.), and user inputs.
• Visualization Components: Highlight the specific visual elements:
• Heatmap for crime risk predictions,
• Line charts for crime forecasting,
• Gauge charts for threat score,
• 3D scatter and map visualizations for neighborhood crime trends.
• User Interaction: Indicate that the user can interact with the parameters (e.g., selecting neighborhoods, days, hours) which feed into the models.
Relationships and Interactions
• Data flows from the Data Sources to the Data Preparation layer where scripts process and load data into the database.
• Experimentation notebooks read the prepared data, generate predictive models, and export model artifacts.
• The Streamlit app (app.py) acts as the orchestrator by:
• Fetching data from the SQLite database,
• Loading the model artifacts (using Joblib or pickle),
• Accepting user input,
• Running prediction functions (risk prediction, forecast, threat assessment, crime trends analysis),
• Displaying the results via visual components.
Architectural Patterns & Design Principles
• Separation of Concerns: Different modules are separated clearly—data ingestion, processing, model training, and UI.
• Modular Design: The project is structured by function (data, experiments, scripts, application) and includes distinct model components.
• Real-Time Interaction: The Streamlit app demonstrates a real-time prediction and visualization pipeline that brings together various machine learning models.
• Reusability: Model artifacts (pickle files) are created and reused in multiple parts of the system.