How enriqmancomp/la_crime_project works
Main Components
• Data Source: Although not explicitly included in the file tree, the raw data would be coming from an external source (CSV files, databases, or other feeds) before being landed into your system.
• Data Ingestion: Represented by the “import_data.sql” script which is responsible for importing the raw data into the system’s database.
• Data Cleaning & Transformation:
• “data_cleaning.sql” & “fix_data_type.sql” scripts – these are used to clean and correct the data.
• “create_partitions.sql” – likely used to partition tables for performance or organization.
• “create_views.sql” – used to define views for easier querying of transformed data.
• Exploratory Data Analysis (EDA): The “eda.sql” script executes the analytical queries which result in summarized statistics and reports (as detailed in the README).
• Data Export & Visualization: The “export_data.sql” script exports processed data, which is then visualized in Tableau (the “Book.twb” Tableau workbook).
Relationships and Interactions
• Raw Data flows into the Data Ingestion phase.
• Ingestion creates raw tables in the database.
• The Data Cleaning & Transformation scripts operate on these tables, correcting data types, cleaning records, and creating optimized partitions and views.
• The EDA script then queries these curated data stores for generating statistical summaries.
• Finally, the Export script sends the cleaned, analyzed data into formats consumable by Tableau, which acts as the reporting/visualization layer.
Architectural Patterns & Design Principles
• ETL (Extract, Transform, Load) Process: The structure clearly separates the process of ingestion, cleaning/transforming, and then analysis/export.
• Modularity: Each SQL script addresses a specific part of the data pipeline, which allows for clearer separation of concerns.
• Reporting Layer: Using Tableau indicates an emphasis on interactive data visualization and dashboard reporting.
Relevant Technologies
• SQL (for data manipulation) and an underlying relational database system (unspecified, but could be PostgreSQL, MySQL, etc.).
• Tableau for visualization and reporting.
Instructing the Principal Software Engineer
• Use clear labels to denote each major module: Data Source, Data Ingestion Module (import_data.sql), Data Cleaning & Transformation (data_cleaning.sql, fix_data_type.sql, create_partitions.sql, create_views.sql), Analysis (eda.sql), Data Export (export_data.sql), and Visualization (Tableau Workbook).
• Incorporate any relevant configuration or deployment artifacts if applicable, though in this project the emphasis remains on the SQL-driven ETL and reporting layers.
Summing Up
• Pay special attention to how the SQL scripts work together to prepare the data and how Tableau consumes the results for insightful analytics.
• Though each process is discrete, ensure that all dependencies and interactions between components (such as how cleaned data feeds into both partitioning and view creation, and then into analytical queries) are clearly marked.