How carobs9/segregation-madrid works
Main Components/Modules
• Mobility Data Extraction:
• Folder: mobility_data
• Purpose: Extracts raw trip data (get_viajes.py) and retrieves/cleans geographical information (get_geometrias.py).
• Data Preprocessing & Cleaning for Segregation Analysis:
• Folder: segregation_indices
• Components: preprocessing.py, assortativity_matrix.py, morans_i.py, weight_matrix_analysis.py, and associated utilities.
• Purpose: Cleans income and geographic data, computes segregation measures, and prepares data for further analysis.
• Trip and Mobility Analysis:
• Folder: trip_analysis
• Components: trip_count.py, trip_distances.py, alongside configuration and utility scripts.
• Purpose: Calculates trip counts and distances based on socio-economic stratifications.
• Network Analysis and Graph Modeling:
• Folder: network
• Components: get_graph.py (building the network), regressions.py, visualization.py, and modules for handling adjacency matrices and graphs.
• Purpose: Converts processed mobility data into a graph structure (network) and performs regression analyses to study weight metrics between districts.
• Descriptive and Exploratory Analysis:
• Folder: data_overview
• Components: main_descriptives.py, viajes_stats.py, figures generation, etc.
• Purpose: Provides general summaries and exploratory visualizations of both mobility and income data.
• Logging and Configuration:
• Folders/Files: logs directory and multiple config.py files within modules.
• Purpose: Captures logs for various stages of analysis; centralizes configuration for reproducibility.
• Dependencies/Environment:
• Files: requirements.txt
• Purpose: Ensures that all needed libraries (Python 3.11 and third-party packages) are installed for each module.
Relationships and Data Flows
• Raw data is fetched and cleaned in the mobility_data module.
• Outputs from mobility_data feed directly into segregation_indices and trip_analysis.
• The segregation_indices and trip_analysis modules preprocess and analyze the data to generate metrics (e.g., assortativity, Moran’s I, trip counts, trip distances).
• Processed data from these modules becomes the basis for constructing the mobility network in the network module.
• The network module then analyzes relationships (e.g., in-weight, out-weight) and creates visualizations via regression and graph models.
• Parallel to specialized analysis, the data_overview module offers overall descriptive statistics and visual summaries.
Architectural Patterns and Principles
• Modular Pipeline: Each folder represents a stage in the analysis workflow. Each component (data extraction, cleaning, network analysis) is self-contained with its own configuration, making the system extendable.
• Separation of Concerns: Clear division between raw data ingestion, preprocessing, analytical computations, and visualization.
• Config-driven Execution: With multiple config.py files, the system expects central configuration parameters (e.g., data paths, study type selections) to be used consistently across modules.
• Logging and Output Generation: Dedicated logs and figures directories ensure reproducibility and traceability of calculation steps.