How victor-antoniassi/junior_analytics_engineer_test_01 works
Key Components to Include:
Data Sources Layer
• Two main data source categories:
• Students Data (2021-2022)
• Schools Data (2021-2022)
Data Processing Pipeline
a. Quality Check Stage
• Connection to Google Sheets for manual verification
• Python/Pandas processing component
• Header validation script component
b. Data Preparation Stage
• Data cleaning scripts
• Unidecode processing component
• Manual correction workflows
c. Database Creation Stage
• SQLite database creation process
• Three main tables (educandos, escolas, escolas_educandos)
Output/Storage Layer
• SQLite database as the final storage solution
• Compressed (.gz) storage format
Processing Details
• File format transformations (CSV → TSV → SQLite)
• Where data cleaning occurs
• [Source Data] → [Processing Pipeline] → [Storage]
• ↓ ↑
• [Data Dictionaries] [Google Sheets]
Additional Elements
• A timeline flow from left to right
• Validation checkpoints
• Data transformation points
• Where manual interventions occur
Key Considerations:
Data Flow
• Clear progression from raw data to processed database
• Validation points and quality checks
• Where transformations occur
Processing Scripts
• Both main scripts:
• compare_delimited_file_headers
• datasets_to_sqlite
• Their dependencies
Quality Control
• Validation points
• Where manual corrections occur
• Data dictionary reference points
Final Database Structure
• Three main tables
• Relationships between tables
• Compression step
Remember to:
• Use consistent symbols throughout
• Version/year indicators for the data sources
• Both automated and manual processes clearly
• Major transformation steps