How winstonfb/wa-historical-business-license-patterns works
Key Components to Include in the System Design:
Data Collection Layer
• Python scraper (wa_blscrape_by_ubi.py) that collects business license images from WA Department of Revenue
• External data sources:
• WA Secretary of State corporation data
• Dow Jones Industrial Average data
• Census Bureau population data
Data Processing Layer
• Image Processing Pipeline:
• license_image_process.sh (ImageMagick for image processing)
• Tesseract OCR component
• Data Processing Scripts:
• process_license_data.py for license text processing
• WA_business_licenses.R for data combination and analysis
Data Storage Layer
• Raw Data Storage:
• Batch results files (batch_n_results.txt)
• External data files (dow_jones_data.csv, wa_combined_intercensal_pop_data.txt)
• Processed Data Storage:
• Shiny app data files (wa_df.rds, wa_license_activity.rds, wa_sector_data.rds)
Visualization Layer (Shiny Application)
• Server component (server.R)
• UI component (ui.R)
• Helper functions (WA_BLS_helper.R)
Design Guidelines:
• Processing steps in the middle
• End with the Shiny visualization app at the bottom
Technology Labels
• Python
• R/Shiny
• ImageMagick
• Tesseract OCR
• RDS storage
Process Steps
• Web scraping
• Image processing
• OCR conversion
• Data merging
• Statistical analysis
• Interactive visualization