How rafapolo/mostre works
Project type and purpose
• This is a monolithic, full-stack Ruby on Rails application called “Mostre!me” that combines:
• A link-tracking/reference creator (“Mostre!me Links”)
• A data miner and explorer for Ministry of Culture (SalicNet) and Ministry of Education (eMec) datasets
• Interactive visualizations (graphs) of public data
• Key use cases:
• Generate short link references and record click metadata
• Crawl and normalize public data from Cultura and eMec systems
• Serve dashboards and D3-based visualizations
File structure
• Top-level Rails MVC layout:
• app/controllers → web controllers (Links, Candidatos, Cultura, Educação, etc.)
• app/models → ActiveRecord models (Projeto, Entidade, Doação, etc.)
• app/views → Haml/Slime templates & partials
• app/assets → JavaScript (CoffeeScript/D3), CSS/SASS, images
• Background processing:
• app/workers/updates_worker.rb
• lib/crawler → custom crawlers for Cultura (minc.rb), MEC
• rake tasks (lib/tasks/*.rake) for TSE, MEC, MINC data pulls
• Configuration & orchestration:
• config/routes.rb → HTTP endpoints & resourceful routes
• config/database.yml, puma.rb, initializers → middleware, asset pipeline
• Persistence:
• db/migrate & schema.rb → definitions for entidades, projetos, cursos, doadores, etc.
• db/seeds.rb → initial data load
• Public/exported assets:
• public/data/*.json, .gexf → precomputed graph data
• doc/ → ERDs and .dot files for offline reference
Insights from the README
• Three main modules launched over time:
1. Link-tracker (2009)
2. Cultura data miner/visualization (2010 → integrated 2013)
3. Educação data miner (2015)
• Emphasis on privacy (no visitor PII), open data, and iterative/interactive queries
a. Main components to show:
• Web server (Puma or Passenger)
• Rails application (MVC layers)
• Relational database (PostgreSQL/MySQL)
• Background job processor (Sidekiq/Resque or custom worker)
• Data crawler subsystem (Rake tasks & lib/crawler modules)
• External data sources (SalicNet API/Pages, eMec system, TSE)
• Front-end client (browser + D3.js visualizations)
• Public assets directory (static JSON/GEXF files)
b. Relationships & data flows:
1. User → HTTP request → Rails router → controller → model → DB → view → HTML/CoffeeScript
2. Front-end JS (app.js, grafo.coffee, chart.js) requests JSON/GEXF from public/data or via API controllers
3. Scheduled Rake tasks call crawler modules → parse external HTML/JSON → write into DB
4. updates_worker picks up DB changes → regenerates graph exports (public/data/*.json/.gexf)
c. Architectural patterns & principles:
• Classic MVC layered architecture
• Asset pipeline for JS/CSS/SASS
• Background job processing for ETL/crawling
• Separation of concerns: link-tracking vs. crawler vs. visualization
d. Key technologies & libraries:
• Ruby 2.x, Rails 4.x
• ActiveRecord, PostgreSQL/MySQL
• Puma (or Passenger), Rack
• CoffeeScript, D3.js (for interactive graphs)
• Haml/Slime, SASS
• Graphviz (.dot), GEXF exports, JSON endpoints
• Rake, God (crawler.god), Sidekiq/Resque (or custom job runner)