# rafapolo/mostre architecture

> Architecture diagram of the GitHub repository rafapolo/mostre, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rafapolo/mostre
- Repository: https://github.com/rafapolo/mostre

## Overview

##### Project type and purpose
- This is a monolithic, full-stack Ruby on Rails application called “Mostre!me” that combines:
- A link-tracking/reference creator (“Mostre!me Links”)
- A data miner and explorer for Ministry of Culture (SalicNet) and Ministry of Education (eMec) datasets
- Interactive visualizations (graphs) of public data
- Key use cases:
- Generate short link references and record click metadata
- Crawl and normalize public data from Cultura and eMec systems
- Serve dashboards and D3-based visualizations

##### File structure
- Top-level Rails MVC layout:
- app/controllers → web controllers (Links, Candidatos, Cultura, Educação, etc.)
- app/models → ActiveRecord models (Projeto, Entidade, Doação, etc.)
- app/views → Haml/Slime templates & partials
- app/assets → JavaScript (CoffeeScript/D3), CSS/SASS, images
- Background processing:
- app/workers/updates_worker.rb
- lib/crawler → custom crawlers for Cultura (minc.rb), MEC
- rake tasks (lib/tasks/*.rake) for TSE, MEC, MINC data pulls
- Configuration & orchestration:
- config/routes.rb → HTTP endpoints & resourceful routes
- config/database.yml, puma.rb, initializers → middleware, asset pipeline
- Persistence:
- db/migrate & schema.rb → definitions for entidades, projetos, cursos, doadores, etc.
- db/seeds.rb → initial data load
- Public/exported assets:
- public/data/*.json, .gexf → precomputed graph data
- doc/ → ERDs and .dot files for offline reference

##### Insights from the README
- Three main modules launched over time:
1. Link-tracker (2009)
2. Cultura data miner/visualization (2010 → integrated 2013)
3. Educação data miner (2015)
- Emphasis on privacy (no visitor PII), open data, and iterative/interactive queries

a. Main components to show:
- Web server (Puma or Passenger)
- Rails application (MVC layers)
- Relational database (PostgreSQL/MySQL)
- Background job processor (Sidekiq/Resque or custom worker)
- Data crawler subsystem (Rake tasks & lib/crawler modules)
- External data sources (SalicNet API/Pages, eMec system, TSE)
- Front-end client (browser + D3.js visualizations)
- Public assets directory (static JSON/GEXF files)
b. Relationships & data flows:
1. User → HTTP request → Rails router → controller → model → DB → view → HTML/CoffeeScript
2. Front-end JS (app.js, grafo.coffee, chart.js) requests JSON/GEXF from public/data or via API controllers
3. Scheduled Rake tasks call crawler modules → parse external HTML/JSON → write into DB
4. updates_worker picks up DB changes → regenerates graph exports (public/data/*.json/.gexf)
c. Architectural patterns & principles:
- Classic MVC layered architecture
- Asset pipeline for JS/CSS/SASS
- Background job processing for ETL/crawling
- Separation of concerns: link-tracking vs. crawler vs. visualization
d. Key technologies & libraries:
- Ruby 2.x, Rails 4.x
- ActiveRecord, PostgreSQL/MySQL
- Puma (or Passenger), Rack
- CoffeeScript, D3.js (for interactive graphs)
- Haml/Slime, SASS
- Graphviz (.dot), GEXF exports, JSON endpoints
- Rake, God (crawler.god), Sidekiq/Resque (or custom job runner)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    subgraph "External Systems" 
        style ExternalSystems fill:none,stroke:#999,stroke-dasharray: 5 5
        SalicNet["SalicNet API/Pages"]:::external
        eMec["eMec System"]:::external
        TSE["TSE"]:::external
    end

    %% Background Job & Crawler
    subgraph "Background & Crawler Layer"
        direction TB
        Crawler["Crawler Modules\n(lib/crawler)"]:::bg
        RakeMINC["Rake Task: minc.rake"]:::bg
        RakeMEC["Rake Task: mec.rake"]:::bg
        RakeTSE["Rake Task: tse.rake"]:::bg
        Crawler -.->|invoked by| RakeMINC
        Crawler -.->|invoked by| RakeMEC
        Crawler -.->|invoked by| RakeTSE
        Crawler -->|writes data| DB[(PostgreSQL/MySQL)]:::db
        DB -->|triggers| UpdatesWorker["updates_worker"]:::bg
        UpdatesWorker -->|exports graph| PublicData["public/data/*.json/.gexf"]:::assets
    end

    %% Web Tier
    subgraph "Web Tier"
        direction TB
        Puma["Puma Web Server"]:::web
    end

    %% Rails Application
    subgraph "Rails Application (MVC)"
        direction TB
        Controllers["Controllers\n(app/controllers)"]:::app
        Models["Models\n(app/models)"]:::app
        Views["Views & Templates\n(app/views)"]:::app
        Assets["Frontend Assets\n(app/assets/javascripts, stylesheets)"]:::frontend
    end

    %% Client
    Browser["Browser + D3.js"]:::frontend

    %% Database
    DB

    %% Public Assets
    PublicData

    %% Relationships
    Browser -->|HTTP requests| Puma
    Puma -->|routes to| Controllers
    Controllers -->|query/update| Models
    Models -->|reads/writes| DB
    Controllers -->|render| Views
    Views -->|serve HTML/CoffeeScript| Browser
    Browser -->|fetch graph data| PublicData
    Browser -->|API JSON| Controllers
    Assets -->|delivered via| Views

    SalicNet -->|external data| Crawler
    eMec -->|external data| Crawler
    TSE -->|external data| Crawler

    %% Click Events
    click Puma "https://github.com/rafapolo/mostre/blob/master/config/puma.rb"
    click Controllers "https://github.com/rafapolo/mostre/tree/master/app/controllers"
    click Models "https://github.com/rafapolo/mostre/tree/master/app/models"
    click Views "https://github.com/rafapolo/mostre/tree/master/app/views"
    click DB "https://github.com/rafapolo/mostre/blob/master/config/database.yml"
    click UpdatesWorker "https://github.com/rafapolo/mostre/blob/master/app/workers/updates_worker.rb"
    click Crawler "https://github.com/rafapolo/mostre/blob/master/lib/crawler/minc.rb"
    click RakeMINC "https://github.com/rafapolo/mostre/blob/master/lib/tasks/minc.rake"
    click RakeMEC "https://github.com/rafapolo/mostre/blob/master/lib/tasks/mec.rake"
    click RakeTSE "https://github.com/rafapolo/mostre/blob/master/lib/tasks/tse.rake"
    click PublicData "https://github.com/rafapolo/mostre/blob/master/public/data/area.json"

    %% Styles
    classDef web fill:#e0f7fa,stroke:#006064,color:#006064
    classDef app fill:#bbdefb,stroke:#0d47a1,color:#0d47a1
    classDef db fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20
    classDef bg fill:#ffe0b2,stroke:#e65100,color:#e65100
    classDef external fill:none,stroke:#999,stroke-dasharray: 5 5,color:#555
    classDef assets fill:#fff9c4,stroke:#f57f17,color:#f57f17
    classDef frontend fill:#f3e5f5,stroke:#4a148c,color:#4a148c
```
