# marcu-loreto/ead_trabalho_final architecture

> Architecture diagram of the GitHub repository marcu-loreto/ead_trabalho_final, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marcu-loreto/ead_trabalho_final
- Repository: https://github.com/marcu-loreto/ead_trabalho_final

## Overview

##### Project type and purpose
- From the README (“EAD_trabalho_Final – Trabalho da disciplina_Extração_Automatica_Dados”) this appears to be a data-extraction pipeline built as a student’s final project. Its goals are likely to connect to one or more data sources (web pages, documents, APIs), run extraction/parsing routines, and persist or expose the cleaned data.

##### (hypothetically) the file structure
- In this case we only have a top-level README.md. That suggests the project may not yet have clearly separated modules or folders. However, a data-extraction system typically splits into:
- Connectors (source fetchers)
- Parsers / transformers
- Storage layer
- API or user interface
- Utilities (configuration, logging, scheduling)

##### README for additional insights
- The README only gives the title and course. No further technical details. Therefore we will propose a canonical architecture for an automatic data-extraction project.

##### Detail is essential
- Middleware, queues or caches that decouple stages.
- Configuration stores and logging sinks as supporting infrastructure.
- Don’t omit the scheduler/orchestrator—this is key in a data-extraction workflow.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    subgraph "Data Sources"
        Web["Web Pages"]:::external
        Files["Files/Docs"]:::external
        ThirdAPI["External APIs"]:::external
    end

    %% Scheduler / Orchestrator
    Scheduler["Scheduler/Orchestrator"]:::scheduler

    %% Extraction Layer
    subgraph "Extraction"
        Scraper["Web Scraper"]:::internal
        PDFParser["PDF Parser"]:::internal
        APIFetcher["API Connector"]:::internal
        Queue["Message Queue"]:::storage
    end

    %% Transformation Layer
    subgraph "Transformation"
        Normalizer["Normalizer"]:::internal
        Validator["Validator"]:::internal
        Enricher["Enricher"]:::internal
    end

    %% Persistence Layer
    subgraph "Persistence"
        DB["Database (Postgres/MongoDB)"]:::storage
        S3["File Storage (S3)"]:::storage
    end

    %% Presentation Layer
    subgraph "Presentation"
        API["REST API (FastAPI)"]:::internal
        Dashboard["Dashboard/UI"]:::internal
        CLI["CLI"]:::internal
    end

    %% Supporting Services
    Config["Configuration"]:::support
    Logging["Logging"]:::support

    %% Data Flow
    Web --> Scraper
    Files --> PDFParser
    ThirdAPI --> APIFetcher

    Scheduler -->|"Triggers"| Scraper
    Scheduler -->|"Triggers"| PDFParser
    Scheduler -->|"Triggers"| APIFetcher

    Scraper --> Queue
    PDFParser --> Queue
    APIFetcher --> Queue

    Queue --> Normalizer
    Normalizer --> Validator
    Validator --> Enricher
    Enricher --> DB
    Enricher --> S3

    DB --> API
    S3 --> API

    API --> Dashboard
    API --> CLI

    %% Configuration & Logging
    Config -.-> Scraper
    Config -.-> PDFParser
    Config -.-> APIFetcher
    Config -.-> Normalizer
    Config -.-> Validator
    Config -.-> Enricher
    Config -.-> DB
    Config -.-> API

    Logging -.-> Scraper
    Logging -.-> PDFParser
    Logging -.-> APIFetcher
    Logging -.-> Normalizer
    Logging -.-> Validator
    Logging -.-> Enricher
    Logging -.-> DB
    Logging -.-> API

    %% Styles
    classDef external fill:#a1f0a1,stroke:#27a327,color:#000
    classDef internal fill:#a1c4f0,stroke:#2271d1,color:#000
    classDef storage fill:#f0c48c,stroke:#d18d1f,color:#000
    classDef scheduler fill:#e4a1f0,stroke:#9f22d1,color:#000
    classDef support fill:#e0e0e0,stroke:#888888,color:#000
```
