# centicmurcia/nlp-consentimientos-medicos architecture

> Architecture diagram of the GitHub repository centicmurcia/nlp-consentimientos-medicos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/centicmurcia/nlp-consentimientos-medicos
- Repository: https://github.com/centicmurcia/nlp-consentimientos-medicos

## Overview

##### Project type and purpose
- This is a Python-based NLP web application (built with Streamlit) whose goal is to measure the legibility and understandability of medical consent texts in Spanish and Catalan.

##### File structure
- Top level:
- data/ – raw consent texts and vocabulary lists (frequency tables for Spanish/Catalan, domain‐specific lexica)
- PDF2Text/ – (optional) tools to convert PDFs to plain text
- readme.md, requirements.txt
- webapp_dev/ – the Streamlit application and core modules:
– app.py (Streamlit UI layer)
– data_processing.py (orchestration of preprocessing, metrics, ML)
– freeling_connection.py (integration with external Freeling analyzer)
– legibilidad.py, specialized_language.py, separasilabas.py, nal.py (linguistic‐metric calculators)
– machine_learning.py (PCA, t-SNE routines)
– experimentacion.ipynb (development/experimentation notebook)

##### README
- Confirms Streamlit front end, metric extraction, and ML grouping.
- Mentions low-power deployment on share.streamlit.io – a monolithic Python  application.

##### Main components & interactions
a. User Interface (Streamlit app)
– Receives file uploads via browser
– Displays progress bars and visualizations (tables, scatterplots)
b. Ingestion & Preprocessing
– If PDF input: PDF2Text module → plain text
– DataProcessing orchestrator calls:
- separasilabas.py (syllable splitting)
- freeling_connection.py (POS tagging, morphological analysis via Freeling server)
- specialized_language.py / nal.py (domain‐specific term counts)
– All modules read from/write to in-memory structures or temp files
c. Metric Computation
– legibilidad.py calculates readability scores (e.g., Flesch, SMOG adapted to Spanish/Catalan)
– Vocabulary statistics use data/vocab-lists as reference
d. Machine Learning
– machine_learning.py performs dimensionality reduction (PCA, t-SNE) on feature vectors
– Produces clusters/groups for visualization
e. Data Sources & Persistence
– data/ folder (raw texts for demo or batch processing)
– No persistent database; ephemeral storage only
f. External Service
– Freeling morphological analyzer (could be a REST or local server)

##### Architectural patterns & design principles
- Layered/Modular monolith: clear separation between UI, orchestration, NLP modules, ML modules, data sources, and external service
- Single‐process deployment (Streamlit) but logically layered
- Separation of concerns: each Python module has one responsibility
- Dependencies declared in requirements.txt

a. Components
- “Streamlit App (app.py)” – UI Layer
- “Data Processing Service” – orchestrator (data_processing.py)
- “NLP Modules” – separasilabas, legibilidad, specialized_language, nal
- “ML Module” – PCA/t-SNE (machine_learning.py)
- “External Freeling Service”
- “Data Sources” – data/, vocab-lists, optional PDF2Text
b. Relationships & Data flow
1. User → Streamlit App (upload files)
2. Streamlit App → Data Processing Service (invoke with file handles)
3. Data Processing Service → PDF2Text (if needed) → returns text
4. Data Processing Service → NLP Modules → produce feature vectors
5. NLP Modules → optionally call Freeling Service (POS tags)
6. Data Processing Service → ML Module → clusters + coordinates
7. ML Module → returns results to Streamlit App → Browser (visualization)
c. Important patterns
- Orchestration: single entrypoint (data_processing.py) calling specialized modules
- Data folders and vocab lists shown as read-only resources
d. Technologies
- Python 3.x, Streamlit, Pandas, scikit-learn, Freeling, (possibly NLTK/spaCy), requirements.txt for dependency management

##### Final advice
- Be as detailed as possible: break out each Python module and its responsibility
- Clearly call out external dependencies and static data resources
- Use layering: UI layer at top, processing layer in middle, data/externals at bottom
- This yields an accurate, maintainable view of the system suitable for architects and developers alike.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers: UI -> Processing -> Data/External
    Browser["Browser / User"]:::frontend
    App["Streamlit App\n(app.py)"]:::internal
    DataProc["Data Processing Orchestrator\n(data_processing.py)"]:::internal

    subgraph "Ingestion & NLP Modules"
        PDF2Text["PDF2Text Module"]:::internal
        Syllable["Syllable Splitting\n(separasilabas.py)"]:::internal
        FreelingConn["Freeling External Service\n(freeling_connection.py)"]:::external
        DomainLang["Domain-Specific Language\n(specialized_language.py)"]:::internal
        NAL["NAL Terminology Counts\n(nal.py)"]:::internal
        Readability["Readability Metrics\n(legibilidad.py)"]:::internal
    end

    ML["Machine Learning Module\n(machine_learning.py)"]:::internal

    subgraph "Static Data Sources"
        DataRaw["Raw Text Data\n(data/)"]:::data
        Vocab["Vocabulary Lists\n(data/vocab-lists/)"]:::data
        StaticAssets["Static Assets\n(imgs/)"]:::data
    end

    Requirements["Dependencies\n(requirements.txt)"]:::storage
    Docs["Documentation\n(readme.md, readme_original.md)"]:::storage
    Notebook["Experimentation Notebook\n(experimentacion.ipynb)"]:::storage

    %% Connections
    Browser -->|"uploads files"| App
    App -->|"invokes"| DataProc
    DataProc -->|"if PDF"| PDF2Text
    PDF2Text -->|"plain text"| DataProc

    DataProc -->|"text"| Syllable
    DataProc -->|"text"| DomainLang
    DataProc -->|"text"| NAL
    DataProc -->|"text"| Readability

    Syllable -->|"syllables"| DataProc
    DomainLang -->|"term counts"| DataProc
    NAL -->|"term counts"| DataProc
    Readability -->|"readability scores"| DataProc

    DataProc -->|"requests POS tags"| FreelingConn
    FreelingConn -->|"tags/morphology"| DataProc

    DataProc -->|"feature vectors"| ML
    ML -->|"clusters & coords"| App
    App -->|"visualizations"| Browser

    %% Data sources usage
    DataRaw --- DataProc
    Vocab --- DomainLang
    Vocab --- Readability

    %% Dev & auxiliary
    Requirements --- App
    Docs --- App
    StaticAssets --- App
    Notebook -.-> App

    %% Click events
    click App "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/app.py"
    click DataProc "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/data_processing.py"
    click PDF2Text "https://github.com/centicmurcia/nlp-consentimientos-medicos/tree/master/PDF2Text/"
    click Syllable "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/separasilabas.py"
    click FreelingConn "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/freeling_connection.py"
    click DomainLang "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/specialized_language.py"
    click NAL "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/nal.py"
    click Readability "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/legibilidad.py"
    click ML "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/machine_learning.py"
    click Notebook "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/webapp_dev/experimentacion.ipynb"
    click DataRaw "https://github.com/centicmurcia/nlp-consentimientos-medicos/tree/master/data/"
    click Vocab "https://github.com/centicmurcia/nlp-consentimientos-medicos/tree/master/data/vocab-lists/"
    click StaticAssets "https://github.com/centicmurcia/nlp-consentimientos-medicos/tree/master/imgs/"
    click Requirements "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/requirements.txt"
    click Docs "https://github.com/centicmurcia/nlp-consentimientos-medicos/blob/master/readme.md"

    %% Styles
    classDef internal fill:#cce5ff,stroke:#3399ff,color:#000
    classDef data fill:#d4edda,stroke:#28a745,color:#000
    classDef external fill:#ffe5b4,stroke:#ff8c00,color:#000
    classDef storage fill:#e2e3e5,stroke:#6c757d,color:#000
    classDef frontend fill:#d1ecf1,stroke:#17a2b8,color:#000
```
