How centicmurcia/nlp-consentimientos-medicos works
Project type and purpose
• This is a Python-based NLP web application (built with Streamlit) whose goal is to measure the legibility and understandability of medical consent texts in Spanish and Catalan.
File structure
• Top level:
• data/ – raw consent texts and vocabulary lists (frequency tables for Spanish/Catalan, domain‐specific lexica)
• PDF2Text/ – (optional) tools to convert PDFs to plain text
• readme.md, requirements.txt
• webapp_dev/ – the Streamlit application and core modules:
– app.py (Streamlit UI layer)
– data_processing.py (orchestration of preprocessing, metrics, ML)
– freeling_connection.py (integration with external Freeling analyzer)
– legibilidad.py, specialized_language.py, separasilabas.py, nal.py (linguistic‐metric calculators)
– machine_learning.py (PCA, t-SNE routines)
– experimentacion.ipynb (development/experimentation notebook)
README
• Confirms Streamlit front end, metric extraction, and ML grouping.
• Mentions low-power deployment on share.streamlit.io – a monolithic Python application.
Main components & interactions
a. User Interface (Streamlit app)
– Receives file uploads via browser
– Displays progress bars and visualizations (tables, scatterplots)
b. Ingestion & Preprocessing
– If PDF input: PDF2Text module → plain text
– DataProcessing orchestrator calls:
• separasilabas.py (syllable splitting)
• freeling_connection.py (POS tagging, morphological analysis via Freeling server)
• specialized_language.py / nal.py (domain‐specific term counts)
– All modules read from/write to in-memory structures or temp files
c. Metric Computation
– legibilidad.py calculates readability scores (e.g., Flesch, SMOG adapted to Spanish/Catalan)
– Vocabulary statistics use data/vocab-lists as reference
d. Machine Learning
– machine_learning.py performs dimensionality reduction (PCA, t-SNE) on feature vectors
– Produces clusters/groups for visualization
e. Data Sources & Persistence
– data/ folder (raw texts for demo or batch processing)
– No persistent database; ephemeral storage only
f. External Service
– Freeling morphological analyzer (could be a REST or local server)
Architectural patterns & design principles
• Layered/Modular monolith: clear separation between UI, orchestration, NLP modules, ML modules, data sources, and external service
• Single‐process deployment (Streamlit) but logically layered
• Separation of concerns: each Python module has one responsibility
• Dependencies declared in requirements.txt
a. Components
• “Streamlit App (app.py)” – UI Layer
• “Data Processing Service” – orchestrator (data_processing.py)
• “NLP Modules” – separasilabas, legibilidad, specialized_language, nal
• “ML Module” – PCA/t-SNE (machine_learning.py)
• “External Freeling Service”
• “Data Sources” – data/, vocab-lists, optional PDF2Text
b. Relationships & Data flow
1. User → Streamlit App (upload files)
2. Streamlit App → Data Processing Service (invoke with file handles)
3. Data Processing Service → PDF2Text (if needed) → returns text
4. Data Processing Service → NLP Modules → produce feature vectors
5. NLP Modules → optionally call Freeling Service (POS tags)
6. Data Processing Service → ML Module → clusters + coordinates
7. ML Module → returns results to Streamlit App → Browser (visualization)
c. Important patterns
• Orchestration: single entrypoint (data_processing.py) calling specialized modules
• Data folders and vocab lists shown as read-only resources
d. Technologies
• Python 3.x, Streamlit, Pandas, scikit-learn, Freeling, (possibly NLTK/spaCy), requirements.txt for dependency management
Final advice
• Be as detailed as possible: break out each Python module and its responsibility
• Clearly call out external dependencies and static data resources
• Use layering: UI layer at top, processing layer in middle, data/externals at bottom
• This yields an accurate, maintainable view of the system suitable for architects and developers alike.