How sobdurated/com6064-pipeline works
This is a Turkish social-post data-processing pipeline, runnable locally or through a dashboard-facing server. Its principal workflow fetches posts, preprocesses text, runs transformer and LLM sentiment analysis, then supports geospatial output and province-level aggregation. The runner dynamically executes the steps in order and shares a MongoDB context. The graph preserves documented stages and server capabilities; some step internals and their database interactions were not shown, so uncertain links are omitted or marked unverified.
Main entry points: pipeline.py Pipeline Runner · preprocess.py Text Preprocessing · server.py Run and Log Server · sentiment_analysis.py Transformer Sentiment
Components 13
Pipeline Control
- Run and Log Server
server.py - Pipeline Runner
pipeline.py
Collection and Preparation
- Post Fetch
fetch.py - Text Preprocessing
preprocess.py - Turkish Text Cleaner
preprocess.py
Sentiment Analysis
- Transformer Sentiment
sentiment_analysis.py - LLM Sentiment
sentiment_analysis_llm.py
Geospatial and Aggregation
- Geospatial Output
geospatial.py - Sentiment Aggregation
sentiment_aggregation.py
Other components
- Dashboard Client
- MongoDB
- Serper Search API
- ngrok Tunnel
Connections 22
- Dashboard Client → Run and Log Server: triggers runs
server.py - Run and Log Server → Pipeline Runner: launches subprocess
server.py - Pipeline Runner → Post Fetch: executes
pipeline.py - Pipeline Runner → Text Preprocessing: executes
pipeline.py - Pipeline Runner → Transformer Sentiment: executes
pipeline.py - Pipeline Runner → LLM Sentiment: executes
pipeline.py - Pipeline Runner → Geospatial Output: executes
pipeline.py - Pipeline Runner → Sentiment Aggregation: executes
pipeline.py - Post Fetch → Serper Search API: queries
fetch.py - Post Fetch → Text Preprocessing: passes output
- Text Preprocessing → Turkish Text Cleaner: cleans text
preprocess.py - Text Preprocessing → MongoDB: reads and writes posts
preprocess.py - Text Preprocessing → Transformer Sentiment: passes output
- Transformer Sentiment → LLM Sentiment: passes output
- Transformer Sentiment → MongoDB: updates sentiment
sentiment_analysis.py - LLM Sentiment → Geospatial Output: passes output
- LLM Sentiment → MongoDB: uses post records
- Geospatial Output → MongoDB: updates and reads posts
geospatial.py - Geospatial Output → Sentiment Aggregation: passes output
- Sentiment Aggregation → MongoDB: reads posts, upserts aggregates
sentiment_aggregation.py - Run and Log Server → Dashboard Client: streams run status and logs
server.py - Run and Log Server → ngrok Tunnel: optionally exposes
server.py