# ambedkargpt/social_media_post_generator architecture

> Architecture diagram of the GitHub repository ambedkargpt/social_media_post_generator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ambedkargpt/social_media_post_generator
- Repository: https://github.com/ambedkargpt/social_media_post_generator

## Overview

This is a political-content platform whose central workflow turns current news and a transcript corpus into profile-specific social posts with transcript references. It also exposes user-facing authentication, profile/questionnaire, news, post, and chat capabilities, plus an operator pipeline for channel ingestion and artifact rebuilding. The README documents the core RAG stages; sampled CLI and retrieval code support the retrieval paths shown. Other API and orchestration internals are represented only where their paths and product roles are clear, not as asserted call chains.

Main entry points: [`backend/pipeline_cli.py`](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline_cli.py) (RAG runner), [`backend/worker/watch_handler.py`](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/worker/watch_handler.py) (Channel watcher), [`frontend/src/App.jsx`](https://github.com/ambedkargpt/social_media_post_generator/blob/main/frontend/src/App.jsx) (Web application), [`backend/streamlit_app/app.py`](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/streamlit_app/app.py) (Pipeline console)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["User interfaces"]
  node_web["Web application<br/>[App.jsx]"]
  node_pipeline_ui["Pipeline console<br/>[app.py]"]
end

subgraph group_app["Accounts and content"]
  node_api["HTTP API"]
  node_services["Account and content services"]
  node_mongo[("Application database<br/>[mongo.py]")]
end

subgraph group_rag["News to posts"]
  node_runner["RAG runner<br/>[pipeline_cli.py]"]
  node_news["News acquisition<br/>[news_fetcher.py]"]
  node_scraper["Article extraction<br/>[news_scraper.py]"]
  node_retriever["Relevant arguments<br/>[retriever.py]"]
  node_generator["Profile post generation<br/>[generator.py]"]
  node_references["Posts and references<br/>[pipeline_cli.py]"]
end

subgraph group_corpus["Corpus and retrieval"]
  node_transcripts["Transcript parsing"]
  node_chunks["Argument chunking<br/>[chunker.py]"]
  node_embedding["Chunk embeddings<br/>[embedder.py]"]
  node_index[("Vector retrieval<br/>[vector_store.py]")]
  node_semrag["Semantic graph retrieval<br/>[runtime.py]"]
end

subgraph group_ops["Channel operations"]
  node_orchestrator["Stage orchestration<br/>[runner.py]"]
  node_worker["Channel watcher<br/>[watch_handler.py]"]
  node_artifacts[("Channel artifacts<br/>[s3_loader.py]")]
end

node_person(("Content creator"))
node_newsapi["NewsAPI"]
node_youtube["YouTube feeds"]
node_gemini["Gemini embeddings"]
node_openai["OpenAI generation"]

node_person -->|"uses"| node_web
node_person -->|"operates"| node_pipeline_ui
node_web -->|"requests"| node_api
node_api -->|"dispatches"| node_services
node_services -->|"reads and writes"| node_mongo
node_runner -->|"fetches"| node_news
node_news -->|"requests news"| node_newsapi
node_runner -->|"scrapes articles"| node_scraper
node_runner -->|"parses corpus"| node_transcripts
node_transcripts -->|"produces chunks"| node_chunks
node_chunks -->|"embeds chunks"| node_embedding
node_embedding -->|"requests embeddings"| node_gemini
node_runner -->|"loads or builds"| node_index
node_runner -->|"retrieves chunks"| node_retriever
node_retriever -->|"searches"| node_index
node_runner -.->|"requests candidates"| node_semrag
node_runner -->|"generates posts"| node_generator
node_generator -->|"requests completion"| node_openai
node_runner -->|"outputs posts and citations"| node_references
node_pipeline_ui -->|"runs flow"| node_runner
node_worker -->|"checks uploads"| node_youtube
node_worker -->|"reads and writes state"| node_artifacts
node_orchestrator -->|"loads artifacts"| node_artifacts

click node_web "https://github.com/ambedkargpt/social_media_post_generator/blob/main/frontend/src/App.jsx"
click node_pipeline_ui "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/streamlit_app/app.py"
click node_api "https://github.com/ambedkargpt/social_media_post_generator/tree/main/backend/api/v1"
click node_services "https://github.com/ambedkargpt/social_media_post_generator/tree/main/backend/services"
click node_mongo "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/db/mongo.py"
click node_runner "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline_cli.py"
click node_news "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/news_fetcher.py"
click node_scraper "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/news_scraper.py"
click node_transcripts "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/transcript_parser.py"
click node_chunks "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/chunker.py"
click node_embedding "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/embedder.py"
click node_index "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/vector_store.py"
click node_semrag "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/semrag/runtime.py"
click node_retriever "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/retriever.py"
click node_generator "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/generator.py"
click node_references "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline_cli.py"
click node_orchestrator "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/orchestration/runner.py"
click node_worker "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/worker/watch_handler.py"
click node_artifacts "https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/core/s3_loader.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web,node_pipeline_ui toneBlue
class node_api,node_services,node_mongo toneAmber
class node_runner,node_news,node_scraper,node_retriever,node_generator,node_references,node_newsapi toneMint
class node_transcripts,node_chunks,node_embedding,node_index,node_semrag toneRose
class node_orchestrator,node_worker,node_artifacts,node_person,node_youtube,node_gemini,node_openai toneIndigo
```

## Components

### User interfaces

- [Web application](https://github.com/ambedkargpt/social_media_post_generator/blob/main/frontend/src/App.jsx): `frontend/src/App.jsx`
- [Pipeline console](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/streamlit_app/app.py): `backend/streamlit_app/app.py`

### Accounts and content

- [HTTP API](https://github.com/ambedkargpt/social_media_post_generator/tree/main/backend/api/v1): `backend/api/v1`
- [Account and content services](https://github.com/ambedkargpt/social_media_post_generator/tree/main/backend/services): `backend/services`
- [Application database](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/db/mongo.py): `backend/db/mongo.py`

### News to posts

- [RAG runner](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline_cli.py): `backend/pipeline_cli.py`
- [News acquisition](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/news_fetcher.py): `backend/pipeline/news_fetcher.py`
- [Article extraction](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/news_scraper.py): `backend/pipeline/news_scraper.py`
- [Relevant arguments](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/retriever.py): `backend/pipeline/retriever.py`
- [Profile post generation](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/generator.py): `backend/pipeline/generator.py`
- [Posts and references](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline_cli.py): `backend/pipeline_cli.py`

### Corpus and retrieval

- [Transcript parsing](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/transcript_parser.py): `backend/pipeline/transcript_parser.py`
- [Argument chunking](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/chunker.py): `backend/pipeline/chunker.py`
- [Chunk embeddings](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/embedder.py): `backend/pipeline/embedder.py`
- [Vector retrieval](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/vector_store.py): `backend/pipeline/vector_store.py`
- [Semantic graph retrieval](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/semrag/runtime.py): `backend/semrag/runtime.py`

### Channel operations

- [Stage orchestration](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/pipeline/orchestration/runner.py): `backend/pipeline/orchestration/runner.py`
- [Channel watcher](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/worker/watch_handler.py): `backend/worker/watch_handler.py`
- [Channel artifacts](https://github.com/ambedkargpt/social_media_post_generator/blob/main/backend/core/s3_loader.py): `backend/core/s3_loader.py`

### Other components

- **Content creator**
- **NewsAPI**
- **YouTube feeds**
- **Gemini embeddings**
- **OpenAI generation**

## Connections

- Content creator → Web application: uses
- Content creator → Pipeline console: operates
- Web application → HTTP API: requests
- HTTP API → Account and content services: dispatches
- Account and content services → Application database: reads and writes
- RAG runner → News acquisition: fetches
- News acquisition → NewsAPI: requests news
- RAG runner → Article extraction: scrapes articles
- RAG runner → Transcript parsing: parses corpus
- Transcript parsing → Argument chunking: produces chunks
- Argument chunking → Chunk embeddings: embeds chunks
- Chunk embeddings → Gemini embeddings: requests embeddings
- RAG runner → Vector retrieval: loads or builds
- RAG runner → Relevant arguments: retrieves chunks
- Relevant arguments → Vector retrieval: searches
- RAG runner → Semantic graph retrieval: requests candidates
- RAG runner → Profile post generation: generates posts
- Profile post generation → OpenAI generation: requests completion
- RAG runner → Posts and references: outputs posts and citations
- Pipeline console → RAG runner: runs flow
- Channel watcher → YouTube feeds: checks uploads
- Channel watcher → Channel artifacts: reads and writes state
- Stage orchestration → Channel artifacts: loads artifacts