# sam-raiden/cortex-internaltool architecture

> Architecture diagram of the GitHub repository sam-raiden/cortex-internaltool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sam-raiden/cortex-internaltool
- Repository: https://github.com/sam-raiden/cortex-internaltool

## Overview

#### Purpose and Runtime
TamilSh.AI is a backend-only Python POC for collecting Tamil/Tanglish social content, extracting multimodal signals, embedding and clustering it, and exposing trend intelligence. It requires Python 3.9+, PostgreSQL via Docker, and Alembic-managed schema migrations. The API application entry point is `app/main.py`; collection can be invoked through `app/collectors/__main__.py`.

#### Persistence and Domain Model
PostgreSQL is the system of record for source governance, collection runs, raw content, processed signals, clusters, and trend-analysis entities. Database access and ORM schema are centralized in `app/storage/database.py` and `app/models/schema.py`. Schema evolution is controlled by `migrations/env.py` and the versioned migrations under `migrations/versions/`, including clustering and trend-intelligence additions.

#### Source Registry and Run Orchestration
Configured sources enter through JSON definitions such as `config/sources/rss.json` and `config/sources/youtube.json`. `app/services/source_registry.py` governs source registration/state, while `app/services/collection_run.py` tracks observable collection executions. `app/services/pipeline.py` coordinates pipeline stages; `app/services/scheduler.py` is the scheduling boundary for recurring work.

#### Multi-Platform Collection Boundary
Collectors isolate external-platform acquisition behind a common base in `app/collectors/base.py`. Instagram uses Playwright/browser-based public-profile access, authentication, parsing, and cycle execution in `app/collectors/instagram/browser.py`, `app/collectors/instagram/collector.py`, and `app/collectors/instagram/run_cycle.py`. RSS and YouTube have equivalent collector/cycle boundaries in `app/collectors/rss/collector.py` and `app/collectors/youtube/run_cycle.py`. Collected records are deduplicated by external identifiers before downstream processing.

#### Content Normalization and Signal Processing
Raw platform payloads are normalized into a consistent content representation by `app/processing/normalize_raw_content.py` and `app/processing/normalizer.py`. The processing stage assembles captions/text and derives usable signals through `app/processing/processor.py`, `app/processing/text_assembler.py`, and `app/processing/process_signals.py`. Signal eligibility and composition form an explicit quality boundary in `app/processing/audit_signal_eligibility.py` and `app/processing/signal_composer.py`.

#### Multimodal and AI Enrichment
Media-derived content is handled separately from collector adapters: extraction/validation in `app/processing/media_extractor.py` and `app/processing/media_validation.py`, OCR in `app/processing/ocr.py`, and speech transcription in `app/processing/asr.py`. Language detection and hashtag extraction enrich Tamil/Tanglish text in `app/processing/language_detector.py` and `app/processing/hashtag_extractor.py`. LLM enrichment is isolated behind `app/services/llm_client.py` and applied by `app/processing/llm_enrichment.py`, keeping the external model-provider boundary distinct from core processing.

#### Semantic Clustering and Trend Serving
Eligible text is transformed into multilingual sentence embeddings by `app/processing/build_embeddings.py` and `app/processing/embeddings.py`. HDBSCAN clustering is executed through `app/clustering/run_clustering.py` and `app/clustering/hdbscan_runner.py`; parameter evaluation is separated in `app/clustering/evaluate_parameters.py`. Trend interpretation is produced in `app/processing/trend_intelligence.py` and served through versioned API routes in `app/api/v1.py`, `app/api/trends.py`, and `app/api/endpoints/collection_runs.py`, with read-model assembly in `app/services/trend_view.py`.

Main entry points: [`app/storage/database.py`](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/storage/database.py) (PostgreSQL), [`app/processing/processor.py`](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/processor.py) (Signal processing), [`app/services/pipeline.py`](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/pipeline.py) (Pipeline coordinator), [`app/services/source_registry.py`](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/source_registry.py) (Source registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_scheduler["Scheduler<br/>recurring-work boundary<br/>[scheduler.py]"]
  node_pipeline["Pipeline coordinator<br/>stage orchestrator<br/>[pipeline.py]"]
  node_source_registry["Source registry<br/>source governance<br/>[source_registry.py]"]
  node_collection_runs["Collection runs<br/>run tracking<br/>[collection_run.py]"]
end

subgraph group_collection["Collection"]
  node_source_definitions["Source definitions<br/>JSON configuration<br/>[rss.json]"]
  node_collector_base["Collector boundary<br/>common collector base<br/>[base.py]"]
  node_platform_collectors["Platform collectors<br/>[collector.py]"]
end

subgraph group_processing["Processing &amp; Enrichment"]
  node_normalization["Content normalization<br/>raw-content normalizer"]
  node_signal_processing["Signal processing<br/>caption and signal pipeline<br/>[processor.py]"]
  node_multimodal_enrichment["Multimodal enrichment<br/>[media_extractor.py]"]
  node_llm_enrichment{{"LLM enrichment<br/>model-provider boundary<br/>[llm_client.py]"}}
end

subgraph group_intelligence["Clustering &amp; Trends"]
  node_embeddings["Multilingual embeddings<br/>semantic vectors"]
  node_clustering["HDBSCAN clustering<br/>cluster execution<br/>[run_clustering.py]"]
  node_trend_intelligence["Trend intelligence<br/>trend interpretation"]
end

subgraph group_delivery["API &amp; Persistence"]
  node_api_app["API application<br/>FastAPI app<br/>[main.py]"]
  node_api_routes["Trend API routes<br/>versioned routes<br/>[v1.py]"]
  node_trend_view["Trend read model<br/>view service<br/>[trend_view.py]"]
  node_postgres[("PostgreSQL<br/>system of record<br/>[database.py]")]
  node_orm_schema["Domain schema<br/>ORM models<br/>[schema.py]"]
  node_migrations["Schema migrations<br/>Alembic<br/>[env.py]"]
end

node_api_app -->|"mounts"| node_api_routes
node_api_routes -->|"serves trends"| node_trend_view
node_trend_view -->|"reads"| node_postgres
node_orm_schema -->|"maps"| node_postgres
node_migrations -->|"evolves"| node_orm_schema
node_scheduler -->|"triggers"| node_pipeline
node_pipeline -->|"selects sources"| node_source_registry
node_pipeline -->|"tracks execution"| node_collection_runs
node_source_definitions -->|"registers"| node_source_registry
node_source_registry -->|"configures"| node_collector_base
node_collector_base -->|"implements"| node_platform_collectors
node_platform_collectors -->|"deduplicated raw content"| node_postgres
node_pipeline -->|"orchestrates"| node_normalization
node_postgres -->|"raw records"| node_normalization
node_normalization -->|"canonical content"| node_signal_processing
node_multimodal_enrichment -->|"media signals"| node_signal_processing
node_llm_enrichment -->|"enrichment"| node_signal_processing
node_signal_processing -->|"processed signals"| node_postgres
node_signal_processing -->|"eligible text"| node_embeddings
node_embeddings -->|"vectors"| node_clustering
node_clustering -->|"clusters"| node_trend_intelligence
node_trend_intelligence -->|"trend entities"| node_postgres

click node_api_app "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/main.py"
click node_api_routes "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/api/v1.py"
click node_trend_view "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/trend_view.py"
click node_postgres "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/storage/database.py"
click node_orm_schema "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/models/schema.py"
click node_migrations "https://github.com/sam-raiden/cortex-internaltool/blob/master/migrations/env.py"
click node_scheduler "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/scheduler.py"
click node_pipeline "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/pipeline.py"
click node_source_registry "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/source_registry.py"
click node_collection_runs "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/collection_run.py"
click node_source_definitions "https://github.com/sam-raiden/cortex-internaltool/blob/master/config/sources/rss.json"
click node_collector_base "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/collectors/base.py"
click node_platform_collectors "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/collectors/instagram/collector.py"
click node_normalization "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/normalize_raw_content.py"
click node_signal_processing "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/processor.py"
click node_multimodal_enrichment "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/media_extractor.py"
click node_llm_enrichment "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/llm_client.py"
click node_embeddings "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/build_embeddings.py"
click node_clustering "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/clustering/run_clustering.py"
click node_trend_intelligence "https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/trend_intelligence.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scheduler,node_pipeline,node_source_registry,node_collection_runs toneBlue
class node_source_definitions,node_collector_base,node_platform_collectors toneAmber
class node_normalization,node_signal_processing,node_multimodal_enrichment,node_llm_enrichment toneMint
class node_embeddings,node_clustering,node_trend_intelligence toneRose
class node_api_app,node_api_routes,node_trend_view,node_postgres,node_orm_schema,node_migrations toneIndigo
```

## Components

### Orchestration

- [Scheduler](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/scheduler.py): recurring-work boundary · `app/services/scheduler.py`
- [Pipeline coordinator](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/pipeline.py): stage orchestrator · `app/services/pipeline.py`
- [Source registry](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/source_registry.py): source governance · `app/services/source_registry.py`
- [Collection runs](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/collection_run.py): run tracking · `app/services/collection_run.py`

### Collection

- [Source definitions](https://github.com/sam-raiden/cortex-internaltool/blob/master/config/sources/rss.json): JSON configuration · `config/sources/rss.json`
- [Collector boundary](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/collectors/base.py): common collector base · `app/collectors/base.py`
- [Platform collectors](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/collectors/instagram/collector.py): Instagram, RSS & YouTube adapters · Instagram uses browser-based public-profile collection. · `app/collectors/instagram/collector.py`

### Processing & Enrichment

- [Content normalization](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/normalize_raw_content.py): raw-content normalizer · `app/processing/normalize_raw_content.py`
- [Signal processing](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/processor.py): caption and signal pipeline · Assembles text and applies eligibility and composition rules. · `app/processing/processor.py`
- [Multimodal enrichment](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/media_extractor.py): media, OCR, ASR & language · `app/processing/media_extractor.py`
- [LLM enrichment](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/llm_client.py): model-provider boundary · `app/services/llm_client.py`

### Clustering & Trends

- [Multilingual embeddings](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/build_embeddings.py): semantic vectors · `app/processing/build_embeddings.py`
- [HDBSCAN clustering](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/clustering/run_clustering.py): cluster execution · Parameter evaluation is kept separate from execution. · `app/clustering/run_clustering.py`
- [Trend intelligence](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/processing/trend_intelligence.py): trend interpretation · `app/processing/trend_intelligence.py`

### API & Persistence

- [API application](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/main.py): FastAPI app · Backend entry point. · `app/main.py`
- [Trend API routes](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/api/v1.py): versioned routes · `app/api/v1.py`
- [Trend read model](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/services/trend_view.py): view service · `app/services/trend_view.py`
- [PostgreSQL](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/storage/database.py): system of record · Stores sources, runs, content, signals, clusters, and trends. · `app/storage/database.py`
- [Domain schema](https://github.com/sam-raiden/cortex-internaltool/blob/master/app/models/schema.py): ORM models · `app/models/schema.py`
- [Schema migrations](https://github.com/sam-raiden/cortex-internaltool/blob/master/migrations/env.py): Alembic · `migrations/env.py`

## Connections

- API application → Trend API routes: mounts
- Trend API routes → Trend read model: serves trends
- Trend read model → PostgreSQL: reads
- Domain schema → PostgreSQL: maps
- Schema migrations → Domain schema: evolves
- Scheduler → Pipeline coordinator: triggers
- Pipeline coordinator → Source registry: selects sources
- Pipeline coordinator → Collection runs: tracks execution
- Source definitions → Source registry: registers
- Source registry → Collector boundary: configures
- Collector boundary → Platform collectors: implements
- Platform collectors → PostgreSQL: deduplicated raw content
- Pipeline coordinator → Content normalization: orchestrates
- PostgreSQL → Content normalization: raw records
- Content normalization → Signal processing: canonical content
- Multimodal enrichment → Signal processing: media signals
- LLM enrichment → Signal processing: enrichment
- Signal processing → PostgreSQL: processed signals
- Signal processing → Multilingual embeddings: eligible text
- Multilingual embeddings → HDBSCAN clustering: vectors
- HDBSCAN clustering → Trend intelligence: clusters
- Trend intelligence → PostgreSQL: trend entities