sam-raiden/cortex-internaltool

Purpose and Runtime

TamilSh.AI is a backend-only Python POC for collecting Tamil/Tanglish social content, extracting multimodal signals, embedding and clustering it, and exposing trend intelligence. It requires Python 3.9+, PostgreSQL via Docker, and Alembic-managed schema migrations. The API application entry point is app/main.py; collection can be invoked through app/collectors/__main__.py.

Persistence and Domain Model

PostgreSQL is the system of record for source governance, collection runs, raw content, processed signals, clusters, and trend-analysis entities. Database access and ORM schema are centralized in app/storage/database.py and app/models/schema.py. Schema evolution is controlled by migrations/env.py and the versioned migrations under migrations/versions/, including clustering and trend-intelligence additions.

Source Registry and Run Orchestration

Configured sources enter through JSON definitions such as config/sources/rss.json and config/sources/youtube.json. app/services/source_registry.py governs source registration/state, while app/services/collection_run.py tracks observable collection executions. app/services/pipeline.py coordinates pipeline stages; app/services/scheduler.py is the scheduling boundary for recurring work.

Multi-Platform Collection Boundary

Collectors isolate external-platform acquisition behind a common base in app/collectors/base.py. Instagram uses Playwright/browser-based public-profile access, authentication, parsing, and cycle execution in app/collectors/instagram/browser.py, app/collectors/instagram/collector.py, and app/collectors/instagram/run_cycle.py. RSS and YouTube have equivalent collector/cycle boundaries in app/collectors/rss/collector.py and app/collectors/youtube/run_cycle.py. Collected records are deduplicated by external identifiers before downstream processing.

Content Normalization and Signal Processing

Raw platform payloads are normalized into a consistent content representation by app/processing/normalize_raw_content.py and app/processing/normalizer.py. The processing stage assembles captions/text and derives usable signals through app/processing/processor.py, app/processing/text_assembler.py, and app/processing/process_signals.py. Signal eligibility and composition form an explicit quality boundary in app/processing/audit_signal_eligibility.py and app/processing/signal_composer.py.

Multimodal and AI Enrichment

Media-derived content is handled separately from collector adapters: extraction/validation in app/processing/media_extractor.py and app/processing/media_validation.py, OCR in app/processing/ocr.py, and speech transcription in app/processing/asr.py. Language detection and hashtag extraction enrich Tamil/Tanglish text in app/processing/language_detector.py and app/processing/hashtag_extractor.py. LLM enrichment is isolated behind app/services/llm_client.py and applied by app/processing/llm_enrichment.py, keeping the external model-provider boundary distinct from core processing.

Semantic Clustering and Trend Serving

Eligible text is transformed into multilingual sentence embeddings by app/processing/build_embeddings.py and app/processing/embeddings.py. HDBSCAN clustering is executed through app/clustering/run_clustering.py and app/clustering/hdbscan_runner.py; parameter evaluation is separated in app/clustering/evaluate_parameters.py. Trend interpretation is produced in app/processing/trend_intelligence.py and served through versioned API routes in app/api/v1.py, app/api/trends.py, and app/api/endpoints/collection_runs.py, with read-model assembly in app/services/trend_view.py.