marcom-meniac/reddit-insights

Purpose and Execution

This is a read-only Reddit marketing-intelligence pipeline. It collects public posts from configured technology communities, enriches them with rule-based topic labels and VADER sentiment, then persists results to PostgreSQL for downstream analysis.

main.py is the application entry point and coordinates logging, Reddit collection, processing, and database persistence.

Paths: main.py, README.md, config.example.py

Reddit Collection Boundary

The Reddit adapter encapsulates access to Reddit’s official API, expected to use PRAW and configured credentials. It should be the only component that communicates with Reddit and must preserve the repository’s read-only constraint: no posting, voting, messaging, moderation, or other mutations.

Collection scope—such as monitored subreddits and query limits—is configuration-driven rather than embedded in the orchestration layer.

Paths: reddit.py, config.example.py, main.py

Processing and Enrichment

Raw Reddit post content flows through two independent enrichment stages:

• Topic classification applies a rule-based technology taxonomy to identify themes such as trends, adoption, or customer pain points.

• Sentiment analysis uses VADER to score discussion sentiment.

These stages should remain pure or near-pure transformations over collected text, allowing their rules/models to evolve independently of API and persistence code.

Paths: classifier.py, sentiment.py, main.py

Domain Model Contract

models.py defines the internal representation of collected and enriched Reddit data. It is the contract between ingestion, analysis, and storage: Reddit-specific source fields are normalized into application records, then classification and sentiment outputs are attached before persistence.

Changes to stored fields or enrichment outputs should be coordinated through this model boundary.

Paths: models.py, reddit.py, database.py

Persistence Layer

The database module owns PostgreSQL and SQLAlchemy integration. It provides the boundary between application records and durable relational storage, including connection/session lifecycle and writes. Database schema assets live under sql/reddit, although the exact SQL files are not exposed in the provided tree.

The intended control flow is: retrieve posts → normalize/model → classify and score sentiment → save to PostgreSQL.

Paths: database.py, models.py, sql/reddit

Shared Runtime Concerns

Structured logging is centralized rather than scattered through collectors and processors. Shared utilities support cross-cutting normalization or helper behavior without becoming a second orchestration path. Runtime dependencies include Python 3.11+, PRAW, SQLAlchemy, PostgreSQL, VADER Sentiment, and python-dotenv.

Local output is reserved under output/; it is not the primary system of record, which is PostgreSQL.

Paths: logging_config.py, utils.py, requirements.txt

Operational Boundaries

Secrets and environment-specific database/API settings are expected outside versioned source via a copied config.py, derived from config.example.py. The application is a batch-style Python process invoked with python main.py, not a web service. External failure domains are Reddit API availability/rate limits and PostgreSQL connectivity; orchestration should log and isolate failures at those boundaries.

Paths: config.example.py, main.py, logging_config.py