# almuhs/twitterfilter_mongodb architecture

> Architecture diagram of the GitHub repository almuhs/twitterfilter_mongodb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/almuhs/twitterfilter_mongodb
- Repository: https://github.com/almuhs/twitterfilter_mongodb

## Overview

##### Main Components and Their Roles
- Twitter API Connector & Download Module:
- Components: Files download_tweets.py, twitter_keys.py.
- Key Classes: TwitterLogin (handles authentication using Twitter keys) and TwitterQuery (performs various tweet search operations with parameters like keywords, dates, etc.).
- Technologies: Uses tweepy (for API access) and json (for tweet data export).
- Data Ingestion and MongoDB Management:
- Components: mongodb_manager.py, collection_manager.py, collection_query.py.
- Key Classes:
- MongoManager (manages connection to the MongoDB instance).
- DBManager (handles operations on databases such as loading collections from files or cursors, and managing collection lists).
- CollectionManager (manages the creation, deletion, and indexing of MongoDB collections).
- CollectionQuery and CollectionStatistics (support advanced filtering and statistics retrieval on tweet collections).
- Design Elements: This layer translates the JSON tweet data into documents suitable for MongoDB, including necessary conversion (i.e., date format adjustments).
- User Interface and Application Layer:
- Components: app.py and app.ui.
- UI Framework: PyQt5, with the GUI designed via Qt Designer then converted using pyuic5.
- Key Classes: Ui_MainWindow (from app.ui) and the MainWindow class that links the interface with functionality – it connects to MongoDB, manages filter criteria, shows statistics, and manages collection switching.
- Functionality: Allows users to select databases/collections, apply various filters (keyword, date exact/range, hashtag, user, “no retweet”), and obtain real-time tweet statistics.

##### Relationships and Interactions
- Data Flow:
- The Tweet Download Module interacts directly with the Twitter API (via tweepy) to fetch and export tweet data (in JSON format).
- Once the tweet data is generated, the Data Ingestion layer (with DBManager and CollectionManager) reads the JSON file or cursor, converts them appropriately (e.g., date conversion) and loads the individual documents into a MongoDB collection.
- The User Interface consumes the data stored in MongoDB through queries (via CollectionQuery) and displays both statistics (via CollectionStatistics) and filtered tweet subsets.
- Separation of Concerns:
- The system naturally divides into three distinct layers: Data Acquisition (tweet download), Data Persistence/Management (MongoDB and collection operations), and Presentation/Interaction (GUI with PyQt5).

##### Important Architectural Patterns and Principles
- Layered Architecture: Distinct layers manage external API communication, data persistence, and user interaction.
- Modular Design: Classes and modules such as TwitterQuery, MongoManager, and DBManager are kept focused on well-defined tasks, making the system extensible and easier to maintain.
- Event-Driven Interaction in the GUI: The MainWindow class maps widget events (filter selections, button clicks) to backend methods, supporting a responsive interface.

##### Relevant Technologies
- Python 3: The project is implemented entirely in Python.
- Tweepy: For interacting with Twitter’s API.
- PyMongo: For MongoDB interactions.
- PyQt5: For the user interface.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Acquisition Layer
    subgraph "Data Acquisition Layer"
        ExternalTwitter["Twitter API"]:::api
        TLogin["TwitterLogin"]:::api
        TQuery["TwitterQuery"]:::api
    end

    %% Data Persistence Layer
    subgraph "Data Persistence Layer"
        tweetData["Tweet JSON Data"]:::database
        Mongo["MongoManager"]:::database
        CollMgr["CollectionManager"]:::database
        CollQuery["CollectionQuery"]:::database
        CollStats["CollectionStatistics"]:::database
    end

    %% UI/Presentation Layer
    subgraph "UI/Presentation Layer"
        MainWin["MainWindow"]:::ui
        UiMain["Ui_MainWindow"]:::ui
    end

    %% Connections for Data Acquisition
    ExternalTwitter --> TLogin
    ExternalTwitter --> TQuery
    TQuery -->|"exportsJSON"| tweetData

    %% Data Ingestion Flow
    tweetData -->|"loadsData"| Mongo
    Mongo -->|"providesConn"| CollMgr
    Mongo -->|"providesAccess"| CollQuery
    Mongo -->|"providesAccess"| CollStats

    %% UI Interactions with Database
    MainWin -->|"sendsQuery"| CollQuery
    MainWin -->|"requestsStats"| CollStats
    CollQuery -->|"returnsData"| MainWin
    CollStats -->|"returnsStats"| MainWin
    MainWin -->|"rendersUI"| UiMain

    %% Click Events
    click TLogin "https://github.com/almuhs/twitterfilter_mongodb/blob/master/twitter_keys.py"
    click TQuery "https://github.com/almuhs/twitterfilter_mongodb/blob/master/download_tweets.py"
    click Mongo "https://github.com/almuhs/twitterfilter_mongodb/blob/master/mongodb_manager.py"
    click CollMgr "https://github.com/almuhs/twitterfilter_mongodb/blob/master/collection_manager.py"
    click CollQuery "https://github.com/almuhs/twitterfilter_mongodb/blob/master/collection_query.py"
    click MainWin "https://github.com/almuhs/twitterfilter_mongodb/blob/master/app.py"
    click UiMain "https://github.com/almuhs/twitterfilter_mongodb/blob/master/app.ui"

    %% Styles
    classDef api fill:#ffcccc,stroke:#990000,stroke-width:2px;
    classDef database fill:#ccffcc,stroke:#009900,stroke-width:2px;
    classDef ui fill:#cce5ff,stroke:#003399,stroke-width:2px;
```
