# bachtly/affiliate-crawl-sales-bot architecture

> Architecture diagram of the GitHub repository bachtly/affiliate-crawl-sales-bot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bachtly/affiliate-crawl-sales-bot
- Repository: https://github.com/bachtly/affiliate-crawl-sales-bot

## Overview

##### Project type and purpose
- This is an automation/ETL-style bot written in Python that periodically:
- Crawls products from e-commerce sites (Shopee, Lazada)
- Fetches affiliate links via third-party services (Accesstrade)
- Generates marketing content from templates
- Publishes content to social media (Facebook)
- It uses a scheduler to orchestrate end-to-end “flows,” persists data in MongoDB, and supports monitoring via Docker-Compose + Loki.

##### File structure
- Top-level entrypoint: app.py (launches scheduler and flows)
- config/ holds configuration logic (config.py)
- flows/ defines business flows (e.g. FlowDocongso.py, FlowTuivinu_deprecated.py)
- services/ implements domain services, subdivided by adapter type (adapters/, link/, marketing/, post/, product/, publishing/)
- models/ defines MongoDB document schemas (ProductModel.py, deprecated PublishingModel)
- utils/ contains reusable helpers; per instructions only include Crawler here
- monitor/ holds monitoring stack (docker-compose.yaml, Loki configs)
- requirements.txt, Makefile, and Docker artifacts govern build and runtime

##### README for additional insights
- Design Overview section clarifies two sub-flows: product flow (crawl → affiliate link → content → DB) and publishing flow (retrieve from DB → publish)
- Mentions scheduler as the entrypoint for each “flow”

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestrator Layer
    subgraph "Orchestrator Layer"
        direction TB
        App["app.py\n(Launch Scheduler)"]:::orchestrator
        SchedulerUtil["Scheduler"]:::orchestrator
    end

    %% Flow Layer
    subgraph "Flow Layer"
        direction TB
        FlowDocongso["FlowDocongso"]:::flow
        FlowTuivinu["FlowTuivinu_deprecated"]:::flow
    end

    %% Service Layer
    subgraph "Service Layer"
        direction TB

        subgraph "Adapters"
            direction TB
            AccesstradeAdapter["AccesstradeAdapter"]:::adapter
            FacebookPageAdapter["FacebookPageAdapter"]:::adapter
            ShopeeAdapter["ShopeeAdapter"]:::adapter
        end

        subgraph "Link Generators"
            direction TB
            AccesstradeLink["AccesstradeLink"]:::link
        end

        subgraph "Marketing Content"
            direction TB
            FacebookNewFeedComment["FacebookNewFeedComment"]:::marketing
            FacebookNewfeedAdapter["FacebookNewfeedAdapter"]:::marketing
        end

        subgraph "Product Crawlers"
            direction TB
            ShopeeProduct["ShopeeProduct"]:::product
            LazadaProductAdapter["LazadaProductAdapter_deprecated"]:::product
            LazadaProduct["LazadaProduct_deprecated"]:::product
        end

        subgraph "Publishing Modules"
            direction TB
            FacebookPagePublisher["FacebookPagePublisher"]:::publishing
            ProductPicker["ProductPicker"]:::publishing
        end
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        ProductModel["ProductModel"]:::data
        PublishingModel["PublishingModel_deprecated"]:::data
        Orm["Orm"]:::data
    end

    %% Utilities Layer
    subgraph "Utilities Layer"
        direction TB
        Crawler["Crawler"]:::utility
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        EcomSites["E-commerce Sites"]:::external
        AffiliateSrv["Affiliate Service"]:::external
        FBAPI["Facebook API"]:::external
        MongoDB["MongoDB"]:::external
        Monitoring["Loki/Grafana"]:::external
    end

    %% Connections
    App -->|"launches"| SchedulerUtil
    SchedulerUtil -->|"triggers"| FlowDocongso
    SchedulerUtil -->|"triggers"| FlowTuivinu

    FlowDocongso -->|"crawl products"| ShopeeAdapter
    FlowDocongso -->|"affiliate link"| AccesstradeAdapter
    FlowDocongso -->|"generate link"| AccesstradeLink
    FlowDocongso -->|"generate content"| FacebookNewFeedComment
    FlowDocongso -->|"save to DB"| Orm

    FlowTuivinu -->|"crawl products"| LazadaProductAdapter
    FlowTuivinu -->|"affiliate link"| AccesstradeAdapter
    FlowTuivinu -->|"generate content"| FacebookNewfeedAdapter
    FlowTuivinu -->|"save to DB"| Orm

    ShopeeAdapter -->|calls| EcomSites
    LazadaProductAdapter -->|calls| EcomSites
    AccesstradeAdapter -->|calls| AffiliateSrv
    FacebookPageAdapter -->|calls| FBAPI
    ShopeeProduct -->|calls| EcomSites
    LazadaProduct -->|calls| EcomSites

    AccesstradeLink -->|uses| AffiliateSrv
    FacebookNewFeedComment -->|uses| FBAPI
    FacebookNewfeedAdapter -->|uses| FBAPI

    Orm -->|writes| ProductModel
    Orm -->|writes| PublishingModel
    ProductModel -->|stores in| MongoDB
    PublishingModel -->|stores in| MongoDB

    FlowTuivinu -->|"publish flow"| FacebookPagePublisher
    FacebookPagePublisher -->|selects| ProductPicker
    ProductPicker -->|reads| ProductModel
    FacebookPagePublisher -->|publishes| FBAPI

    %% Utilities logging
    App -->|"logs/metrics"| Monitoring
    SchedulerUtil -->|logs/metrics| Monitoring
    FlowDocongso -->|logs/metrics| Monitoring
    FlowTuivinu -->|logs/metrics| Monitoring

    %% Click events
    click App "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/app.py"
    click SchedulerUtil "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/utils/scheduler/Scheduler.py"
    click FlowDocongso "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/flows/docongso/FlowDocongso.py"
    click FlowTuivinu "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/flows/tuivinu/FlowTuivinu_deprecated.py"
    click AccesstradeAdapter "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/adapters/AccesstradeAdapter.py"
    click FacebookPageAdapter "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/adapters/FacebookPageAdapter.py"
    click ShopeeAdapter "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/adapters/ShopeeAdapter.py"
    click AccesstradeLink "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/link/accesstrade/AccesstradeLink.py"
    click FacebookNewFeedComment "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/marketing/facebook/FacebookNewFeedComment.py"
    click FacebookNewfeedAdapter "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/marketing/facebook/FacebookNewfeedAdapter.py"
    click ShopeeProduct "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/product/shopee/ShopeeProduct.py"
    click LazadaProductAdapter "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/product/lazada/LazadaProductAdapter_deprecated.py"
    click LazadaProduct "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/product/lazada/LazadaProduct_deprecated.py"
    click FacebookPagePublisher "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/publishing/facebook/FacebookPagePublisher.py"
    click ProductPicker "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/services/publishing/facebook/ProductPicker.py"
    click ProductModel "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/models/ProductModel.py"
    click PublishingModel "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/models/PublishingModel_deprecated.py"
    click Orm "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/utils/database/Orm.py"
    click Crawler "https://github.com/bachtly/affiliate-crawl-sales-bot/blob/master/utils/crawler/Crawler.py"

    %% Styles
    classDef orchestrator fill:#D6E9F8,stroke:#31708F,color:#31708F;
    classDef flow fill:#E2F0D9,stroke:#3C763D,color:#3C763D;
    classDef service fill:#FDF5E6,stroke:#8A6D3B,color:#8A6D3B;
    classDef adapter fill:#D9EDF7,stroke:#31708F,color:#31708F;
    classDef link fill:#FCF8E3,stroke:#8A6D3B,color:#8A6D3B;
    classDef marketing fill:#F2DEDE,stroke:#A94442,color:#A94442;
    classDef product fill:#DFF0D8,stroke:#3C763D,color:#3C763D;
    classDef publishing fill:#F5F5F5,stroke:#666666,color:#666666;
    classDef data fill:#FAFAD2,stroke:#8A6D3B,color:#8A6D3B;
    classDef utility fill:#D3D3D3,stroke:#555555,color:#555555;
    classDef external fill:none,stroke-dasharray: 5 5,stroke:#555555,color:#555555;
```
