# eliasdabbas/advertools architecture

> Architecture diagram of the GitHub repository eliasdabbas/advertools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eliasdabbas/advertools
- Repository: https://github.com/eliasdabbas/advertools

## Overview

##### Project type and purpose
- This is an open-source Python library (advertools) for online marketing data wrangling and analysis.
- It is **not** a full-stack web application but a collection of independent functions and modules that can be imported into user code or invoked via a CLI.
- It follows the UNIX philosophy: small focused functions, text (DataFrame) streams, modular and extensible.

##### File-structure analysis
- Top-level artifacts:
- `.github/`, `tox.ini`, CI configs (GitHub Actions, Travis, CodeQL) – developer pipeline
- `docs/` – Sphinx source + generated HTML – documentation subsystem
- `advertools/` – the core Python package, subdivided into feature modules
- `tests/` – pytest test suite, fixtures, and sample data
- `setup.py`, `pyproject.toml`, `setup.cfg`, `Makefile` – packaging and build scripts
- Within `advertools/`:
- feature modules (`spider.py`, `sitemaps.py`, `serp.py`, `twitter.py`, `youtube.py`, etc.)
- helper modules (`_regex_helpers.py`, `_yt_helpers.py`) and `url_builders.py`
- a CLI entrypoint (`cli.py`) and CLI docs generator (`cli_docs/`)
- `code_recipes/` – examples & extension points
- `pkg_data/` – static data assets (emoji dataset)

##### Key insights from README
- Each function returns a pandas DataFrame or file.
- Integration points: HTTP requests (Scrapy for crawling, requests to APIs), file parsers (logs, XML), DataFrame pipelines.
- UNIX philosophy: do one thing, well-documented, composable.

a. **Main components**
1. User Interface Layer
- Python import API (advertools.*)
- CLI (`advertools cli`)
2. Core Library Layer
- Feature modules (crawl, sitemap, robots, serp, kw_generate, ad_create, ad_from_string, extract, emoji, url_builders, etc.)
- Helper/utility modules (`_regex_helpers.py`, `_yt_helpers.py`)
3. Data I/O & Processing Layer
- HTTP/Scrapy for web crawling & API calls
- File parsers (logs, XML sitemaps, robots.txt)
- pandas DataFrame construction
4. Extension & Example Layer
- `code_recipes/` for custom pipelines
- CLI docs subsystem (`cli_docs/`)
5. Developer Tooling & CI Layer
- Tests (`tests/`) with pytest & sample data
- Documentation build (Sphinx in `docs/`)
- CI/CD pipelines (GitHub Actions, Travis, CodeQL)
- Packaging & distribution (`setup.py`, PyPI)

b. **Component relationships & interactions**
- Extension layer (`code_recipes`) depends on Core Library
- CI/CD pipelines orchestrate Tests, Docs build, Packaging

c. **Architectural patterns & design principles**
- **Modular**: each function lives in its own module, importable independently
- **Layered**: clear separation between user interface, core logic, I/O processing, and developer tooling
- **Plugin-style**: code_recipes illustrate custom pipelines and extendability
- **Pipeline**: data flows from HTTP/File → parsers → DataFrame → user analysis

d. **Technologies & frameworks**
- Python 3.x, pandas, Scrapy, requests
- Sphinx for docs, pytest for tests
- setuptools / pyproject.toml for packaging
- GitHub Actions, Travis CI, CodeQL for continuous integration
- ReadTheDocs / PyPI for hosting

##### Final advice
- **essential architectural elements** without clutter – group similar modules
- **how data moves** through the system, from user invocation to DataFrame output
- **modularity** and **extensibility** – this is the library’s key value

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interface Layer"
        API["advertools/__init__.py"]:::user
        CLI["advertools/cli.py"]:::user
    end

    subgraph "Core Library Layer"
        spider["advertools/spider.py"]:::core
        header_spider["advertools/header_spider.py"]:::core
        image_spider["advertools/image_spider.py"]:::core
        crawlytics["advertools/crawlytics.py"]:::core
        sitemaps["advertools/sitemaps.py"]:::core
        robots["advertools/robotstxt.py"]:::core
        serp["advertools/serp.py"]:::core
        url_builders["advertools/url_builders.py"]:::core
        urlytics["advertools/urlytics.py"]:::core
        twitter["advertools/twitter.py"]:::core
        youtube["advertools/youtube.py"]:::core
        ad_create["advertools/ad_create.py"]:::core
        ad_from_string["advertools/ad_from_string.py"]:::core
        kw_generate["advertools/kw_generate.py"]:::core
        extract["advertools/extract.py"]:::core
        word_tokenize["advertools/word_tokenize.py"]:::core
        word_frequency["advertools/word_frequency.py"]:::core
        stopwords["advertools/stopwords.py"]:::core
        logs["advertools/logs.py"]:::core
        emoji["advertools/emoji.py"]:::core
        dns["advertools/reverse_dns_lookup.py"]:::core
    end

    subgraph "Helper/Utility Modules"
        regex_helpers["advertools/_regex_helpers.py"]:::core
        yt_helpers["advertools/_yt_helpers.py"]:::core
        regex_module["advertools/regex.py"]:::core
    end

    subgraph "Data I/O & Processing Layer"
        http["HTTP/Scrapy & requests"]:::io
        file_parsers["File parsers (logs, XML)"]:::io
        dataframe["pandas DataFrame"]:::io
        external_web["Websites"]:::extsys
        twitter_api["Twitter API"]:::extsys
        youtube_api["YouTube API"]:::extsys
        log_files["Log files"]:::extsys
    end

    subgraph "Extension & Example Layer"
        recipes["advertools/code_recipes/"]:::ext
        cli_docs["advertools/cli_docs/"]:::ext
    end

    subgraph "Developer Tooling & CI Layer"
        tests["tests/"]:::dev
        docs["docs/"]:::dev
        ci[".github/workflows/ci.yml\n.travis.yml\n.github/workflows/codeql.yml"]:::dev
        packaging["setup.py\npyproject.toml\nsetup.cfg\nMakefile"]:::dev
    end

    %% Data flow
    API -->|"import API"| spider
    CLI -->|"invoke CLI"| spider
    HTTP_DEP[end User]:::user --> API
    HTTP_DEP --> CLI

    external_web --> http
    twitter_api --> http
    youtube_api --> http
    log_files --> file_parsers

    http -->|"crawling/API calls"| spider
    http --> serp
    file_parsers --> sitemaps
    file_parsers --> robots
    file_parsers --> logs

    spider --> dataframe
    header_spider --> dataframe
    image_spider --> dataframe
    crawlytics --> dataframe
    sitemaps --> dataframe
    robots --> dataframe
    serp --> dataframe
    url_builders --> dataframe
    urlytics --> dataframe
    twitter --> dataframe
    youtube --> dataframe
    ad_create --> dataframe
    ad_from_string --> dataframe
    kw_generate --> dataframe
    extract --> dataframe
    word_tokenize --> dataframe
    word_frequency --> dataframe
    stopwords --> dataframe
    logs --> dataframe
    emoji --> dataframe
    dns --> dataframe

    dataframe -->|"DataFrame output"| HTTP_DEP

    %% Dependencies
    spider -.-> regex_helpers
    serp -.-> regex_helpers
    youtube -.-> yt_helpers
    header_spider -.-> regex_helpers
    image_spider -.-> regex_helpers
    extract -.-> regex_helpers
    url_builders -.-> regex_module

    %% Extension dependencies
    recipes --> spider
    recipes --> serp
    cli_docs --> CLI

    %% CI/CD workflows
    tests -->|"run tests on"| packaging
    ci --> tests
    ci --> docs
    packaging -->|"package release"| docs

    %% Click Events
    click API "https://github.com/eliasdabbas/advertools/blob/master/advertools/__init__.py"
    click CLI "https://github.com/eliasdabbas/advertools/blob/master/advertools/cli.py"
    click spider "https://github.com/eliasdabbas/advertools/blob/master/advertools/spider.py"
    click header_spider "https://github.com/eliasdabbas/advertools/blob/master/advertools/header_spider.py"
    click image_spider "https://github.com/eliasdabbas/advertools/blob/master/advertools/image_spider.py"
    click crawlytics "https://github.com/eliasdabbas/advertools/blob/master/advertools/crawlytics.py"
    click sitemaps "https://github.com/eliasdabbas/advertools/blob/master/advertools/sitemaps.py"
    click robots "https://github.com/eliasdabbas/advertools/blob/master/advertools/robotstxt.py"
    click serp "https://github.com/eliasdabbas/advertools/blob/master/advertools/serp.py"
    click url_builders "https://github.com/eliasdabbas/advertools/blob/master/advertools/url_builders.py"
    click urlytics "https://github.com/eliasdabbas/advertools/blob/master/advertools/urlytics.py"
    click twitter "https://github.com/eliasdabbas/advertools/blob/master/advertools/twitter.py"
    click youtube "https://github.com/eliasdabbas/advertools/blob/master/advertools/youtube.py"
    click ad_create "https://github.com/eliasdabbas/advertools/blob/master/advertools/ad_create.py"
    click ad_from_string "https://github.com/eliasdabbas/advertools/blob/master/advertools/ad_from_string.py"
    click kw_generate "https://github.com/eliasdabbas/advertools/blob/master/advertools/kw_generate.py"
    click extract "https://github.com/eliasdabbas/advertools/blob/master/advertools/extract.py"
    click word_tokenize "https://github.com/eliasdabbas/advertools/blob/master/advertools/word_tokenize.py"
    click word_frequency "https://github.com/eliasdabbas/advertools/blob/master/advertools/word_frequency.py"
    click stopwords "https://github.com/eliasdabbas/advertools/blob/master/advertools/stopwords.py"
    click logs "https://github.com/eliasdabbas/advertools/blob/master/advertools/logs.py"
    click emoji "https://github.com/eliasdabbas/advertools/blob/master/advertools/emoji.py"
    click dns "https://github.com/eliasdabbas/advertools/blob/master/advertools/reverse_dns_lookup.py"
    click regex_helpers "https://github.com/eliasdabbas/advertools/blob/master/advertools/_regex_helpers.py"
    click yt_helpers "https://github.com/eliasdabbas/advertools/blob/master/advertools/_yt_helpers.py"
    click regex_module "https://github.com/eliasdabbas/advertools/blob/master/advertools/regex.py"
    click recipes "https://github.com/eliasdabbas/advertools/tree/master/advertools/code_recipes/"
    click cli_docs "https://github.com/eliasdabbas/advertools/tree/master/advertools/cli_docs/"
    click tests "https://github.com/eliasdabbas/advertools/tree/master/tests/"
    click docs "https://github.com/eliasdabbas/advertools/tree/master/docs/"
    click ci "https://github.com/eliasdabbas/advertools/blob/master/.github/workflows/ci.yml"
    click packaging "https://github.com/eliasdabbas/advertools/blob/master/setup.py"

    %% Styles
    classDef user fill:#CCF,stroke:#333,stroke-width:1px
    classDef core fill:#CFC,stroke:#333,stroke-width:1px
    classDef io fill:#FC9,stroke:#333,stroke-width:1px
    classDef ext fill:#FFA,stroke:#333,stroke-width:1px
    classDef dev fill:#DDA,stroke:#333,stroke-width:1px
    classDef extsys fill:#FEC,stroke:#333,stroke-width:1px
```
