# apache/bigtop architecture

> Architecture diagram of the GitHub repository apache/bigtop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/bigtop
- Repository: https://github.com/apache/bigtop

## Overview

##### Main components
a. Packaging Layer
- bigtop-packages: RPM/DEB spec files and build scripts for each Hadoop subproject.
- bigtop-toolchain: Puppet manifests to bootstrap developer/build environments (Java, Maven, Gradle, Puppet).
b. Deployment Layer
- bigtop-deploy: Puppet & Juju bundles to configure clusters.
- provisioner/docker: Dockerfiles, Docker Compose, and helper scripts to spin up pseudo-clusters for smoke testing.
c. Testing Layer
- bigtop-test-framework: iTest utilities (Groovy/Java) that provide a harness for integration tests.
- bigtop-tests/test-artifacts: actual smoke-test and integration-test source code per component.
- bigtop-tests/test-execution: Maven POM modules that invoke test-framework against deployed clusters.
d. Build Orchestration
- Root Gradle/Maven build (gradlew, build.gradle, pom.xml, bigtop.bom): ties all modules together.
e. Site/Docs
- src/site + Maven Site plugin: documentation source and generation.

##### Relationships & data flows
a. Build → Package
- Root build invokes bigtop-toolchain to set up environment → bigtop-packages produces .rpm/.deb artifacts.
b. Package → Deploy
- bigtop-deploy (Puppet/Juju) or provisioner/docker pulls packages from local repo → configures Hadoop cluster nodes.
c. Deploy → Test
- bigtop-test-framework + test-execution bring up smoke/integration tests defined in test-artifacts → run against live cluster.
d. Feedback Loop
- Test results feed back into CI (Travis/Jenkins) to validate packaging and deployment.

##### Architectural patterns & technologies
- Modular multi-project build (Gradle + Maven BOM)
- Infrastructure as Code (Puppet, Juju, Docker Compose)
- Test harness in Groovy/Java (iTest framework)
- Continuous Integration pipelines (Travis CI badges, Jenkinsfiles)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Build Orchestration
    subgraph "Build Orchestration"
        RootBuild["Root Gradle/Maven Build"]:::orchestration
    end

    %% Packaging Layer
    subgraph "Packaging Layer"
        Toolchain["bigtop-toolchain: Puppet manifests"]:::packaging
        Packages["bigtop-packages: RPM/DEB specs"]:::packaging
    end

    %% Deployment Layer
    subgraph "Deployment Layer"
        PJ["bigtop-deploy: Puppet & Juju bundles"]:::deployment
        DockerProv["provisioner/docker: Docker Compose & scripts"]:::deployment
    end

    %% Testing Layer
    subgraph "Testing Layer"
        TestFramework["bigtop-test-framework: iTest harness"]:::testing
        TestExec["test-execution: Maven modules"]:::testing
        TestArtifacts["test-artifacts: smoke/integration source"]:::testing
    end

    %% Site/Docs
    Docs["src/site: Maven Site documentation"]:::docs

    %% Continuous Integration
    CI["CI (Travis/Jenkins)"]:::external

    %% Flows
    RootBuild -->|"Gradle/Maven tasks"| Toolchain
    Toolchain -->|"build scripts"| Packages
    Packages -->|"Puppet apply / Juju deploy"| PJ
    Packages -->|"Docker Compose"| DockerProv
    PJ -->|"configures cluster"| TestFramework
    DockerProv -->|"sets up pseudo-cluster"| TestFramework
    TestExec -->|"invokes harness"| TestFramework
    TestFramework -->|"executes tests"| TestArtifacts
    TestExec -->|"collects results"| CI
    CI -->|"feedback loop"| RootBuild
    RootBuild -->|"site generation"| Docs

    %% Click Events
    click RootBuild "https://github.com/apache/bigtop/tree/master/gradlew"
    click Toolchain "https://github.com/apache/bigtop/tree/master/bigtop_toolchain"
    click Packages "https://github.com/apache/bigtop/tree/master/bigtop-packages/src"
    click PJ "https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet"
    click PJ "https://github.com/apache/bigtop/tree/master/bigtop-deploy/juju"
    click DockerProv "https://github.com/apache/bigtop/tree/master/provisioner/docker"
    click TestFramework "https://github.com/apache/bigtop/tree/master/bigtop-test-framework"
    click TestArtifacts "https://github.com/apache/bigtop/tree/master/bigtop-tests/test-artifacts"
    click TestExec "https://github.com/apache/bigtop/tree/master/bigtop-tests/test-execution"
    click Docs "https://github.com/apache/bigtop/tree/master/src/site"

    %% Styles
    classDef packaging fill:#cce5ff,stroke:#0062cc,stroke-width:1px
    classDef deployment fill:#d4edda,stroke:#155724,stroke-width:1px
    classDef testing fill:#fff3cd,stroke:#856404,stroke-width:1px
    classDef orchestration fill:#e2e3e5,stroke:#6c757d,stroke-width:1px
    classDef docs fill:#fff3b0,stroke:#856404,stroke-width:1px
    classDef external fill:#f8d7da,stroke:#721c24,stroke-width:1px
```
