How apache/bigtop works
Main components
a. Packaging Layer
• bigtop-packages: RPM/DEB spec files and build scripts for each Hadoop subproject.
• bigtop-toolchain: Puppet manifests to bootstrap developer/build environments (Java, Maven, Gradle, Puppet).
b. Deployment Layer
• bigtop-deploy: Puppet & Juju bundles to configure clusters.
• provisioner/docker: Dockerfiles, Docker Compose, and helper scripts to spin up pseudo-clusters for smoke testing.
c. Testing Layer
• bigtop-test-framework: iTest utilities (Groovy/Java) that provide a harness for integration tests.
• bigtop-tests/test-artifacts: actual smoke-test and integration-test source code per component.
• bigtop-tests/test-execution: Maven POM modules that invoke test-framework against deployed clusters.
d. Build Orchestration
• Root Gradle/Maven build (gradlew, build.gradle, pom.xml, bigtop.bom): ties all modules together.
e. Site/Docs
• src/site + Maven Site plugin: documentation source and generation.
Relationships & data flows
a. Build → Package
• Root build invokes bigtop-toolchain to set up environment → bigtop-packages produces .rpm/.deb artifacts.
b. Package → Deploy
• bigtop-deploy (Puppet/Juju) or provisioner/docker pulls packages from local repo → configures Hadoop cluster nodes.
c. Deploy → Test
• bigtop-test-framework + test-execution bring up smoke/integration tests defined in test-artifacts → run against live cluster.
d. Feedback Loop
• Test results feed back into CI (Travis/Jenkins) to validate packaging and deployment.
Architectural patterns & technologies
• Modular multi-project build (Gradle + Maven BOM)
• Infrastructure as Code (Puppet, Juju, Docker Compose)
• Test harness in Groovy/Java (iTest framework)
• Continuous Integration pipelines (Travis CI badges, Jenkinsfiles)