samirdash4u/dr_dc_switchover

Purpose and operating model

This is a shell-script automation repository for Appsone/HEAL DC-to-DR switchover. It supports isolated environments through an offline artifact boundary: DC exports state, an operator securely transfers a tarball, and DR restores or validates that state. DNS switching remains external/manual.

Primary entry points are the ordered DC and DR scripts, with a likely top-level coordinator at orchestrator.sh. Certificate import is separately invoked through importcert.sh.

Configuration and shared runtime

The scripts are configuration-driven and intended to use the same configuration in both sites:

• config/config.sh supplies database, backup/restore, and environment parameters.

• config/dcnomadjoblist.txt and config/drnomadjoblist.txt define expected Nomad workloads per site.

• config/workerentry.sh defines connector database endpoint updates.

• common/logger.sh and common/utils.sh provide shared script behavior.

The runtime is Bash/shell on hosts with access to local DC or DR infrastructure and command-line clients for the platform services.

DC readiness gate

The DC pipeline begins with prerequisite validation in dc/01_prerequisites.sh. This is the safety boundary before producing a switchover bundle: it checks availability/readiness of Consul, Nomad, MySQL, OpenSearch, disk capacity, and required Nomad jobs, using configured expected job lists such as config/dcnomadjoblist.txt.

Failure here should prevent downstream export stages from producing a potentially incomplete recovery artifact.

DC export pipeline

DC state is collected in explicit ordered stages:

• Database backup: dc/02_db_dump.sh

• Consul KV export: dc/03_consul_export.sh

• Nomad job/image definition export: dc/04_nomad_export.sh

• Bundle construction: dc/05_package_artifacts.sh

These stages transform live platform state into portable files, then package them as a timestamped switchover_bundle_<timestamp>.tar.gz. The tarball is the repository’s principal cross-environment contract; no direct DC-to-DR network flow is required or assumed.

Transfer boundary

Artifact movement is intentionally outside automation. An operator transfers the generated bundle to DR through an approved secure mechanism. This separates DC credentials and live service access from DR application, and allows the two sites to execute independently.

DR application and reconciliation pipeline

DR processing is also ordered:

• DR readiness validation: dr/01_prerequisites.sh

• Artifact extraction: dr/02_unpack.sh

• Database restore: dr/03_db_restore.sh

• Connector receiver endpoint rewrite: dr/04_update_entry.sh

• Consul comparison/reconciliation reporting: dr/05_consul_match.sh

• Nomad image/version comparison reporting: dr/06_nomad_match.sh

dr/03_db_restore.sh applies the database portion of the transferred state. The later Consul and Nomad stages are described as identifying missing keys and job image mismatches, making them validation/reconciliation boundaries rather than an assumed blind overwrite of DR runtime configuration.

Endpoint and certificate operations

Connector endpoint updates are data-level DR adaptations controlled by config/workerentry.sh and applied by dr/04_update_entry.sh. They support a default endpoint or database-specific overrides while preserving protocol/path behavior.

Certificate handling is an adjacent operational capability: importcert.sh uses certificate definitions in config/certs.sh to fetch/import certificates, skip existing aliases, or overwrite them. It is independent of the artifact pipeline but may be required for service connectivity after switchover.