sumiya-roshan/pfl-ingestion-framework

Purpose and Entry Points

This is a Databricks-oriented, metadata/configuration-driven ingestion framework. It appears designed to ingest from JDBC, MongoDB, S3, and SFTP sources, then persist raw/bronze data and optionally write back to S3.

Primary execution surfaces:

• Python entry point: src/main/main.py

• Databricks job definition: jobs/ingestion_job.json

• Environment-specific deployment/runtime variables: config/.env.dev, config/.env.uat, config/.env.prod

Orchestration and Control Plane

The orchestration layer likely coordinates a run by loading source configuration, selecting a connector, invoking extraction, choosing a writer, and recording audit outcomes.

• src/ingestion/utils/orchestrator.py

• src/ingestion/utils/config_manager.py

• src/main/main.py

Configuration is represented in database DDL rather than hard-coded source definitions, making the framework suitable for onboarding multiple source systems through metadata.

• config/ddl/ingestion_config.sql

• config/ddl/config_source_system.sql

• config/ddl/s3_config_master.sql

Connector Boundary

Connectors share a common abstraction and are selected through a factory. This isolates source-specific connectivity and extraction behavior from orchestration and output handling.

• Base contract: src/ingestion/connectors/base_connector.py

• Selection/creation: src/ingestion/connectors/factory.py

Implemented source adapters define the supported external ingestion boundaries:

• JDBC databases: src/ingestion/connectors/jdbc_connector.py

• MongoDB: src/ingestion/connectors/mongo_connector.py

• Amazon S3: src/ingestion/connectors/s3_connector.py

• SFTP servers: src/ingestion/connectors/sftp_connector.py

Data Landing and Storage Boundary

Output writing is separated from extraction. The bronze writer is the likely Databricks/lakehouse landing path, while the S3 writer provides object-storage delivery or staging.

• src/ingestion/utils/writers/bronze_writer.py

• src/ingestion/utils/writers/s3_writer.py

• src/ingestion/utils/writers/__init__.py

The main data flow is therefore: job/main entry point → configuration lookup → connector factory → source extraction → bronze and/or S3 writer → audit logging.

Auditability, Secrets, and Operational Services

Operational concerns are centralized in utilities rather than embedded in individual connectors.

• Audit recording: src/ingestion/utils/audit.py

• Audit-table schema: config/ddl/audit_log.sql

• Secret access boundary: src/ingestion/utils/secrets.py

• Logging: src/ingestion/utils/logger.py

Material external/runtime dependencies implied by the structure are Databricks job execution, source databases/MongoDB/SFTP, Amazon S3, a metadata/audit database or catalog, and a secrets provider accessible from the Databricks runtime.

Deployment and Package Layout

The Python implementation is packaged under src/ingestion, with connector and utility layers forming the reusable framework core.

• src/ingestion/__init__.py

• src/ingestion/connectors/__init__.py

• requirements.txt

Environment files and SQL DDL are operational inputs: environments supply runtime values, while DDL provisions the metadata and audit control plane required before ingestion jobs can operate.