How atlanhq/application-sdk works
Project type & purpose
• This is an “open-source toolkit / PaaS SDK” in Python.
• It provides:
– A FastAPI server for local development & extension integrations
– A Temporal-driven worker for running workflows & activities
– A suite of pluggable clients, inputs, outputs, and transformers
– Doc-generation and test utilities
High-level components
A. Core SDK library (application_sdk/)
1. activities/ – business-logic units (metadata_extraction, query_extraction)
2. workflows/ – orchestration definitions (metadata_extraction, query_extraction)
3. clients/ – connectors to Temporal, SQL, workflow orchestration APIs
4. server/fastapi/ – HTTP server, routers, middleware for local dev & partner apps
5. worker.py – entry point for Temporal worker, registering activities & workflows
6. common/ – shared utilities (AWS creds, dataframe helpers, error codes)
7. inputs/ & outputs/ – pluggable I/O adapters (SQL, JSON, Parquet, object store, statestore, secretstore, eventstore, iceberg)
8. transformers/ – data-transformation templates (SQL schemas → metadata resources)
9. observability/ – instrumentation adapters (metrics, traces, logging)
10. docgen/ – automated documentation exporters (mkdocs), parsers, model definitions
11. test_utils/ – e2e, Hypothesis, scale-data generators
B. Infrastructure definitions (components/)
– Kubernetes or CloudFormation YAMLs for secretstore, objectstore, statestore, eventstore
C. Examples & docs
– examples/ showing how a partner app uses the SDK
– docs/ – user guides, architecture.md, getting-started
D. External systems
1. Atlan platform (REST API & metadata store)
2. Temporal server (workflow engine)
3. External data sources & sinks: SQL databases, object stores, secret stores, iceberg catalog, event buses, state stores
4. Observability backends (Prometheus, Jaeger, CloudWatch, etc.)
Relationships & data flows
• A partner’s application → imports SDK → invokes CLI/server or directly instantiates clients/workflows
• If using FastAPI: HTTP request → SDK server layer → triggers a workflow (via workflow client)
• workflow client → Temporal server → orchestrates activities → worker pulls tasks
• Worker → executes activities (e.g., metadata_extraction, query_extraction) → uses input adapters to read data → transformer modules → uses output adapters to write results → may call Atlan REST APIs via client or SQL via client/sql.py
• Observability decorators wrap activities & server routes, emitting metrics & traces to external backends
• Docgen module introspects the SDK’s manifest and directory structure to produce mkdocs pages
Key architectural patterns & principles
– Separation of concerns: server vs. worker vs. clients vs. IO vs. transformers
– Adapter/strategy pattern for inputs & outputs (common interface, pluggable implementations)
– Decorator pattern for observability
– Declarative workflow definitions (workflows/) vs. imperative activity code (activities/)
– Infrastructure as code under components/ for reproducible environments
Detail level
• Major sub-packages in the Core SDK layer
• Both synchronous (HTTP) and asynchronous (Temporal) flows
• Infrastructure YAMLs (components/) as deployment artifacts