How atlanhq/atlan-python works
Main Components
• Client Layer: This includes modules under pyatlan/client that implement functionality such as asset manipulation, audit, authentication (SSO, credential management), and API interaction. This layer encapsulates how the SDK communicates with Atlan’s backend.
• Core Models & Business Logic: Modules under pyatlan/model and pyatlan/model/assets represent the domain models, including asset types (e.g., ADLS, API, airflow, etc.), lineage, and contracts. These provide structured representations of data and business rules.
• Caching: The pyatlan/cache directory contains classes that manage caching of tags, custom metadata, enumerations, user/group caches, etc. This reduces redundant API calls and speeds up operations.
• Event Handling: Modules under pyatlan/events handle atlan events (e.g., atlan_event_handler, lambda_handler) and possibly the event ingestion or processing related to asset changes.
• Code Generation & Template Handling: The pyatlan/generator directory, with its extensive subdirectories and Jinja2 templates, is responsible for generating code files (like class generators, method stubs, and documentation) based on asset definitions and configurations.
• Packaging & Distribution: Files like setup.py, MANIFEST.in, and the pyatlan/pkg directory indicate mechanisms to package, build, and possibly distribute the SDK, with specific modules for creating package files.
• Testing & QA: The tests directory, along with integration and unit tests, outlines the testing strategy, including test utilities. This ensures that each component functions as expected.
• Documentation & CI/CD: The docs folder (including Sphinx-based documentation files) and configurations in .github/workflows (and similar CI/CD configuration files) capture the automation and documentation generation aspects, as well as quality assurance measures.
• Containerization & Deployment: The containers directory (with Dockerfile) suggests the SDK can be run or built in a containerized manner, to ease deployment and integration testing.
Relationships and Interactions
• At the highest level, the Client Layer utilizes the Core Models to form requests to Atlan. This is performed using the functions provided in the client modules.
• Before executing API requests, caching components can intercept and serve redundant data requests.
• When changes occur in Atlan, event handlers (in the events module) pick up these updates, possibly triggering generator functions to update documentation or code stubs via the generator layer.
• The packaging and distribution systems interact with continuous integration processes (as defined by GitHub workflows and the Makefile) for building, testing, and distributing the SDK.
• Documentation is generated using Sphinx (as evidenced by the docs directory and accompanying make files), which integrates with the code generation components.
Architectural Patterns/Principles
• Modular Design: Separation between client, model, cache, events, and generator reflects a clean separation of concerns.
• Template-Driven Code Generation: The use of Jinja2 templates in the generator ensures that code/documentation generation follows a systematic and consistent pattern.
• Testing-First/Quality Assurance Emphasis: A rich set of unit and integration tests suggests robust validation through continuous testing.
• CI/CD Integration: The presence of GitHub workflows, pre-commit hooks, and QA scripts points to an automated build and deploy pipeline.
Relevant Technologies & Frameworks
• Python 3.8+ is the base runtime.
• Jinja2 is used for code and documentation template generation.
• Sphinx is used for generating documentation.
• Pytest is the main testing framework.
• CI/CD workflows are managed through GitHub Actions.
• Docker (via the containers directory) is used for container-based deployments/cross-environment testing.