How bluesky/tiled works
Project Type & Purpose
• Tiled is essentially a full-stack data-access service. Its goal is to provide structured access (slicing, caching, subselection) to data stored in a variety of formats and sources. Note that although its core logic is in Python, it exposes HTTP APIs that serve both browser clients and Python-based workflows.
• The project is composed of several major subsystems (server, client, adapters, catalog, authentication) and includes deployment configurations (Docker, Helm, CI/CD).
File Structure Analysis – Key Components
• Top-level directories include:
• “tiled”: Contains the primary backend/server code with submodules for adapters (data format converters), catalog (data discovery and search), client (Python client logic), and server (HTTP API endpoints, authentication, caching).
• “web-frontend”: Houses the user interface code for the web client (React-based) that interacts with the backend APIs.
• “tiled/authn_database” and “tiled/catalog”: Separate internal services focused on authentication and cataloging (indexing/search capabilities).
• “docs”: Provides detailed internal documentation and explanations of features like caching, security, and API usage.
• “helm”, “docker-compose.yml” and continuous_integration folders: These specify deployment, container orchestration, and CI/CD pipeline configurations.
• Numerous configuration and testing files (e.g., Dockerfile, helm charts, CI workflows, pre-commit, mypy, and tests) also suggest rigorous adherence to development, deployment, and code quality standards.
Guidance
a. Main Components:
• Client Layer:
• Python Client (tiled/client) and Web Frontend (web-frontend).
• API / Service Layer:
• Tiled Server (tiled/server) which exposes HTTP API endpoints for different data modalities (arrays, tables, etc.).
• Catalog Service (tiled/catalog) responsible for indexing, structured data discovery, and fulltext search.
• Authentication & Authorization:
• Authentication database (tiled/authn_database) to manage user authentication, session tokens, and access control.
• Access Policies (tiled/access_policies) for enforcing data access rules based on web security standards.
• Data Storage / External Services:
• Underlying databases, filesystems, or external services that provide the raw data (not directly stored in this repo but represented as external integrations).
• Infrastructure & Deployment:
• Docker configurations and Helm charts outlining container deployment.
• CI/CD pipelines (GitHub workflows, as seen in the .github/workflows directory).
• Monitoring (monitoring_example) configurations with Prometheus and Grafana dashboards.
b. Relationships and Interactions:
• Clients (Python and Web Browsers) send requests (HTTP/REST API) to the Tiled Server.
• The Tiled Server (including its authentication and caching modules) processes these requests.
• When accessing data, the Tiled Server leverages various adapters to interface with different data storage backends.
• The Catalog Service ensures that data is well-indexed so that search and structured retrieval can happen quickly.
• External authentication providers (OIDC, SAML, etc.) might integrate with the authn_database component.
• Data flows between these components must capture both request and response processes, including caching (both client and server side).
• Deployment and orchestration are managed via Docker, Docker Compose, and Helm, indicating how these components are containerized and deployed in a Kubernetes environment.
c. Architectural Patterns/Design Principles:
• Micro-modular Design: Separation between data access (adapters), data organization (catalog), authentication (authn_database), and client interfaces.
• Separation of Concerns: Clear split between client-side (web-frontend and Python client) and server-side logic (HTTP APIs, caching, authentication).
• Extensibility: Use of adapters to abstract and extend support for various data formats and legacy data sources.
• Containerized Deployment: Dockerfiles, docker-compose, and Helm charts show a container-first design for deployment and scaling.
d. Relevant Technologies & Frameworks:
• Python as the main backend language.
• React (or similar) for the web frontend.
• HTTP/REST APIs (indicated by API endpoints in the server routes and client’s usage example).
• Container orchestration (Docker, Helm) and CI/CD automation (GitHub workflows).
• You capture both internal flows (e.g., server-to-server when catalog queries are made) and external flows (client to server).
• Adding notes or callouts highlighting performance optimizations (like caching) and security (access control).