# bluesky/tiled architecture

> Architecture diagram of the GitHub repository bluesky/tiled, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bluesky/tiled
- Repository: https://github.com/bluesky/tiled

## Overview

##### Project Type & Purpose
- Tiled is essentially a full-stack data-access service. Its goal is to provide structured access (slicing, caching, subselection) to data stored in a variety of formats and sources. Note that although its core logic is in Python, it exposes HTTP APIs that serve both browser clients and Python-based workflows.
- The project is composed of several major subsystems (server, client, adapters, catalog, authentication) and includes deployment configurations (Docker, Helm, CI/CD).

##### File Structure Analysis – Key Components
- Top-level directories include:
- “tiled”: Contains the primary backend/server code with submodules for adapters (data format converters), catalog (data discovery and search), client (Python client logic), and server (HTTP API endpoints, authentication, caching).
- “web-frontend”: Houses the user interface code for the web client (React-based) that interacts with the backend APIs.
- “tiled/authn_database” and “tiled/catalog”: Separate internal services focused on authentication and cataloging (indexing/search capabilities).
- “docs”: Provides detailed internal documentation and explanations of features like caching, security, and API usage.
- “helm”, “docker-compose.yml” and continuous_integration folders: These specify deployment, container orchestration, and CI/CD pipeline configurations.
- Numerous configuration and testing files (e.g., Dockerfile, helm charts, CI workflows, pre-commit, mypy, and tests) also suggest rigorous adherence to development, deployment, and code quality standards.

##### Guidance
a. Main Components:
- Client Layer:
- Python Client (tiled/client) and Web Frontend (web-frontend).
- API / Service Layer:
- Tiled Server (tiled/server) which exposes HTTP API endpoints for different data modalities (arrays, tables, etc.).
- Catalog Service (tiled/catalog) responsible for indexing, structured data discovery, and fulltext search.
- Authentication & Authorization:
- Authentication database (tiled/authn_database) to manage user authentication, session tokens, and access control.
- Access Policies (tiled/access_policies) for enforcing data access rules based on web security standards.
- Data Storage / External Services:
- Underlying databases, filesystems, or external services that provide the raw data (not directly stored in this repo but represented as external integrations).
- Infrastructure & Deployment:
- Docker configurations and Helm charts outlining container deployment.
- CI/CD pipelines (GitHub workflows, as seen in the .github/workflows directory).
- Monitoring (monitoring_example) configurations with Prometheus and Grafana dashboards.

b. Relationships and Interactions:
- Clients (Python and Web Browsers) send requests (HTTP/REST API) to the Tiled Server.
- The Tiled Server (including its authentication and caching modules) processes these requests.
- When accessing data, the Tiled Server leverages various adapters to interface with different data storage backends.
- The Catalog Service ensures that data is well-indexed so that search and structured retrieval can happen quickly.
- External authentication providers (OIDC, SAML, etc.) might integrate with the authn_database component.
- Data flows between these components must capture both request and response processes, including caching (both client and server side).
- Deployment and orchestration are managed via Docker, Docker Compose, and Helm, indicating how these components are containerized and deployed in a Kubernetes environment.

c. Architectural Patterns/Design Principles:
- Micro-modular Design: Separation between data access (adapters), data organization (catalog), authentication (authn_database), and client interfaces.
- Separation of Concerns: Clear split between client-side (web-frontend and Python client) and server-side logic (HTTP APIs, caching, authentication).
- Extensibility: Use of adapters to abstract and extend support for various data formats and legacy data sources.
- Containerized Deployment: Dockerfiles, docker-compose, and Helm charts show a container-first design for deployment and scaling.

d. Relevant Technologies & Frameworks:
- Python as the main backend language.
- React (or similar) for the web frontend.
- HTTP/REST APIs (indicated by API endpoints in the server routes and client’s usage example).
- Container orchestration (Docker, Helm) and CI/CD automation (GitHub workflows).

- You capture both internal flows (e.g., server-to-server when catalog queries are made) and external flows (client to server).
- Adding notes or callouts highlighting performance optimizations (like caching) and security (access control).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Tier
    subgraph "Client Tier"
        PC["Python Client"]:::client
        WF["Web Frontend"]:::client
    end

    %% API/Service Tier
    subgraph "API/Service Tier"
        TS["Tiled Server"]:::api
        %% Internal service components within the server
        AD["Adapters Module"]:::service
        CT["Catalog Service"]:::service
        ADB["Authentication Database"]:::auth
        AP["Access Policies"]:::auth
    end

    %% External Data Sources Tier
    subgraph "External Data Sources"
        EDS(("External Data Sources")):::external
    end

    %% Deployment/Infrastructure Tier
    subgraph "Deployment/Infrastructure Tier"
        DF["Dockerfile"]:::deploy
        DC["docker-compose.yml"]:::deploy
        CI["CI/CD Pipelines"]:::deploy
        Helm["Helm Charts"]:::deploy
    end

    %% Connections between tiers
    PC -->|"HTTP_Request"| TS
    WF -->|"HTTP_Request"| TS

    %% Tiled Server communicating with internal services
    TS -->|"calls"| AD
    TS -->|"queries"| CT
    TS -->|"authenticates"| ADB
    TS -->|"enforces"| AP

    %% Adapters fetching external data
    AD -->|"fetches"| EDS

    %% Click Events
    click PC "https://github.com/bluesky/tiled/tree/main/tiled/client"
    click WF "https://github.com/bluesky/tiled/tree/main/web-frontend"
    click TS "https://github.com/bluesky/tiled/tree/main/tiled/server"
    click AD "https://github.com/bluesky/tiled/tree/main/tiled/adapters"
    click CT "https://github.com/bluesky/tiled/tree/main/tiled/catalog"
    click ADB "https://github.com/bluesky/tiled/tree/main/tiled/authn_database"
    click AP "https://github.com/bluesky/tiled/blob/main/tiled/access_policies.py"
    click DF "https://github.com/bluesky/tiled/tree/main/Dockerfile"
    click DC "https://github.com/bluesky/tiled/blob/main/docker-compose.yml"
    click CI "https://github.com/bluesky/tiled/tree/main/continuous_integration/docker-configs"
    click Helm "https://github.com/bluesky/tiled/tree/main/helm"

    %% Styles
    classDef client fill:#a2d5f2,stroke:#333,stroke-width:2px;
    classDef api fill:#ffcc5c,stroke:#333,stroke-width:2px;
    classDef service fill:#96ceb4,stroke:#333,stroke-width:2px;
    classDef auth fill:#ff6f69,stroke:#333,stroke-width:2px;
    classDef external fill:#c3447a,stroke:#333,stroke-width:2px;
    classDef deploy fill:#88d8b0,stroke:#333,stroke-width:2px;
```
