# spinoco/connector architecture

> Architecture diagram of the GitHub repository spinoco/connector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/spinoco/connector
- Repository: https://github.com/spinoco/connector

## Overview

##### Project Type and Purpose
- This is an open-source, single‐service data connector written in TypeScript/Node.js.
- Its job is to poll the Spinoco Platform HTTP API for tasks and related data (recordings, transcriptions, metadata, etc.) and write them to a storage backend (local FS or cloud).

##### File Structure
- Top level:
- Dockerfile, docker-compose-example.yml → containerization/deployment
- src/ → service code
- src/config.ts → environment-variable config loader
- src/http.ts → HTTP client for Spinoco API
- src/sync.ts → orchestration of fetch + storage
- src/storage/ → pluggable providers: local, AWS S3, GCS, Azure
- src/interfaces/ → TypeScript contracts
- build.sh, webpack.*.js → bundling into dist/bundle.js
- README describes environment variables, supported data types, and how to run via Docker.

##### Main Components & Their Interactions
a. Configuration Loader
- Reads env vars (SP_API_TOKEN, SP_TASK_SYNC_*, storage provider config).
b. HTTP Client
- Authenticates to the Spinoco REST API, retrieves task lists and per-task data streams.
c. Sync Controller
- Applies filters (skills, hashtags, date), iterates tasks, invokes HTTP client, delegates to storage.
d. Storage Abstraction Layer
- Defines IStorage interface.
- Four implementations:
- LocalFileSystem (writes to disk)
- AWS S3 (uploads to S3)
- Google Cloud Storage
- Azure Blob
e. Logging
- Centralized logger (log level controlled by SP_LOG_LEVEL)
f. Deployment Unit
- Bundled Node.js binary (bundle.js) inside a Docker image, orchestrated via docker-compose or Kubernetes.

##### Extensibility
- Webpack build step—source TS → bundled JS.
- Environment variables as configuration injection points.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External API
    SpinocoAPI["Spinoco Platform API"]:::external

    %% Deployment & Bundling
    subgraph "Deployment & Bundling" 
        direction TB
        BuildScript["build.sh"]:::deployment
        WebpackDev["webpack.dev.js"]:::deployment
        WebpackDist["webpack.dist.js"]:::deployment
        DockerfileNode["Dockerfile"]:::deployment
        DockerCompose["docker-compose-example.yml"]:::deployment
        EnvDefaults["env.defaults"]:::deployment
        EnvDev["env.dev"]:::deployment
        PackageDef["package.json / package-lock.json"]:::deployment
    end

    %% Service Container
    subgraph "Spinoco Data Connector Service (Docker Container)"
        direction TB
        Config["Configuration Loader"]:::internal
        HTTPClient["HTTP Client"]:::internal
        SyncController["Sync Controller"]:::internal
        Logger["Logger"]:::internal
        Utils["Utility Functions / Filters"]:::internal
        IStorage["IStorage Interface"]:::internal

        subgraph "Storage Implementations"
            direction TB
            LocalFS["Local File System"]:::storage
            AWSS3["AWS S3 Storage"]:::storage
            GCS["Google Cloud Storage"]:::storage
            AzureBlob["Azure Blob Storage"]:::storage
        end
    end

    %% Data Flows
    Config -->|provides settings| SyncController
    SyncController -->|uses| HTTPClient
    HTTPClient -->|requests/receives| SpinocoAPI
    SyncController -->|delegates store to| IStorage
    IStorage -->|impl| LocalFS
    IStorage -->|impl| AWSS3
    IStorage -->|impl| GCS
    IStorage -->|impl| AzureBlob
    SyncController -->|uses| Utils
    Config -->|env vars| Config
    Config -->|env defaults| EnvDefaults
    Config -->|env dev| EnvDev

    %% Logging flows
    Config -->|logs to| Logger
    HTTPClient -->|logs to| Logger
    SyncController -->|logs to| Logger
    IStorage -->|logs to| Logger
    Utils -->|logs to| Logger

    %% Build & Deploy flows
    WebpackDev -->|bundles TS→JS| WebpackDist
    WebpackDist -->|produces bundle.js| DockerfileNode
    BuildScript -->|runs build| WebpackDist
    DockerfileNode -->|builds image| DockerCompose

    %% Click Events
    click Config "https://github.com/spinoco/connector/blob/main/src/config.ts"
    click Config "https://github.com/spinoco/connector/blob/main/src/interfaces/config.ts"
    click HTTPClient "https://github.com/spinoco/connector/blob/main/src/http.ts"
    click HTTPClient "https://github.com/spinoco/connector/blob/main/src/interfaces/http-error.ts"
    click HTTPClient "https://github.com/spinoco/connector/blob/main/src/interfaces/http-server.ts"
    click HTTPClient "https://github.com/spinoco/connector/blob/main/src/interfaces/paging-query.ts"
    click SyncController "https://github.com/spinoco/connector/blob/main/src/sync.ts"
    click SyncController "https://github.com/spinoco/connector/blob/main/src/index.ts"
    click IStorage "https://github.com/spinoco/connector/blob/main/src/interfaces/task-sync.ts"
    click LocalFS "https://github.com/spinoco/connector/blob/main/src/storage/local-storage.ts"
    click AWSS3 "https://github.com/spinoco/connector/blob/main/src/storage/aws-s3-storage.ts"
    click GCS "https://github.com/spinoco/connector/blob/main/src/storage/gcs-storage.ts"
    click AzureBlob "https://github.com/spinoco/connector/blob/main/src/storage/azure-storage.ts"
    click Logger "https://github.com/spinoco/connector/blob/main/src/logging.ts"
    click Utils "https://github.com/spinoco/connector/blob/main/src/utils.ts"
    click DockerfileNode "https://github.com/spinoco/connector/tree/main/Dockerfile"
    click DockerCompose "https://github.com/spinoco/connector/blob/main/docker-compose-example.yml"
    click BuildScript "https://github.com/spinoco/connector/blob/main/build.sh"
    click WebpackDev "https://github.com/spinoco/connector/blob/main/webpack.dev.js"
    click WebpackDist "https://github.com/spinoco/connector/blob/main/webpack.dist.js"
    click EnvDefaults "https://github.com/spinoco/connector/blob/main/env.defaults"
    click EnvDev "https://github.com/spinoco/connector/blob/main/env.dev"
    click PackageDef "https://github.com/spinoco/connector/blob/main/package.json"
    click PackageDef "https://github.com/spinoco/connector/blob/main/package-lock.json"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef deployment fill:#FFE4B5,stroke:#333,stroke-width:1px
```
