# tns-schrauwen/jt-labnotebook architecture

> Architecture diagram of the GitHub repository tns-schrauwen/jt-labnotebook, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tns-schrauwen/jt-labnotebook
- Repository: https://github.com/tns-schrauwen/jt-labnotebook

## Overview

1. This repo is a hybrid bioinformatics lab notebook system: a Python application (`biolab`) for collecting/organizing notebook metadata and a Rust crate (`biotrack`) for scanning, classifying, and indexing project files. The published product is a static documentation site under `site/`.

2. The primary human-authored source content lives in `docs/`. It is organized by notebook concepts such as notes, projects, registry entries, runs, datasets, experiments, and tags. The generated HTML in `site/` mirrors this structure and is the read-only presentation layer.

3. The notebook runtime state is kept in the hidden `.biolab/` directory. It contains config, annotation data, scan state, cron/log output, and per-project state files. That makes `.biolab/` the operational boundary between source docs and the indexing/sync machinery.

4. The Python package in `src/biolab/` is the orchestration layer. Its main subsystems are:
   - `cli.py` and `api/` for user-facing entry points,
   - `collectors/` for ingesting metadata,
   - `watchers/` for observing changes in the filesystem and external workflow systems,
   - `database.py` and `models.py` for persisted notebook state,
   - `generators/markdown.py` for producing markdown outputs,
   - `config.py` for runtime configuration.

5. The watcher subsystem is important because it connects the notebook to bioinformatics execution environments. There are dedicated watchers for the filesystem, Nextflow, and Slurm, so run state and job status can be pulled from local files and scheduler/workflow outputs into the notebook database and documentation model.

6. The data model appears centered on lab artifacts such as projects, runs, annotations, and collected observations. Those entities are then rendered into markdown and indexed into the generated site, rather than being served from a dynamic web backend.

7. The Rust crate `biotrack/` is a separate, performance-oriented analysis engine. It has its own CLI, configuration, logging, tests, and benches, suggesting it is usable standalone and focused on fast scanning/indexing of large directory trees.

8. Inside `biotrack`, the pipeline is: scan the filesystem (`scanner/`), build an in-memory tree (`tree/`), classify entries (`classifier/`), infer relationships between items (`relationship/`), and persist/query results in SQLite (`store/`). The `hasher/blake3.rs` module indicates content hashing is part of the identity/indexing strategy.

9. The tree and classifier layers are the core abstraction boundary in `biotrack`: scanner entries are normalized, organized into an arena/tree structure, then enriched with file-type or sample/pipeline-stage semantics before relationship inference and storage.

10. External systems matter mainly at the edges: the local filesystem is the primary source of truth, Nextflow and Slurm provide workflow/job metadata, SQLite is the persistence layer for indexed state, and MkDocs turns the notebook content into the static `site/` output.

11. Deployment is static-site oriented. `mkdocs.yml`, `site/`, `.github/workflows/deploy.yml`, and `scripts/auto_sync.sh` indicate the docs are built and published automatically, with no always-on application server required for the public-facing part.

12. The repo also contains precomputed state and generated artifacts for several named bioinformatics projects/runs under `docs/projects`, `docs/registry`, and `docs/runs`, plus corresponding state files under `.biolab/state/`. Those artifacts are the bridge between operational pipeline tracking and the notebook documentation.

Main entry points: [`src/biolab/watchers`](https://github.com/tns-schrauwen/jt-labnotebook/tree/main/src/biolab/watchers) (Watchers), [`src/biolab/database.py`](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/database.py) (Database), [`docs`](https://github.com/tns-schrauwen/jt-labnotebook/tree/main/docs) (Docs), [`biotrack/src/scanner/mod.rs`](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/scanner/mod.rs) (Scanner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_content["Notebook content"]
  node_docs["Docs<br/>content tree"]
  node_site["Site"]
  node_mkdocs_yml["MkDocs<br/>site config<br/>[mkdocs.yml]"]
end

subgraph group_group_biolab["Biolab app"]
  node_biolab_cli["CLI<br/>python entrypoint<br/>[cli.py]"]
  node_biolab_api["API<br/>python entrypoint<br/>[__init__.py]"]
  node_biolab_collectors["Collectors<br/>ingest layer"]
  node_biolab_watchers["Watchers<br/>event sources"]
  node_biolab_db[("Database<br/>state store<br/>[database.py]")]
  node_biolab_models["Models<br/>domain model<br/>[models.py]"]
  node_biolab_markdown["Markdown<br/>generator<br/>[markdown.py]"]
  node_biolab_config["Config<br/>[config.py]"]
end

subgraph group_group_biotrack["Biotrack engine"]
  node_biotrack_cli["CLI<br/>rust entrypoint<br/>[mod.rs]"]
  node_biotrack_scan["Scanner<br/>filesystem scan<br/>[mod.rs]"]
  node_biotrack_tree["Tree<br/>[mod.rs]"]
  node_biotrack_classify["Classifier<br/>semantic classify<br/>[mod.rs]"]
  node_biotrack_rel["Relationships<br/>inference layer<br/>[mod.rs]"]
  node_biotrack_store[("SQLite store<br/>persistence layer<br/>[mod.rs]")]
  node_biotrack_hash{{"BLAKE3<br/>content hashing<br/>[blake3.rs]"}}
  node_biotrack_config["Config<br/>[mod.rs]"]
end

subgraph group_group_ops["Ops and deploy"]
  node_biolab_state[(".biolab<br/>runtime state<br/>[.biolab]")]
  node_deploy_workflow["Deploy workflow<br/>github actions<br/>[deploy.yml]"]
  node_sync_script["Sync script<br/>automation script<br/>[auto_sync.sh]"]
end

node_docs -->|"source content"| node_biolab_collectors
node_biolab_watchers -->|"feeds"| node_biolab_collectors
node_biolab_collectors -->|"builds"| node_biolab_models
node_biolab_models -->|"persists"| node_biolab_db
node_biolab_watchers -->|"updates"| node_biolab_db
node_biolab_db -->|"renders"| node_biolab_markdown
node_biolab_markdown -->|"writes"| node_docs
node_biolab_config -->|"configures"| node_biolab_cli
node_biolab_cli -->|"invokes"| node_biolab_api
node_biolab_api -->|"reads/writes"| node_biolab_db
node_biolab_state -->|"holds"| node_biolab_db
node_biolab_state -->|"captures"| node_biolab_watchers
node_biotrack_config -->|"configures"| node_biotrack_cli
node_biotrack_cli -->|"starts"| node_biotrack_scan
node_biotrack_scan -->|"normalizes"| node_biotrack_tree
node_biotrack_tree -->|"enriches"| node_biotrack_classify
node_biotrack_classify -->|"infers"| node_biotrack_rel
node_biotrack_rel -->|"persists"| node_biotrack_store
node_biotrack_scan -->|"hashes"| node_biotrack_hash
node_biotrack_store -->|"queries"| node_biotrack_tree
node_docs -->|"publishes"| node_site
node_mkdocs_yml -->|"builds"| node_site
node_deploy_workflow -->|"deploys"| node_site
node_sync_script -->|"syncs"| node_biolab_state
node_sync_script -->|"syncs"| node_docs
node_biolab_watchers -->|"observes"| node_biotrack_scan
node_biolab_watchers -->|"indexes"| node_biotrack_store

click node_docs "https://github.com/tns-schrauwen/jt-labnotebook/tree/main/docs"
click node_site "https://github.com/tns-schrauwen/jt-labnotebook/tree/main/site"
click node_mkdocs_yml "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/mkdocs.yml"
click node_biolab_cli "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/cli.py"
click node_biolab_api "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/api/__init__.py"
click node_biolab_collectors "https://github.com/tns-schrauwen/jt-labnotebook/tree/main/src/biolab/collectors"
click node_biolab_watchers "https://github.com/tns-schrauwen/jt-labnotebook/tree/main/src/biolab/watchers"
click node_biolab_db "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/database.py"
click node_biolab_models "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/models.py"
click node_biolab_markdown "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/generators/markdown.py"
click node_biolab_config "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/config.py"
click node_biolab_state "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/.biolab"
click node_biotrack_cli "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/cli/mod.rs"
click node_biotrack_scan "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/scanner/mod.rs"
click node_biotrack_tree "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/tree/mod.rs"
click node_biotrack_classify "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/classifier/mod.rs"
click node_biotrack_rel "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/relationship/mod.rs"
click node_biotrack_store "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/store/mod.rs"
click node_biotrack_hash "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/hasher/blake3.rs"
click node_biotrack_config "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/config/mod.rs"
click node_deploy_workflow "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/.github/workflows/deploy.yml"
click node_sync_script "https://github.com/tns-schrauwen/jt-labnotebook/blob/main/scripts/auto_sync.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_docs,node_site,node_mkdocs_yml toneBlue
class node_biolab_cli,node_biolab_api,node_biolab_collectors,node_biolab_watchers,node_biolab_db,node_biolab_models,node_biolab_markdown,node_biolab_config toneAmber
class node_biotrack_cli,node_biotrack_scan,node_biotrack_tree,node_biotrack_classify,node_biotrack_rel,node_biotrack_store,node_biotrack_hash,node_biotrack_config toneMint
class node_biolab_state,node_deploy_workflow,node_sync_script toneRose
```

## Components

### Notebook content

Human-authored notebook content and generated static site

- [Docs](https://github.com/tns-schrauwen/jt-labnotebook/tree/main/docs): content tree · Source notebook content organized by notes, projects, registry, runs, datasets, experiments, and tags · `docs`
- [Site](https://github.com/tns-schrauwen/jt-labnotebook/tree/main/site): static site · Generated read-only HTML presentation of the notebook · `site`
- [MkDocs](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/mkdocs.yml): site config · Static site build configuration · `mkdocs.yml`

### Biolab app

Python orchestration, state, collectors, and watchers

- [CLI](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/cli.py): python entrypoint · User-facing command entry point · `src/biolab/cli.py`
- [API](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/api/__init__.py): python entrypoint · Programmatic interface for notebook operations · `src/biolab/api/__init__.py`
- [Collectors](https://github.com/tns-schrauwen/jt-labnotebook/tree/main/src/biolab/collectors): ingest layer · Metadata ingestion and normalization · `src/biolab/collectors`
- [Watchers](https://github.com/tns-schrauwen/jt-labnotebook/tree/main/src/biolab/watchers): event sources · Filesystem and workflow/job observers · `src/biolab/watchers`
- [Database](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/database.py): state store · Persisted notebook state management · `src/biolab/database.py`
- [Models](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/models.py): domain model · Notebook entities such as projects, runs, annotations, and observations · `src/biolab/models.py`
- [Markdown](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/generators/markdown.py): generator · Renders notebook data into markdown outputs · `src/biolab/generators/markdown.py`
- [Config](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/src/biolab/config.py): runtime config · Python runtime configuration · `src/biolab/config.py`

### Biotrack engine

Rust scanning, classification, storage, and CLI

- [CLI](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/cli/mod.rs): rust entrypoint · Standalone analysis command entry point · `biotrack/src/cli/mod.rs`
- [Scanner](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/scanner/mod.rs): filesystem scan · Walks and normalizes directory entries · `biotrack/src/scanner/mod.rs`
- [Tree](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/tree/mod.rs): arena tree · Builds and queries an in-memory indexed tree · `biotrack/src/tree/mod.rs`
- [Classifier](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/classifier/mod.rs): semantic classify · Assigns file-type, sample, and pipeline-stage semantics · `biotrack/src/classifier/mod.rs`
- [Relationships](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/relationship/mod.rs): inference layer · Infers links between tracked items · `biotrack/src/relationship/mod.rs`
- [SQLite store](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/store/mod.rs): persistence layer · Persists and queries indexed scan results · `biotrack/src/store/mod.rs`
- [BLAKE3](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/hasher/blake3.rs): content hashing · Content hashing used for identity and indexing · `biotrack/src/hasher/blake3.rs`
- [Config](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/biotrack/src/config/mod.rs): rust config · Rust settings and defaults · `biotrack/src/config/mod.rs`

### Ops and deploy

Runtime state and static-site publishing

- [.biolab](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/.biolab): runtime state · Operational boundary for annotations, scan state, logs, and per-project state · `.biolab`
- [Deploy workflow](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/.github/workflows/deploy.yml): github actions · Automated static-site publishing pipeline · `.github/workflows/deploy.yml`
- [Sync script](https://github.com/tns-schrauwen/jt-labnotebook/blob/main/scripts/auto_sync.sh): automation script · Local sync automation for notebook and site state · `scripts/auto_sync.sh`

## Connections

- Docs → Collectors: source content · Notebook markdown is ingested and normalized
- Watchers → Collectors: feeds · Filesystem and workflow events trigger metadata ingestion
- Collectors → Models: builds · Collected metadata is mapped into domain objects
- Models → Database: persists · Notebook entities are stored in the database
- Watchers → Database: updates · Observed filesystem and job state updates persisted notebook state
- Database → Markdown: renders · Persisted notebook state is converted into markdown outputs
- Markdown → Docs: writes · Generated markdown feeds the notebook content tree
- Config → CLI: configures · Runtime settings drive the Python entry point
- CLI → API: invokes · CLI routes user actions into the programmatic API
- API → Database: reads/writes · API orchestrates notebook state access
- .biolab → Database: holds · Hidden runtime state is the operational backing store
- .biolab → Watchers: captures · Watcher outputs and scan state are retained in .biolab
- Config → CLI: configures · Rust settings drive the standalone CLI
- CLI → Scanner: starts · CLI launches filesystem scanning
- Scanner → Tree: normalizes · Scanned entries are organized into the tree arena
- Tree → Classifier: enriches · Tree nodes are classified with semantic labels
- Classifier → Relationships: infers · Classified items feed relationship inference
- Relationships → SQLite store: persists · Inferred relationships and indexed state are written to SQLite
- Scanner → BLAKE3: hashes · File content hashes support identity and indexing
- SQLite store → Tree: queries · Stored results support tree queries and retrieval
- Docs → Site: publishes · Documentation is rendered into the static site
- MkDocs → Site: builds · MkDocs configuration produces the generated site
- Deploy workflow → Site: deploys · CI publishes the static output
- Sync script → .biolab: syncs · Local automation keeps operational state aligned
- Sync script → Docs: syncs · Automation updates notebook content from tracked state
- Watchers → Scanner: observes · Watcher-derived state can be fed by filesystem scan outputs
- Watchers → SQLite store: indexes · Workflow/job observations are captured in indexed state
