How tns-schrauwen/jt-labnotebook works
1. This repo is a hybrid bioinformatics lab notebook system: a Python application (biolab) for collecting/organizing notebook metadata and a Rust crate (biotrack) for scanning, classifying, and indexing project files. The published product is a static documentation site under site/.
2. The primary human-authored source content lives in docs/. It is organized by notebook concepts such as notes, projects, registry entries, runs, datasets, experiments, and tags. The generated HTML in site/ mirrors this structure and is the read-only presentation layer.
3. The notebook runtime state is kept in the hidden .biolab/ directory. It contains config, annotation data, scan state, cron/log output, and per-project state files. That makes .biolab/ the operational boundary between source docs and the indexing/sync machinery.
4. The Python package in src/biolab/ is the orchestration layer. Its main subsystems are:
- cli.py and api/ for user-facing entry points,
- collectors/ for ingesting metadata,
- watchers/ for observing changes in the filesystem and external workflow systems,
- database.py and models.py for persisted notebook state,
- generators/markdown.py for producing markdown outputs,
- config.py for runtime configuration.
5. The watcher subsystem is important because it connects the notebook to bioinformatics execution environments. There are dedicated watchers for the filesystem, Nextflow, and Slurm, so run state and job status can be pulled from local files and scheduler/workflow outputs into the notebook database and documentation model.
6. The data model appears centered on lab artifacts such as projects, runs, annotations, and collected observations. Those entities are then rendered into markdown and indexed into the generated site, rather than being served from a dynamic web backend.
7. The Rust crate biotrack/ is a separate, performance-oriented analysis engine. It has its own CLI, configuration, logging, tests, and benches, suggesting it is usable standalone and focused on fast scanning/indexing of large directory trees.
8. Inside biotrack, the pipeline is: scan the filesystem (scanner/), build an in-memory tree (tree/), classify entries (classifier/), infer relationships between items (relationship/), and persist/query results in SQLite (store/). The hasher/blake3.rs module indicates content hashing is part of the identity/indexing strategy.
9. The tree and classifier layers are the core abstraction boundary in biotrack: scanner entries are normalized, organized into an arena/tree structure, then enriched with file-type or sample/pipeline-stage semantics before relationship inference and storage.
10. External systems matter mainly at the edges: the local filesystem is the primary source of truth, Nextflow and Slurm provide workflow/job metadata, SQLite is the persistence layer for indexed state, and MkDocs turns the notebook content into the static site/ output.
11. Deployment is static-site oriented. mkdocs.yml, site/, .github/workflows/deploy.yml, and scripts/auto_sync.sh indicate the docs are built and published automatically, with no always-on application server required for the public-facing part.
12. The repo also contains precomputed state and generated artifacts for several named bioinformatics projects/runs under docs/projects, docs/registry, and docs/runs, plus corresponding state files under .biolab/state/. Those artifacts are the bridge between operational pipeline tracking and the notebook documentation.
Main entry points: watchers Watchers · database.py Database · docs Docs · mod.rs Scanner
Components 22
Notebook content
Human-authored notebook content and generated static site
Biolab app
Python orchestration, state, collectors, and watchers
- CLI · User-facing command entry point
src/biolab/cli.py - API · Programmatic interface for notebook operations
src/biolab/api/__init__.py - Collectors · Metadata ingestion and normalization
src/biolab/collectors - Watchers · Filesystem and workflow/job observers
src/biolab/watchers - Database · Persisted notebook state management
src/biolab/database.py - Models · Notebook entities such as projects, runs, annotations, and observations
src/biolab/models.py - Markdown · Renders notebook data into markdown outputs
src/biolab/generators/markdown.py - Config · Python runtime configuration
src/biolab/config.py
Biotrack engine
Rust scanning, classification, storage, and CLI
- CLI · Standalone analysis command entry point
biotrack/src/cli/mod.rs - Scanner · Walks and normalizes directory entries
biotrack/src/scanner/mod.rs - Tree · Builds and queries an in-memory indexed tree
biotrack/src/tree/mod.rs - Classifier · Assigns file-type, sample, and pipeline-stage semantics
biotrack/src/classifier/mod.rs - Relationships · Infers links between tracked items
biotrack/src/relationship/mod.rs - SQLite store · Persists and queries indexed scan results
biotrack/src/store/mod.rs - BLAKE3 · Content hashing used for identity and indexing
biotrack/src/hasher/blake3.rs - Config · Rust settings and defaults
biotrack/src/config/mod.rs
Ops and deploy
Runtime state and static-site publishing
- .biolab · Operational boundary for annotations, scan state, logs, and per-project state
.biolab - Deploy workflow · Automated static-site publishing pipeline
.github/workflows/deploy.yml - Sync script · Local sync automation for notebook and site state
scripts/auto_sync.sh
Connections 27
- Docs → Collectors: source content · Notebook markdown is ingested and normalized
- Watchers → Collectors: feeds · Filesystem and workflow events trigger metadata ingestion
- Collectors → Models: builds · Collected metadata is mapped into domain objects
- Models → Database: persists · Notebook entities are stored in the database
- Watchers → Database: updates · Observed filesystem and job state updates persisted notebook state
- Database → Markdown: renders · Persisted notebook state is converted into markdown outputs
- Markdown → Docs: writes · Generated markdown feeds the notebook content tree
- Config → CLI: configures · Runtime settings drive the Python entry point
- CLI → API: invokes · CLI routes user actions into the programmatic API
- API → Database: reads/writes · API orchestrates notebook state access
- .biolab → Database: holds · Hidden runtime state is the operational backing store
- .biolab → Watchers: captures · Watcher outputs and scan state are retained in .biolab
- Config → CLI: configures · Rust settings drive the standalone CLI
- CLI → Scanner: starts · CLI launches filesystem scanning
- Scanner → Tree: normalizes · Scanned entries are organized into the tree arena
- Tree → Classifier: enriches · Tree nodes are classified with semantic labels
- Classifier → Relationships: infers · Classified items feed relationship inference
- Relationships → SQLite store: persists · Inferred relationships and indexed state are written to SQLite
- Scanner → BLAKE3: hashes · File content hashes support identity and indexing
- SQLite store → Tree: queries · Stored results support tree queries and retrieval
- Docs → Site: publishes · Documentation is rendered into the static site
- MkDocs → Site: builds · MkDocs configuration produces the generated site
- Deploy workflow → Site: deploys · CI publishes the static output
- Sync script → .biolab: syncs · Local automation keeps operational state aligned
- Sync script → Docs: syncs · Automation updates notebook content from tracked state
- Watchers → Scanner: observes · Watcher-derived state can be fed by filesystem scan outputs
- Watchers → SQLite store: indexes · Workflow/job observations are captured in indexed state