How sensein/brainkb works
Project type and purpose
• BrainKB is a full-fledged, microservices-based knowledge-base platform for neuroscience. It comprises multiple back-end services (ingestion, query, chat, ML, user management, authentication) communicating via REST and messaging, backed by GraphDB/RDF stores, Postgres caches, and RabbitMQ, and (formerly) a Django-based web UI.
File structure
• Top-level directories each represent one service or infra component:
• APItokenmanager (Django JWT user & scope manager)
• ingestion_service (producer + worker)
• query_service
• usermanagement_service
• chat_service
• ml_service
• graphdb (GraphDB triple store)
• oxigraph (alternative graph store)
• rabbit-mq (message broker)
• sparql_queries, linkml-models (ancillary artifacts)
• Each “_service” folder follows the same pattern: core/routers (API endpoints), core/models (ORM/pydantic), core/database, core/security, tests, Dockerfile, compose files for dev & prod.
• Presence of docker-compose files indicates containerized deployment; configuration and env files show external dependency wiring.
README for additional insights
• Confirms microservices: ingest (with RabbitMQ), query (GraphDB SPARQL), JWT manager, GraphDB, RabbitMQ, plus documentation link.
• WebApp is obsolete—focus on back-end services.
a. Main components
• API Gateway / Client Applications (future web UI, CLI, other clients)
• APItokenmanager (Django): issues and verifies JWTs, manages scopes
• usermanagement_service: user CRUD, roles
• ingestion_service:
• Producer API receives data → validates → enqueues to RabbitMQ
• Worker listens to RabbitMQ → processes messages → writes to GraphDB (or RDF store)
• query_service: handles SPARQL queries and updates against GraphDB
• chat_service: provides conversational interfaces, caches user context in Postgres, may query GraphDB
• ml_service: NLP/NER and GPT-based services, reads from GraphDB or other data stores, writes results back
• Data stores:
• GraphDB (primary RDF triple store)
• Oxigraph (secondary/experimental graph store)
• Postgres instances (per-service caches: chat_service, ml_service, usermanagement_service)
• RabbitMQ: message broker for ingestion pipeline
• SPARQL queries & LinkML schemas: offline artifacts used by query and ingestion services
b. Relationships and interactions
• Clients → APItokenmanager to obtain JWTs
• Clients → each service’s REST API (ingestion, query, chat, ml, usermgmt), passing JWT bearer tokens
• ingestion_service Producer → RabbitMQ → ingestion_service Worker → GraphDB
• query_service ↔ GraphDB (via SPARQL HTTP endpoint)
• chat_service ↔ GraphDB (for context) and ↔ its Postgres cache
• ml_service ↔ GraphDB and ↔ its Postgres database
• usermanagement_service ↔ its Postgres database
• All services validate JWTs via APItokenmanager’s public keys or introspection
c. Architectural patterns and principles
• Microservices with single-responsibility boundaries
• Event-driven ingestion pipeline using RabbitMQ
• Polyglot persistence: RDF store for KGs, relational caches for fast lookups
• JWT-based centralized authentication/authorization
• Containerization and environment-specific docker-compose setups
d. Key technologies
• Django (APItokenmanager)
• FastAPI (likely for other core services based on routers/pydantic)
• RabbitMQ (messaging)
• GraphDB & Oxigraph (triple stores)
• Postgres (caching and relational storage)
• Docker & Docker Compose