# aliahmed369000000-ai/neural-service-mesh architecture

> Architecture diagram of the GitHub repository aliahmed369000000-ai/neural-service-mesh, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aliahmed369000000-ai/neural-service-mesh
- Repository: https://github.com/aliahmed369000000-ai/neural-service-mesh

## Overview

1. **What this repo is**
   Neural Service Mesh is a local, modular AI system with a graph-based runtime, multiple cognitive engines, a knowledge ingestion layer, sensor inputs, and persistence for training/checkpoints. It is not just one app; it is a platform for running, training, and evolving a “mesh” of AI capabilities.

2. **Primary entrypoints**
   The runtime appears to start from `main.py`, with additional execution surfaces in `api/app.py` (Flask API), `Streamlit_app.py` / `streamlit_app.py` (UI), `wsgi.py` (deployment entrypoint), and helper scripts like `train_simulate.py`, `fix_and_run.py`, and `restore_weights.py`.

3. **Core execution model**
   The `core/` package is the structural backbone: `engine.py`, `graph.py`, `node.py`, and `registry.py` suggest a node-and-graph orchestration model where capabilities are registered as nodes, connected in a graph, and executed by a central engine.

4. **AI/cognition subsystem**
   The `ai/` package is the main intelligence layer. It includes planning, reasoning, routing, optimization, memory, scoring, validation, and governance components. The Phase 7 additions called out in the README are especially important: `evolution_pipeline.py`, `governance_p7.py`, `code_generator.py`, `objectives.py`, `phase6_validator.py`, `sandbox_lab.py`, and `self_awareness.py`. These suggest the system can propose changes, validate them, and evolve itself under policy constraints.

5. **Memory and learning**
   The repo has several overlapping learning/memory mechanisms: `ai/memory_engine.py`, `episodic_memory.py`, `memory_consolidator.py`, `experience_store.py`, `experience_trainer.py`, and `brain_checkpoint.py`. Persistent memory also lives outside code in `memory/episodic.db` and multiple JSON/NPY checkpoint files under `checkpoints/`.

6. **Sensors and input capture**
   `sensors/` defines the system’s inbound observation layer. `api_sensor.py`, `filesystem_sensor.py`, `log_sensor.py`, and `webhook_sensor.py` feed into `sensor_hub.py`, meaning the mesh can ingest events from APIs, files, logs, and webhooks rather than only interactive user input.

7. **Services and processing pipeline**
   `services/` provides the operational layer that handles incoming work: `input_service.py`, `processor_service.py`, `output_service.py`, and `dynamic_node.py`. Architecturally, this is likely the bridge between raw sensor events and the core graph/AI modules.

8. **Knowledge subsystem**
   The `knowledge/` package stores and reasons over domain data. It includes `knowledge_store.py`, `cognitive_graph.py`, `qa_engine.py`, `relation_inferencer.py`, and Quran-specific training/indexing artifacts such as `quran_index.json`, many `quran_chunk_*.json` files, and `quran_continuous_trainer.py`. This layer is clearly intended to build a structured knowledge graph and answer/query engine.

9. **Knowledge ingestion and source management**
   `knowledge_sources/` is the ingestion pipeline into the knowledge layer. It has a base source abstraction plus source-specific modules for Quran data, web fetching, concept extraction, domain sources, metadata, tracking, validation, and quality scoring. This is the boundary where external content is normalized into the internal knowledge model.

10. **Storage and persistence**
   `storage/db.py` and `storage/file_storage.py` handle durable state, while `data/` holds runtime JSON artifacts and SQLite state such as `mesh.db`. The repo also tracks model state in `weights/`, `checkpoints/`, and NumPy files like `nsm_embedding.npz` and `dynamic_layer.npy`, which are part of the system’s learned state rather than source code.

11. **World/simulation boundary**
   `world_model/environment_model.py` and `world_model/environment.json` define a simulated or abstract environment used by the AI/training loop. This is a separate boundary from the knowledge store: knowledge represents facts and relations, while the world model represents stateful dynamics or scenarios.

12. **Observability and audit trail**
   `logs/mesh_logger.py` and the many JSON logs in `data/` indicate the system is designed to record activity, training results, source tracking, and graph metrics. This is important for understanding how decisions, ingestion, and evolution are audited.

13. **UI and human interaction**
   The repo includes both a Streamlit-facing UI and an HTML dashboard (`dashboard.html`, `NSM_Agent_v8.html`, `nsm_chat_ui.py`, `nsm_chat.py`). These are likely thin presentation layers over the core engine and API, rather than the center of the architecture.

14. **Overall data flow**
   The most likely flow is: sensors and external sources ingest data → connectors/services transform it → core graph/engine routes work to AI modules → knowledge and memory layers persist results → storage/checkpoints retain state → logs/metrics capture behavior → the evolution/training pipeline updates models and policies. The architecture is therefore a closed loop of observation, reasoning, storage, and self-improvement.

Main entry points: [`core/engine.py`](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/engine.py) (Engine), [`ai/reasoning_pipeline.py`](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/reasoning_pipeline.py) (Reasoning), [`ai/evolution_pipeline.py`](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/evolution_pipeline.py) (Evolution), [`knowledge/knowledge_store.py`](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/knowledge/knowledge_store.py) (Store)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_main_py["Main<br/>entrypoint<br/>[main.py]"]
  node_api_app["API<br/>flask app<br/>[app.py]"]
  node_streamlit_ui["UI<br/>streamlit app<br/>[streamlit_app.py]"]
end

subgraph group_core["Core Mesh"]
  node_core_engine{{"Engine<br/>graph runtime<br/>[engine.py]"}}
  node_core_graph["Graph<br/>[graph.py]"]
  node_core_node["Node<br/>[node.py]"]
  node_core_registry["Registry<br/>[registry.py]"]
end

subgraph group_cognition["Cognition"]
  node_ai_reasoning{{"Reasoning<br/>cognitive pipeline"}}
  node_ai_memory[("Memory<br/>[memory_engine.py]")]
  node_ai_evolution{{"Evolution<br/>self-improvement"}}
end

subgraph group_knowledge["Knowledge"]
  node_knowledge_store[("Store<br/>knowledge graph<br/>[knowledge_store.py]")]
  node_qa_engine["QA<br/>question answering<br/>[qa_engine.py]"]
end

subgraph group_ingest["Ingestion"]
  node_sensor_hub{{"Sensor Hub<br/>input hub<br/>[sensor_hub.py]"}}
  node_input_service["Input<br/>[input_service.py]"]
  node_processor_service["Processor"]
  node_output_service["Output<br/>[output_service.py]"]
  node_source_manager{{"Sources<br/>ingestion manager<br/>[source_manager.py]"}}
end

subgraph group_state["State"]
  node_storage_db[("Storage<br/>persistence<br/>[db.py]")]
  node_checkpoints["Checkpoints<br/>trained state"]
end

subgraph group_observability["Observability"]
  node_mesh_logger["Logger<br/>audit log<br/>[mesh_logger.py]"]
end

subgraph group_world["World"]
  node_world_model["World Model<br/>simulation"]
end

node_main_py -->|"starts"| node_core_engine
node_api_app -->|"invokes"| node_core_engine
node_streamlit_ui -->|"drives"| node_core_engine
node_core_engine -->|"executes"| node_core_graph
node_core_engine -->|"resolves"| node_core_registry
node_core_engine -->|"runs"| node_core_node
node_sensor_hub -->|"feeds"| node_input_service
node_input_service -->|"normalizes"| node_processor_service
node_processor_service -->|"dispatches"| node_ai_reasoning
node_ai_reasoning -->|"returns"| node_output_service
node_ai_reasoning -->|"consults"| node_ai_memory
node_ai_reasoning -->|"queries"| node_knowledge_store
node_ai_evolution -->|"persists"| node_storage_db
node_ai_evolution -->|"saves"| node_checkpoints
node_source_manager -->|"ingests"| node_knowledge_store
node_source_manager -->|"primes"| node_qa_engine
node_qa_engine -->|"answers"| node_output_service
node_knowledge_store -->|"stores"| node_storage_db
node_core_engine -->|"logs"| node_mesh_logger
node_ai_evolution -->|"simulates"| node_world_model
node_sensor_hub -->|"observes"| node_mesh_logger

click node_main_py "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/main.py"
click node_api_app "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/api/app.py"
click node_streamlit_ui "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/streamlit_app.py"
click node_core_engine "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/engine.py"
click node_core_graph "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/graph.py"
click node_core_node "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/node.py"
click node_core_registry "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/registry.py"
click node_sensor_hub "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/sensors/sensor_hub.py"
click node_input_service "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/services/input_service.py"
click node_processor_service "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/services/processor_service.py"
click node_output_service "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/services/output_service.py"
click node_ai_reasoning "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/reasoning_pipeline.py"
click node_ai_memory "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/memory_engine.py"
click node_ai_evolution "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/evolution_pipeline.py"
click node_knowledge_store "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/knowledge/knowledge_store.py"
click node_qa_engine "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/knowledge/qa_engine.py"
click node_source_manager "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/knowledge_sources/source_manager.py"
click node_storage_db "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/storage/db.py"
click node_checkpoints "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/checkpoints/brain_checkpoint_20260611T192910Z.json"
click node_mesh_logger "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/logs/mesh_logger.py"
click node_world_model "https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/world_model/environment_model.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_api_app,node_streamlit_ui,node_mesh_logger toneBlue
class node_core_engine,node_core_graph,node_core_node,node_core_registry,node_world_model toneAmber
class node_ai_reasoning,node_ai_memory,node_ai_evolution toneMint
class node_knowledge_store,node_qa_engine toneRose
class node_sensor_hub,node_input_service,node_processor_service,node_output_service,node_source_manager toneIndigo
class node_storage_db,node_checkpoints toneTeal
```

## Components

### Runtime

Entry points and presentation surfaces

- [Main](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/main.py): entrypoint · Primary local runtime entrypoint. · `main.py`
- [API](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/api/app.py): flask app · HTTP service entrypoint for external calls. · `api/app.py`
- [UI](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/streamlit_app.py): streamlit app · Interactive dashboard and chat interface. · `streamlit_app.py`

### Core Mesh

Graph execution and orchestration

- [Engine](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/engine.py): graph runtime · Central orchestrator for node execution and routing. · `core/engine.py`
- [Graph](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/graph.py): mesh graph · Defines the node-and-edge topology of the mesh. · `core/graph.py`
- [Node](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/node.py): node model · Execution unit abstraction used by the engine. · `core/node.py`
- [Registry](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/core/registry.py): capability registry · Registers available nodes and capabilities. · `core/registry.py`

### Cognition

Planning, reasoning, optimization, and self-improvement

- [Reasoning](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/reasoning_pipeline.py): cognitive pipeline · Planning, routing, and decision-making layer. · `ai/reasoning_pipeline.py`
- [Memory](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/memory_engine.py): memory system · Episodic and consolidated memory management. · `ai/memory_engine.py`
- [Evolution](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/ai/evolution_pipeline.py): self-improvement · Pipeline for proposing, validating, and refining changes. · `ai/evolution_pipeline.py`

### Knowledge

Structured knowledge, QA, and relation inference

- [Store](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/knowledge/knowledge_store.py): knowledge graph · Persistent knowledge and structured relations. · `knowledge/knowledge_store.py`
- [QA](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/knowledge/qa_engine.py): question answering · Answers queries over the knowledge layer. · `knowledge/qa_engine.py`

### Ingestion

Source management and external data normalization

- [Sensor Hub](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/sensors/sensor_hub.py): input hub · Aggregates events from sensor sources. · `sensors/sensor_hub.py`
- [Input](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/services/input_service.py): service · Receives and normalizes incoming work. · `services/input_service.py`
- [Processor](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/services/processor_service.py): service · Transforms inputs into actionable tasks. · `services/processor_service.py`
- [Output](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/services/output_service.py): service · Publishes responses and downstream results. · `services/output_service.py`
- [Sources](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/knowledge_sources/source_manager.py): ingestion manager · Coordinates external knowledge sources and validation. · `knowledge_sources/source_manager.py`

### State

Persistence, checkpoints, and learned artifacts

- [Storage](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/storage/db.py): persistence · Durable database and file-backed state. · `storage/db.py`
- [Checkpoints](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/checkpoints/brain_checkpoint_20260611T192910Z.json): trained state · Saved model weights, memory, and training snapshots. · `checkpoints/brain_checkpoint_20260611T192910Z.json`

### Observability

Logging, metrics, and audit trails

- [Logger](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/logs/mesh_logger.py): audit log · System logging and operational trace capture. · `logs/mesh_logger.py`

### World

Simulated environment boundary

- [World Model](https://github.com/aliahmed369000000-ai/neural-service-mesh/blob/main/world_model/environment_model.py): simulation · Abstract environment used by training and reasoning loops. · `world_model/environment_model.py`

## Connections

- Main → Engine: starts · Bootstraps the mesh runtime.
- API → Engine: invokes · Routes API requests into the engine.
- UI → Engine: drives · UI interactions flow into core execution.
- Engine → Graph: executes · Uses the graph topology for orchestration.
- Engine → Registry: resolves · Looks up available capabilities and nodes.
- Engine → Node: runs · Executes node units within the mesh.
- Sensor Hub → Input: feeds · Collected sensor events become incoming work.
- Input → Processor: normalizes · Prepares work for processing.
- Processor → Reasoning: dispatches · Sends tasks to cognitive pipelines.
- Reasoning → Output: returns · Produces responses or actions.
- Reasoning → Memory: consults · Reads and updates memory during reasoning.
- Reasoning → Store: queries · Uses structured knowledge for inference.
- Evolution → Storage: persists · Stores learned state and system changes.
- Evolution → Checkpoints: saves · Writes model and training snapshots.
- Sources → Store: ingests · Normalizes external sources into the knowledge layer.
- Sources → QA: primes · Builds queryable knowledge artifacts.
- QA → Output: answers · Serves knowledge-backed responses.
- Store → Storage: stores · Durable backing for knowledge artifacts.
- Engine → Logger: logs · Execution events are audited.
- Evolution → World Model: simulates · Uses the world boundary for training and evaluation.
- Sensor Hub → Logger: observes · Sensor activity is traced for auditability.