# satvik-sharma-05/ulpf architecture

> Architecture diagram of the GitHub repository satvik-sharma-05/ulpf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/satvik-sharma-05/ulpf
- Repository: https://github.com/satvik-sharma-05/ulpf

## Overview

ULPF normalizes heterogeneous logs into a shared schema, preserves their raw bytes, and stores records in Neo4j for investigation. Users can upload logs or preview parsing in Parser Lab, inspect analytics and alerts, or ask questions through the chat interface. Batch and Kafka-based ingestion are distinct entry paths; analytics and chatbot services read the graph for different forms of exploration. README documents parser synthesis, OCSF/ECS exports, and a tamper-evident ledger; the sampled excerpts do not establish every API-to-module call, so uncertain wiring is omitted.

Main entry points: [`app/backend/ingestion_pipeline/core/parser.py`](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/parser.py) (Log parser), [`app/backend/ingestion_pipeline/realtime/consumer_service.py`](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/realtime/consumer_service.py) (Realtime consumer), [`app/backend/ingestion_pipeline/batch/run_pipeline.py`](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/batch/run_pipeline.py) (Batch ingestion), [`app/backend/chatbot_pipeline/chatbot.py`](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/chatbot.py) (Chat orchestration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Log ingestion"]
  node_ingest_api["Ingestion API<br/>[api.py]"]
  node_batch["Batch ingestion<br/>[run_pipeline.py]"]
  node_realtime["Realtime consumer"]
end

subgraph group_trust["Normalization and trust"]
  node_parser["Log parser<br/>[parser.py]"]
  node_registry["Parser registry<br/>[registry.py]"]
  node_fallback["Generic fallback"]
  node_schema["Common schema<br/>[graph_schema.py]"]
  node_embeddings["Log embeddings<br/>[embeddings.py]"]
  node_ledger["Tamper ledger<br/>[chain.py]"]
  node_merkle["Merkle proofs<br/>[merkle.py]"]
end

subgraph group_investigate["Investigation services"]
  node_analytics_api["Analytics API<br/>[api.py]"]
  node_queries["Analytics queries<br/>[queries.py]"]
  node_exports["Export formats<br/>[exporters.py]"]
  node_chat_api["Chat API<br/>[api.py]"]
  node_chatbot["Chat orchestration<br/>[chatbot.py]"]
  node_graph_rag["Graph retrieval<br/>[graph_rag.py]"]
  node_cypher["Text to Cypher<br/>[text_to_cypher.py]"]
end

subgraph group_experience["User experience"]
  node_frontend["Web application<br/>[App.jsx]"]
  node_lab["Parser Lab<br/>[ParserLabPage.jsx]"]
  node_chat_page["Chat experience<br/>[ChatPage.jsx]"]
  node_analytics_page["Analytics views<br/>[AnalyticsPage.jsx]"]
  node_upload["Upload client<br/>[ingestApi.js]"]
  node_analytics_client["Analytics client<br/>[analyticsApi.js]"]
  node_chat_client["Chat client<br/>[api.js]"]
end

node_operator(("Log engineer"))
node_sources["Raw logs"]
node_kafka["Kafka"]
node_neo4j[("Log graph")]
node_local_ai["Local AI services"]

node_operator -->|"uses"| node_frontend
node_sources -->|"supplies logs"| node_batch
node_sources -->|"publishes raw events"| node_kafka
node_kafka -->|"delivers events"| node_realtime
node_frontend -->|"uploads or previews"| node_upload
node_upload -->|"calls"| node_ingest_api
node_ingest_api -->|"parses preview"| node_parser
node_batch -->|"parses logs"| node_parser
node_realtime -->|"parses events"| node_parser
node_parser -->|"detects format"| node_registry
node_parser -->|"falls back"| node_fallback
node_parser -->|"normalizes records"| node_schema
node_batch -.->|"generates vectors"| node_embeddings
node_realtime -.->|"embeds events"| node_embeddings
node_batch -->|"writes records"| node_neo4j
node_realtime -.->|"writes records"| node_neo4j
node_realtime -->|"publishes parsed events"| node_kafka
node_neo4j -->|"anchors records"| node_ledger
node_ledger -->|"builds proofs"| node_merkle
node_analytics_page -->|"requests views"| node_analytics_client
node_analytics_client -->|"calls"| node_analytics_api
node_analytics_api -->|"runs aggregations"| node_queries
node_queries -->|"reads graph"| node_neo4j
node_analytics_api -->|"streams exports"| node_exports
node_chat_page -->|"sends questions"| node_chat_client
node_chat_client -->|"calls"| node_chat_api
node_chat_api -->|"dispatches request"| node_chatbot
node_chatbot -->|"plans graph query"| node_cypher
node_chatbot -->|"retrieves evidence"| node_graph_rag
node_cypher -->|"queries graph"| node_neo4j
node_graph_rag -->|"retrieves records"| node_neo4j
node_chatbot -.->|"uses offline AI"| node_local_ai
node_lab -->|"requests parse preview"| node_upload
node_frontend -->|"shows"| node_analytics_page
node_frontend -->|"shows"| node_chat_page

click node_ingest_api "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/api.py"
click node_batch "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/batch/run_pipeline.py"
click node_realtime "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/realtime/consumer_service.py"
click node_parser "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/parser.py"
click node_registry "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/parsers/registry.py"
click node_fallback "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/parsers/generic_fallback.py"
click node_schema "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/graph_schema.py"
click node_embeddings "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/embeddings.py"
click node_ledger "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/ledger/chain.py"
click node_merkle "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/ledger/merkle.py"
click node_analytics_api "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/api.py"
click node_queries "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/queries.py"
click node_exports "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/exporters.py"
click node_chat_api "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/api.py"
click node_chatbot "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/chatbot.py"
click node_graph_rag "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/graph_rag.py"
click node_cypher "https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/text_to_cypher.py"
click node_frontend "https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/App.jsx"
click node_lab "https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/pages/ParserLabPage.jsx"
click node_chat_page "https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/pages/ChatPage.jsx"
click node_analytics_page "https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/pages/AnalyticsPage.jsx"
click node_upload "https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/ingestApi.js"
click node_analytics_client "https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/analyticsApi.js"
click node_chat_client "https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/api.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_api,node_batch,node_realtime toneBlue
class node_parser,node_registry,node_fallback,node_schema,node_embeddings,node_ledger,node_merkle,node_neo4j toneAmber
class node_analytics_api,node_queries,node_exports,node_chat_api,node_chatbot,node_graph_rag,node_cypher toneMint
class node_frontend,node_lab,node_chat_page,node_analytics_page,node_upload,node_analytics_client,node_chat_client toneRose
class node_operator,node_sources,node_kafka,node_local_ai toneIndigo
```

## Components

### Log ingestion

- [Ingestion API](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/api.py): `app/backend/ingestion_pipeline/api.py`
- [Batch ingestion](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/batch/run_pipeline.py): `app/backend/ingestion_pipeline/batch/run_pipeline.py`
- [Realtime consumer](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/realtime/consumer_service.py): `app/backend/ingestion_pipeline/realtime/consumer_service.py`

### Normalization and trust

- [Log parser](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/parser.py): `app/backend/ingestion_pipeline/core/parser.py`
- [Parser registry](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/parsers/registry.py): `app/backend/ingestion_pipeline/core/parsers/registry.py`
- [Generic fallback](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/parsers/generic_fallback.py): `app/backend/ingestion_pipeline/core/parsers/generic_fallback.py`
- [Common schema](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/graph_schema.py): `app/backend/ingestion_pipeline/core/graph_schema.py`
- [Log embeddings](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/ingestion_pipeline/core/embeddings.py): `app/backend/ingestion_pipeline/core/embeddings.py`
- [Tamper ledger](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/ledger/chain.py): `app/backend/analytics_pipeline/ledger/chain.py`
- [Merkle proofs](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/ledger/merkle.py): `app/backend/analytics_pipeline/ledger/merkle.py`

### Investigation services

- [Analytics API](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/api.py): `app/backend/analytics_pipeline/api.py`
- [Analytics queries](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/queries.py): `app/backend/analytics_pipeline/queries.py`
- [Export formats](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/analytics_pipeline/exporters.py): `app/backend/analytics_pipeline/exporters.py`
- [Chat API](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/api.py): `app/backend/chatbot_pipeline/api.py`
- [Chat orchestration](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/chatbot.py): `app/backend/chatbot_pipeline/chatbot.py`
- [Graph retrieval](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/graph_rag.py): `app/backend/chatbot_pipeline/graph_rag.py`
- [Text to Cypher](https://github.com/satvik-sharma-05/ulpf/blob/main/app/backend/chatbot_pipeline/text_to_cypher.py): `app/backend/chatbot_pipeline/text_to_cypher.py`

### User experience

- [Web application](https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/App.jsx): `app/frontend/src/App.jsx`
- [Parser Lab](https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/pages/ParserLabPage.jsx): `app/frontend/src/pages/ParserLabPage.jsx`
- [Chat experience](https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/pages/ChatPage.jsx): `app/frontend/src/pages/ChatPage.jsx`
- [Analytics views](https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/pages/AnalyticsPage.jsx): `app/frontend/src/pages/AnalyticsPage.jsx`
- [Upload client](https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/ingestApi.js): `app/frontend/src/ingestApi.js`
- [Analytics client](https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/analyticsApi.js): `app/frontend/src/analyticsApi.js`
- [Chat client](https://github.com/satvik-sharma-05/ulpf/blob/main/app/frontend/src/api.js): `app/frontend/src/api.js`

### Other components

- **Log engineer**
- **Raw logs**
- **Kafka**
- **Log graph**
- **Local AI services**

## Connections

- Log engineer → Web application: uses
- Raw logs → Batch ingestion: supplies logs
- Raw logs → Kafka: publishes raw events
- Kafka → Realtime consumer: delivers events
- Web application → Upload client: uploads or previews
- Upload client → Ingestion API: calls
- Ingestion API → Log parser: parses preview
- Batch ingestion → Log parser: parses logs
- Realtime consumer → Log parser: parses events
- Log parser → Parser registry: detects format
- Log parser → Generic fallback: falls back
- Log parser → Common schema: normalizes records
- Batch ingestion → Log embeddings: generates vectors
- Realtime consumer → Log embeddings: embeds events
- Batch ingestion → Log graph: writes records
- Realtime consumer → Log graph: writes records
- Realtime consumer → Kafka: publishes parsed events
- Log graph → Tamper ledger: anchors records
- Tamper ledger → Merkle proofs: builds proofs
- Analytics views → Analytics client: requests views
- Analytics client → Analytics API: calls
- Analytics API → Analytics queries: runs aggregations
- Analytics queries → Log graph: reads graph
- Analytics API → Export formats: streams exports
- Chat experience → Chat client: sends questions
- Chat client → Chat API: calls
- Chat API → Chat orchestration: dispatches request
- Chat orchestration → Text to Cypher: plans graph query
- Chat orchestration → Graph retrieval: retrieves evidence
- Text to Cypher → Log graph: queries graph
- Graph retrieval → Log graph: retrieves records
- Chat orchestration → Local AI services: uses offline AI
- Parser Lab → Upload client: requests parse preview
- Web application → Analytics views: shows
- Web application → Chat experience: shows