How sudobhavik/layarustparser works
ULPF receives firewall/syslog lines, classifies and parses them into lossless OCSF NetworkActivity events, then batches and persists evidence in Parquet with an RFC 6962 Merkle ledger for verification. Its CLI also exposes onboarding, evaluation, and serving workflows; the AI crate supplies the tiered parser and template-learning components. The README documents these capabilities, while sampled source confirms core parsing, socket ingestion, and parts of persistence and CLI behavior; some subsystem wiring is therefore left unspecified.
Main entry points: main.rs ULPF CLI · handlers.rs Serve Handlers · pipeline.rs Three-Tier Pipeline · mod.rs Universal Parser
Components 22
Ingest and Parse
- Socket Ingest
crates/ulpf-core/src/ingest/socket.rs - Universal Parser
crates/ulpf-core/src/parser/mod.rs - Format Classifier
crates/ulpf-core/src/parser/classifier.rs - Vendor Extractors
crates/ulpf-core/src/parser/extractors/mod.rs - OCSF Schema
crates/ulpf-core/src/schema/ocsf.rs
AI Pipeline
- Three-Tier Pipeline
crates/ulpf-ai/src/pipeline.rs - Drain Miner
crates/ulpf-ai/src/drain.rs - Laya Triage
crates/ulpf-ai/src/laya.rs - Parser Onboarding
crates/ulpf-ai/src/onboarder.rs - Engine Evaluation
crates/ulpf-ai/src/evaluator.rs - Engine Duel
crates/ulpf-ai/src/duel.rs
Evidence Integrity
- Block Batcher
crates/ulpf-integrity/src/batcher.rs - Merkle Ledger
crates/ulpf-integrity/src/merkle.rs - Parquet Storage
crates/ulpf-integrity/src/storage.rs - Tamper Checks
crates/ulpf-integrity/src/tamper.rs
CLI and Operations
- ULPF CLI
crates/ulpf-cli/src/main.rs - Serve Handlers
crates/ulpf-cli/src/serve/handlers.rs - Service State
crates/ulpf-cli/src/serve/state.rs
Other components
- Syslog Sources
- Operator
- Parquet Files
- Ledger File
Connections 27
- Syslog Sources → Socket Ingest: sends logs
- ULPF CLI → Socket Ingest: starts ingest
- Socket Ingest → Universal Parser: parses batches
- Universal Parser → Format Classifier: classifies format
- Universal Parser → Vendor Extractors: dispatches parse
- Vendor Extractors → OCSF Schema: builds events
- ULPF CLI → Three-Tier Pipeline: processes lines
- Three-Tier Pipeline → Universal Parser: parses known logs
- Three-Tier Pipeline → Drain Miner: mines templates
- Three-Tier Pipeline → Laya Triage: triages samples
- ULPF CLI → Block Batcher: batches events
- Block Batcher → Merkle Ledger: roots blocks
- Block Batcher → Parquet Storage: writes blocks
- Merkle Ledger → Ledger File: records roots
- Parquet Storage → Parquet Files: writes Parquet
- ULPF CLI → Merkle Ledger: verifies roots
- ULPF CLI → Tamper Checks: runs verification
- Operator → ULPF CLI: runs commands
- Operator → Serve Handlers: uses service
- Serve Handlers → Service State: reads state
- Serve Handlers → Tamper Checks: checks evidence
- Serve Handlers → Parser Onboarding: validates samples
- Serve Handlers → Parquet Storage: reads records
- ULPF CLI → Parser Onboarding: onboards formats
- ULPF CLI → Engine Evaluation: evaluates engines
- Engine Evaluation → Three-Tier Pipeline: scores pipeline
- ULPF CLI → Engine Duel: compares engines