# somilbuilds/log-analyzer architecture

> Architecture diagram of the GitHub repository somilbuilds/log-analyzer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/somilbuilds/log-analyzer
- Repository: https://github.com/somilbuilds/log-analyzer

## Overview

This is a web-server log analytics pipeline with two result paths: replayed CLF records feed Spark streaming analytics for live MongoDB-backed dashboard views, while HDFS-backed batch processing produces status-code counts and R time-series anomaly analysis. The streaming job invokes the Bloom, DGIM, and Flajolet–Martin implementations. README documents the Hive and batch capabilities, but their runtime wiring is not sampled; those relationships are shown only where documentation states them.

Main entry points: [`streaming/streaming_job.py`](https://github.com/somilbuilds/log-analyzer/blob/master/streaming/streaming_job.py) (Spark Streaming Job), [`batch/mapreduce_job`](https://github.com/somilbuilds/log-analyzer/tree/master/batch/mapreduce_job) (MapReduce Status Counts), [`ingestion/log_replayer.py`](https://github.com/somilbuilds/log-analyzer/blob/master/ingestion/log_replayer.py) (Log Replayer), [`batch/hive_tables.sql`](https://github.com/somilbuilds/log-analyzer/blob/master/batch/hive_tables.sql) (Hive Analytics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Log Ingestion"]
  node_replayer["Log Replayer<br/>[log_replayer.py]"]
  node_jsonfiles["JSON Stream Files"]
end

subgraph group_realtime["Live Analytics"]
  node_sparkjob["Spark Streaming Job<br/>[streaming_job.py]"]
  node_bloom["Bloom Filter<br/>[bloom_filter.py]"]
  node_dgim["DGIM Counter<br/>[dgim.py]"]
  node_fm["FM Estimator<br/>[flajolet_martin.py]"]
  node_mongo[("MongoDB")]
  node_hdfs[("HDFS Parquet")]
end

subgraph group_batch["Batch Analytics"]
  node_hive["Hive Analytics<br/>[hive_tables.sql]"]
  node_hivequeries["Hive Queries<br/>[queries.sql]"]
  node_mapreduce["MapReduce Status Counts"]
  node_mapper["Status Mapper<br/>[mapper.py]"]
  node_reducer["Status Reducer<br/>[reducer.py]"]
  node_ranalysis["R Time-Series Analysis<br/>[analysis.R]"]
end

subgraph group_presentation["Results"]
  node_dashboard["Streamlit Dashboard<br/>[app.py]"]
  node_plots["Anomaly Plots"]
end

node_operator(("Pipeline Operator"))
node_logfile["CLF Log File"]

node_operator -->|"starts replay"| node_replayer
node_logfile -->|"reads logs"| node_replayer
node_replayer -->|"writes batches"| node_jsonfiles
node_jsonfiles -->|"streams records"| node_sparkjob
node_sparkjob -->|"checks hosts"| node_bloom
node_sparkjob -->|"counts 5xx"| node_dgim
node_sparkjob -->|"estimates distinct"| node_fm
node_sparkjob -->|"writes aggregates"| node_mongo
node_sparkjob -->|"writes Parquet"| node_hdfs
node_hdfs -->|"stores batch data"| node_hive
node_hive -->|"queries tables"| node_hivequeries
node_hdfs -->|"reads raw logs"| node_mapreduce
node_mapreduce -->|"maps records"| node_mapper
node_mapreduce -->|"reduces counts"| node_reducer
node_mongo -->|"reads aggregates"| node_ranalysis
node_mongo -->|"serves live metrics"| node_dashboard
node_ranalysis -->|"generates plots"| node_plots
node_plots -->|"displays analysis"| node_dashboard

click node_logfile "https://github.com/somilbuilds/log-analyzer/blob/master/data/raw/access_log"
click node_replayer "https://github.com/somilbuilds/log-analyzer/blob/master/ingestion/log_replayer.py"
click node_sparkjob "https://github.com/somilbuilds/log-analyzer/blob/master/streaming/streaming_job.py"
click node_bloom "https://github.com/somilbuilds/log-analyzer/blob/master/streaming/bloom_filter.py"
click node_dgim "https://github.com/somilbuilds/log-analyzer/blob/master/streaming/dgim.py"
click node_fm "https://github.com/somilbuilds/log-analyzer/blob/master/streaming/flajolet_martin.py"
click node_hive "https://github.com/somilbuilds/log-analyzer/blob/master/batch/hive_tables.sql"
click node_hivequeries "https://github.com/somilbuilds/log-analyzer/blob/master/batch/queries.sql"
click node_mapreduce "https://github.com/somilbuilds/log-analyzer/tree/master/batch/mapreduce_job"
click node_mapper "https://github.com/somilbuilds/log-analyzer/blob/master/batch/mapreduce_job/mapper.py"
click node_reducer "https://github.com/somilbuilds/log-analyzer/blob/master/batch/mapreduce_job/reducer.py"
click node_ranalysis "https://github.com/somilbuilds/log-analyzer/blob/master/analytics/analysis.R"
click node_dashboard "https://github.com/somilbuilds/log-analyzer/blob/master/dashboard/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_replayer,node_jsonfiles toneBlue
class node_sparkjob,node_bloom,node_dgim,node_fm,node_mongo,node_hdfs toneAmber
class node_hive,node_hivequeries,node_mapreduce,node_mapper,node_reducer,node_ranalysis toneMint
class node_dashboard,node_plots toneRose
class node_operator toneIndigo
class node_logfile toneTeal
```

## Components

### Log Ingestion

- [Log Replayer](https://github.com/somilbuilds/log-analyzer/blob/master/ingestion/log_replayer.py): `ingestion/log_replayer.py`
- **JSON Stream Files**

### Live Analytics

- [Spark Streaming Job](https://github.com/somilbuilds/log-analyzer/blob/master/streaming/streaming_job.py): `streaming/streaming_job.py`
- [Bloom Filter](https://github.com/somilbuilds/log-analyzer/blob/master/streaming/bloom_filter.py): `streaming/bloom_filter.py`
- [DGIM Counter](https://github.com/somilbuilds/log-analyzer/blob/master/streaming/dgim.py): `streaming/dgim.py`
- [FM Estimator](https://github.com/somilbuilds/log-analyzer/blob/master/streaming/flajolet_martin.py): `streaming/flajolet_martin.py`
- **MongoDB**
- **HDFS Parquet**

### Batch Analytics

- [Hive Analytics](https://github.com/somilbuilds/log-analyzer/blob/master/batch/hive_tables.sql): `batch/hive_tables.sql`
- [Hive Queries](https://github.com/somilbuilds/log-analyzer/blob/master/batch/queries.sql): `batch/queries.sql`
- [MapReduce Status Counts](https://github.com/somilbuilds/log-analyzer/tree/master/batch/mapreduce_job): `batch/mapreduce_job`
- [Status Mapper](https://github.com/somilbuilds/log-analyzer/blob/master/batch/mapreduce_job/mapper.py): `batch/mapreduce_job/mapper.py`
- [Status Reducer](https://github.com/somilbuilds/log-analyzer/blob/master/batch/mapreduce_job/reducer.py): `batch/mapreduce_job/reducer.py`
- [R Time-Series Analysis](https://github.com/somilbuilds/log-analyzer/blob/master/analytics/analysis.R): `analytics/analysis.R`

### Results

- [Streamlit Dashboard](https://github.com/somilbuilds/log-analyzer/blob/master/dashboard/app.py): `dashboard/app.py`
- **Anomaly Plots**

### Other components

- **Pipeline Operator**
- [CLF Log File](https://github.com/somilbuilds/log-analyzer/blob/master/data/raw/access_log): `data/raw/access_log`

## Connections

- Pipeline Operator → Log Replayer: starts replay
- CLF Log File → Log Replayer: reads logs
- Log Replayer → JSON Stream Files: writes batches
- JSON Stream Files → Spark Streaming Job: streams records
- Spark Streaming Job → Bloom Filter: checks hosts
- Spark Streaming Job → DGIM Counter: counts 5xx
- Spark Streaming Job → FM Estimator: estimates distinct
- Spark Streaming Job → MongoDB: writes aggregates
- Spark Streaming Job → HDFS Parquet: writes Parquet
- HDFS Parquet → Hive Analytics: stores batch data
- Hive Analytics → Hive Queries: queries tables
- HDFS Parquet → MapReduce Status Counts: reads raw logs
- MapReduce Status Counts → Status Mapper: maps records
- MapReduce Status Counts → Status Reducer: reduces counts
- MongoDB → R Time-Series Analysis: reads aggregates
- MongoDB → Streamlit Dashboard: serves live metrics
- R Time-Series Analysis → Anomaly Plots: generates plots
- Anomaly Plots → Streamlit Dashboard: displays analysis