How somilbuilds/log-analyzer works
This is a web-server log analytics pipeline with two result paths: replayed CLF records feed Spark streaming analytics for live MongoDB-backed dashboard views, while HDFS-backed batch processing produces status-code counts and R time-series anomaly analysis. The streaming job invokes the Bloom, DGIM, and Flajolet–Martin implementations. README documents the Hive and batch capabilities, but their runtime wiring is not sampled; those relationships are shown only where documentation states them.
Main entry points: streaming_job.py Spark Streaming Job · mapreduce_job MapReduce Status Counts · log_replayer.py Log Replayer · hive_tables.sql Hive Analytics
Components 18
Log Ingestion
- Log Replayer
ingestion/log_replayer.py - JSON Stream Files
Live Analytics
- Spark Streaming Job
streaming/streaming_job.py - Bloom Filter
streaming/bloom_filter.py - DGIM Counter
streaming/dgim.py - FM Estimator
streaming/flajolet_martin.py - MongoDB
- HDFS Parquet
Batch Analytics
- Hive Analytics
batch/hive_tables.sql - Hive Queries
batch/queries.sql - MapReduce Status Counts
batch/mapreduce_job - Status Mapper
batch/mapreduce_job/mapper.py - Status Reducer
batch/mapreduce_job/reducer.py - R Time-Series Analysis
analytics/analysis.R
Results
- Streamlit Dashboard
dashboard/app.py - Anomaly Plots
Other components
- Pipeline Operator
- CLF Log File
data/raw/access_log
Connections 18
- Pipeline Operator → Log Replayer: starts replay
- CLF Log File → Log Replayer: reads logs
- Log Replayer → JSON Stream Files: writes batches
- JSON Stream Files → Spark Streaming Job: streams records
- Spark Streaming Job → Bloom Filter: checks hosts
- Spark Streaming Job → DGIM Counter: counts 5xx
- Spark Streaming Job → FM Estimator: estimates distinct
- Spark Streaming Job → MongoDB: writes aggregates
- Spark Streaming Job → HDFS Parquet: writes Parquet
- HDFS Parquet → Hive Analytics: stores batch data
- Hive Analytics → Hive Queries: queries tables
- HDFS Parquet → MapReduce Status Counts: reads raw logs
- MapReduce Status Counts → Status Mapper: maps records
- MapReduce Status Counts → Status Reducer: reduces counts
- MongoDB → R Time-Series Analysis: reads aggregates
- MongoDB → Streamlit Dashboard: serves live metrics
- R Time-Series Analysis → Anomaly Plots: generates plots
- Anomaly Plots → Streamlit Dashboard: displays analysis