# tableau/logshark architecture

> Architecture diagram of the GitHub repository tableau/logshark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tableau/logshark
- Repository: https://github.com/tableau/logshark

## Overview

1. **What this repo is**
   LogShark is a .NET Core 3.1 command-line log analysis pipeline for Tableau Server/Desktop logs. It is not a web app; it is a batch processor that reads a log set, extracts structured events, and generates Tableau-friendly analytical outputs.

2. **Top-level runtime flow**
   Execution starts in `LogShark/Program.cs`, flows into `LogSharkRunner`, then into the log processing engine (`TableauLogsProcessor` / `TableauLogsExtractor`). The runner takes command-line parameters plus JSON config from `LogShark/Config/LogSharkConfig.json`, resolves the input log set, and coordinates parsing, plugin execution, and output generation.

3. **Input boundary: log sets and file discovery**
   `LogShark.Shared/LogReading` and `LogShark/TableauLogsExtractor.cs` define the ingestion layer. This layer handles zipped log sets, nested folders, and individual files, and normalizes them into per-log-file metadata and line streams. The shared containers (`LogFileInfo`, `LogSetInfo`, `LogLine`, `ReadLogLineResult`, etc.) are the handoff objects between discovery and parsing.

4. **Log parsing layer**
   The repository supports multiple reader strategies under `LogShark.Shared/LogReading/Readers`: plain line-by-line, multiline Java logs, CSV, YAML, native JSON logs, Prep logs, and Windows netstat output. `ILogReader` abstracts the parsing contract; each reader converts raw text into typed line records that downstream plugins can match against.

5. **Plugin architecture**
   The core analysis is plugin-driven. `LogShark/Plugins` contains one plugin per Tableau subsystem, such as Apache, Backgrounder, Bridge, Config, Hyper, Postgres, ResourceManager, Tabadmin, Vizportal, VizqlDesktop, and others. `PluginFactory`, `PluginInitializer`, and `PluginManager` provide the composition boundary: they select which plugins run, wire dependencies, and aggregate execution results.

6. **How plugins map logs to datasets**
   Each plugin owns the parsing rules and output schema for its subsystem. The plugin typically transforms raw log lines into domain events and then into tabular datasets, reflected by the many model classes under each plugin folder. Shared parsers and regex utilities live under `LogShark/Plugins/Shared` and `LogShark.Shared` to avoid duplicating timestamp/process parsing logic.

7. **Output generation boundary**
   Output is separated from analysis through the `LogShark/Writers` layer. Writers can emit:
   - **CSV**
   - **Hyper** files for Tableau
   - **SQL/Postgres**-backed outputs via Npgsql
   This separation is important: plugins produce structured data, while writers decide how that data is persisted and packaged.

8. **Workbook creation and publishing**
   The repository includes workbook templates in `LogShark/Workbooks` and generators/publishers in `LogShark/Writers/Csv`, `LogShark/Writers/Hyper`, and `LogShark/Writers/Sql`. These components turn datasets into workbook-ready artifacts, using packaged `.twbx` templates and generated data sources (`.csv` or `.hyper`) as inputs. This is the main “analysis to user-facing artifact” bridge.

9. **Configuration and environment handling**
   `LogShark/LogSharkConfiguration.cs`, `LogSharkCommandLineParameters.cs`, and the config extensions define how the tool is customized. `EnvironmentController` / `EnvironmentControllerBase` provide environment-specific behavior, while shared helpers like `OutputDirTrimmer` and `OutputDirInitializer` manage file-system output shape and cleanup.

10. **Telemetry and metrics**
   The `LogShark/Metrics` subsystem is an optional side channel for run telemetry: hardware, process, user, and run-summary inspection feed a `MetricUploader`. This is separate from the main log-analysis pipeline and appears intended for operational monitoring rather than user output.

11. **Notifications, summaries, and execution results**
   `ProcessingNotificationsCollector`, `RunSummary`, and the various `Process*Result` containers form the execution-reporting layer. They collect progress, warnings, counts, and completion status so the CLI can report what was processed and what succeeded or failed.

12. **Tests and fixtures as architecture evidence**
   The test suite is extensive and mirrors the production boundaries: readers, plugin parsers, writers, metrics, output dir handling, and end-to-end runs. The large expected-output fixtures in `LogShark.Tests/TestData/EndToEndTests/Expected` show that the system is validated at the artifact level, not just unit level, across CSV, Hyper, and workbook outputs.

13. **External dependencies that materially matter**
   The main external/runtime dependencies are the .NET runtime, Tableau Hyper output, and Npgsql/Postgres for SQL-related writing. Docker is supported through `LogShark/Dockerfile`, and the README shows containerized execution as a first-class deployment path for processing log sets in isolation.

14. **Mental model for an architecture diagram**
   A clear diagram would show: input log set and config on the left; file discovery and log readers in the middle; a plugin fan-out stage per Tableau subsystem; then writer backends producing CSV/Hyper/SQL and packaged Tableau workbooks on the right. Cross-cutting boxes should cover shared parsing utilities, metrics/telemetry, and run notifications.

Main entry points: [`LogShark/Plugins/PluginManager.cs`](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/PluginManager.cs) (Plugin manager), [`LogShark/LogSharkRunner.cs`](https://github.com/tableau/logshark/blob/master/LogShark/LogSharkRunner.cs) (Runner), [`LogShark/TableauLogsProcessor.cs`](https://github.com/tableau/logshark/blob/master/LogShark/TableauLogsProcessor.cs) (Processor), [`LogShark/Writers/IWriterFactory.cs`](https://github.com/tableau/logshark/blob/master/LogShark/Writers/IWriterFactory.cs) (Writer factory)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_program(("Program<br/>cli entry<br/>[Program.cs]"))
  node_runner["Runner<br/>orchestrator<br/>[LogSharkRunner.cs]"]
  node_processor["Processor<br/>pipeline core"]
end

subgraph group_ingestion["Ingestion"]
  node_extractor[("Extractor<br/>log discovery")]
  node_log_types["Log types<br/>shared metadata<br/>[LogSetInfo.cs]"]
  node_readers["Readers<br/>reader strategies<br/>[ILogReader.cs]"]
  node_shared_parse["Shared parse<br/>parsing helpers"]
end

subgraph group_plugins["Plugins"]
  node_plugin_factory["Plugin factory<br/>plugin composition<br/>[PluginFactory.cs]"]
  node_plugin_manager["Plugin manager<br/>plugin runtime<br/>[PluginManager.cs]"]
  node_apache["Apache<br/>plugin<br/>[ApachePlugin.cs]"]
  node_backgrounder["Backgrounder<br/>plugin"]
  node_bridge["Bridge<br/>plugin<br/>[BridgePlugin.cs]"]
  node_resource_manager["ResourceMgr<br/>plugin"]
  node_vizql_desktop["Vizql Desktop<br/>plugin"]
  node_config_plugin["Config<br/>plugin<br/>[ConfigPlugin.cs]"]
end

subgraph group_output["Output"]
  node_writer_factory["Writer factory<br/>output selection<br/>[IWriterFactory.cs]"]
  node_csv_writer["CSV writer<br/>file output<br/>[CsvFileWriter.cs]"]
  node_hyper_writer["Hyper writer<br/>tableau output<br/>[HyperWriter.cs]"]
  node_sql_writer[("SQL writer<br/>postgres output<br/>[SqlWriter.cs]")]
  node_workbooks["Workbooks<br/>templates<br/>[Apache.twbx]"]
end

subgraph group_crosscutting["Cross-cutting"]
  node_config["Config"]
  node_metrics["Metrics<br/>telemetry<br/>[MetricUploader.cs]"]
  node_notifications["Notifications<br/>run reporting"]
end

subgraph group_tests["Tests"]
  node_e2e_tests["E2E tests<br/>validation<br/>[EndToEndTest.cs]"]
  node_reader_tests["Reader tests<br/>validation"]
  node_plugin_tests["Plugin tests<br/>validation"]
  node_writer_tests["Writer tests<br/>validation"]
end

node_program -->|"starts"| node_runner
node_runner -->|"reads"| node_config
node_runner -->|"invokes"| node_processor
node_processor -->|"discovers"| node_extractor
node_extractor -->|"yields"| node_log_types
node_extractor -->|"feeds"| node_readers
node_readers -->|"uses"| node_shared_parse
node_processor -->|"selects"| node_plugin_factory
node_plugin_factory -->|"wires"| node_plugin_manager
node_plugin_manager -->|"runs"| node_apache
node_plugin_manager -->|"runs"| node_backgrounder
node_plugin_manager -->|"runs"| node_bridge
node_plugin_manager -->|"runs"| node_resource_manager
node_plugin_manager -->|"runs"| node_vizql_desktop
node_plugin_manager -->|"runs"| node_config_plugin
node_plugin_manager -->|"hands off"| node_writer_factory
node_writer_factory -->|"creates"| node_csv_writer
node_writer_factory -->|"creates"| node_hyper_writer
node_writer_factory -->|"creates"| node_sql_writer
node_hyper_writer -->|"packages"| node_workbooks
node_sql_writer -.->|"supports"| node_workbooks
node_runner -.->|"reports"| node_metrics
node_runner -.->|"emits"| node_notifications
node_processor -.->|"updates"| node_notifications
node_e2e_tests -.->|"validates"| node_processor
node_reader_tests -.->|"validates"| node_readers
node_plugin_tests -.->|"validates"| node_plugin_manager
node_writer_tests -.->|"validates"| node_sql_writer

click node_program "https://github.com/tableau/logshark/blob/master/LogShark/Program.cs"
click node_runner "https://github.com/tableau/logshark/blob/master/LogShark/LogSharkRunner.cs"
click node_processor "https://github.com/tableau/logshark/blob/master/LogShark/TableauLogsProcessor.cs"
click node_extractor "https://github.com/tableau/logshark/blob/master/LogShark.Shared/LogReading/TableauLogsExtractor.cs"
click node_log_types "https://github.com/tableau/logshark/blob/master/LogShark.Shared/LogReading/Containers/LogSetInfo.cs"
click node_readers "https://github.com/tableau/logshark/blob/master/LogShark.Shared/LogReading/Readers/ILogReader.cs"
click node_shared_parse "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Shared/TimestampParsers.cs"
click node_plugin_factory "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/PluginFactory.cs"
click node_plugin_manager "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/PluginManager.cs"
click node_apache "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Apache/ApachePlugin.cs"
click node_backgrounder "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Backgrounder/BackgrounderPlugin.cs"
click node_bridge "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Bridge/BridgePlugin.cs"
click node_resource_manager "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/ResourceManager/ResourceManagerPlugin.cs"
click node_vizql_desktop "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/VizqlDesktop/VizqlDesktopPlugin.cs"
click node_config_plugin "https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Config/ConfigPlugin.cs"
click node_writer_factory "https://github.com/tableau/logshark/blob/master/LogShark/Writers/IWriterFactory.cs"
click node_csv_writer "https://github.com/tableau/logshark/blob/master/LogShark/Writers/Csv/CsvFileWriter.cs"
click node_hyper_writer "https://github.com/tableau/logshark/blob/master/LogShark/Writers/Hyper/HyperWriter.cs"
click node_sql_writer "https://github.com/tableau/logshark/blob/master/LogShark/Writers/Sql/SqlWriter.cs"
click node_workbooks "https://github.com/tableau/logshark/blob/master/LogShark/Workbooks/Apache.twbx"
click node_config "https://github.com/tableau/logshark/blob/master/LogShark/LogSharkConfiguration.cs"
click node_metrics "https://github.com/tableau/logshark/blob/master/LogShark/Metrics/MetricUploader.cs"
click node_notifications "https://github.com/tableau/logshark/blob/master/LogShark/ProcessingNotificationsCollector.cs"
click node_e2e_tests "https://github.com/tableau/logshark/blob/master/Logshark.Tests/E2E/EndToEndTest.cs"
click node_reader_tests "https://github.com/tableau/logshark/blob/master/Logshark.Tests/LogParser/TableauLogsExtractorTests.cs"
click node_plugin_tests "https://github.com/tableau/logshark/blob/master/Logshark.Tests/Plugins/PluginManagerTests.cs"
click node_writer_tests "https://github.com/tableau/logshark/blob/master/Logshark.Tests/Writers/Sql/NpgsqlRepositoryTests.cs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_program,node_runner,node_processor toneBlue
class node_extractor,node_log_types,node_readers,node_shared_parse toneAmber
class node_plugin_factory,node_plugin_manager,node_apache,node_backgrounder,node_bridge,node_resource_manager,node_vizql_desktop,node_config_plugin toneMint
class node_writer_factory,node_csv_writer,node_hyper_writer,node_sql_writer,node_workbooks toneRose
class node_config,node_metrics,node_notifications toneIndigo
class node_e2e_tests,node_reader_tests,node_plugin_tests,node_writer_tests toneTeal
```

## Components

### Runtime

CLI entrypoint and orchestration

- [Program](https://github.com/tableau/logshark/blob/master/LogShark/Program.cs): cli entry · Command-line entrypoint · `LogShark/Program.cs`
- [Runner](https://github.com/tableau/logshark/blob/master/LogShark/LogSharkRunner.cs): orchestrator · Coordinates the full log analysis run · `LogShark/LogSharkRunner.cs`
- [Processor](https://github.com/tableau/logshark/blob/master/LogShark/TableauLogsProcessor.cs): pipeline core · Drives extraction, plugin execution, and output assembly · `LogShark/TableauLogsProcessor.cs`

### Ingestion

Log discovery and line parsing

- [Extractor](https://github.com/tableau/logshark/blob/master/LogShark.Shared/LogReading/TableauLogsExtractor.cs): log discovery · Finds log sets, files, and streams · `LogShark.Shared/LogReading/TableauLogsExtractor.cs`
- [Log types](https://github.com/tableau/logshark/blob/master/LogShark.Shared/LogReading/Containers/LogSetInfo.cs): shared metadata · Log type and file metadata containers · `LogShark.Shared/LogReading/Containers/LogSetInfo.cs`
- [Readers](https://github.com/tableau/logshark/blob/master/LogShark.Shared/LogReading/Readers/ILogReader.cs): reader strategies · Parses plain, multiline, CSV, YAML, JSON, prep, and netstat logs · `LogShark.Shared/LogReading/Readers/ILogReader.cs`
- [Shared parse](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Shared/TimestampParsers.cs): parsing helpers · Timestamp, process, and regex parsing shared by plugins · `LogShark/Plugins/Shared/TimestampParsers.cs`

### Plugins

Subsystem analyzers for Tableau logs

- [Plugin factory](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/PluginFactory.cs): plugin composition · Selects and wires plugin instances · `LogShark/Plugins/PluginFactory.cs`
- [Plugin manager](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/PluginManager.cs): plugin runtime · Executes plugins and aggregates their results · `LogShark/Plugins/PluginManager.cs`
- [Apache](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Apache/ApachePlugin.cs): plugin · HTTP request analysis · `LogShark/Plugins/Apache/ApachePlugin.cs`
- [Backgrounder](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Backgrounder/BackgrounderPlugin.cs): plugin · Job and subscription analysis · `LogShark/Plugins/Backgrounder/BackgrounderPlugin.cs`
- [Bridge](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Bridge/BridgePlugin.cs): plugin · Bridge activity and protocol analysis · `LogShark/Plugins/Bridge/BridgePlugin.cs`
- [ResourceMgr](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/ResourceManager/ResourceManagerPlugin.cs): plugin · CPU, memory, and threshold analysis · `LogShark/Plugins/ResourceManager/ResourceManagerPlugin.cs`
- [Vizql Desktop](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/VizqlDesktop/VizqlDesktopPlugin.cs): plugin · Desktop session and query analysis · `LogShark/Plugins/VizqlDesktop/VizqlDesktopPlugin.cs`
- [Config](https://github.com/tableau/logshark/blob/master/LogShark/Plugins/Config/ConfigPlugin.cs): plugin · Config and process topology analysis · `LogShark/Plugins/Config/ConfigPlugin.cs`

### Output

Writers and workbook generation

- [Writer factory](https://github.com/tableau/logshark/blob/master/LogShark/Writers/IWriterFactory.cs): output selection · Chooses CSV, Hyper, or SQL writers · `LogShark/Writers/IWriterFactory.cs`
- [CSV writer](https://github.com/tableau/logshark/blob/master/LogShark/Writers/Csv/CsvFileWriter.cs): file output · Generates CSV datasets and workbook inputs · `LogShark/Writers/Csv/CsvFileWriter.cs`
- [Hyper writer](https://github.com/tableau/logshark/blob/master/LogShark/Writers/Hyper/HyperWriter.cs): tableau output · Builds Hyper extracts and packaged workbooks · `LogShark/Writers/Hyper/HyperWriter.cs`
- [SQL writer](https://github.com/tableau/logshark/blob/master/LogShark/Writers/Sql/SqlWriter.cs): postgres output · Persists results through Npgsql/Postgres · `LogShark/Writers/Sql/SqlWriter.cs`
- [Workbooks](https://github.com/tableau/logshark/blob/master/LogShark/Workbooks/Apache.twbx): templates · Packaged Tableau workbook templates · `LogShark/Workbooks/Apache.twbx`

### Cross-cutting

Configuration, metrics, and reporting

- [Config](https://github.com/tableau/logshark/blob/master/LogShark/LogSharkConfiguration.cs): runtime config · CLI parameters and JSON configuration · `LogShark/LogSharkConfiguration.cs`
- [Metrics](https://github.com/tableau/logshark/blob/master/LogShark/Metrics/MetricUploader.cs): telemetry · Run telemetry and environment inspection · `LogShark/Metrics/MetricUploader.cs`
- [Notifications](https://github.com/tableau/logshark/blob/master/LogShark/ProcessingNotificationsCollector.cs): run reporting · Progress, warnings, counts, and completion status · `LogShark/ProcessingNotificationsCollector.cs`

### Tests

Primary validation suites

- [E2E tests](https://github.com/tableau/logshark/blob/master/Logshark.Tests/E2E/EndToEndTest.cs): validation · Artifact-level end-to-end regression coverage · `Logshark.Tests/E2E/EndToEndTest.cs`
- [Reader tests](https://github.com/tableau/logshark/blob/master/Logshark.Tests/LogParser/TableauLogsExtractorTests.cs): validation · Parser and extraction coverage · `Logshark.Tests/LogParser/TableauLogsExtractorTests.cs`
- [Plugin tests](https://github.com/tableau/logshark/blob/master/Logshark.Tests/Plugins/PluginManagerTests.cs): validation · Subsystem parser and transformer coverage · `Logshark.Tests/Plugins/PluginManagerTests.cs`
- [Writer tests](https://github.com/tableau/logshark/blob/master/Logshark.Tests/Writers/Sql/NpgsqlRepositoryTests.cs): validation · Output backend coverage · `Logshark.Tests/Writers/Sql/NpgsqlRepositoryTests.cs`

## Connections

- Program → Runner: starts · CLI hands control to the runner
- Runner → Config: reads · Loads command-line parameters and JSON settings
- Runner → Processor: invokes · Begins the main analysis pipeline
- Processor → Extractor: discovers · Resolves log sets and file streams
- Extractor → Log types: yields · Produces normalized file and set metadata
- Extractor → Readers: feeds · Hands raw text to reader strategies
- Readers → Shared parse: uses · Relies on shared timestamp and process parsing
- Processor → Plugin factory: selects · Builds the plugin set for the run
- Plugin factory → Plugin manager: wires · Composes the plugin execution boundary
- Plugin manager → Apache: runs · Executes subsystem analyzers
- Plugin manager → Backgrounder: runs · Executes subsystem analyzers
- Plugin manager → Bridge: runs · Executes subsystem analyzers
- Plugin manager → ResourceMgr: runs · Executes subsystem analyzers
- Plugin manager → Vizql Desktop: runs · Executes subsystem analyzers
- Plugin manager → Config: runs · Executes subsystem analyzers
- Plugin manager → Writer factory: hands off · Passes structured datasets to output creation
- Writer factory → CSV writer: creates · Builds CSV output path
- Writer factory → Hyper writer: creates · Builds Hyper output path
- Writer factory → SQL writer: creates · Builds SQL/Postgres output path
- Hyper writer → Workbooks: packages · Binds extracts into Tableau workbook templates
- SQL writer → Workbooks: supports · Feeds workbook-ready outputs where applicable
- Runner → Metrics: reports · Optional operational telemetry channel
- Runner → Notifications: emits · Captures progress and completion reporting
- Processor → Notifications: updates · Publishes processing results and warnings
- E2E tests → Processor: validates · Checks full run outputs
- Reader tests → Readers: validates · Checks parsing and extraction behavior
- Plugin tests → Plugin manager: validates · Checks plugin orchestration and parsers
- Writer tests → SQL writer: validates · Checks SQL output persistence
