How tableau/logshark works
1. What this repo is
LogShark is a .NET Core 3.1 command-line log analysis pipeline for Tableau Server/Desktop logs. It is not a web app; it is a batch processor that reads a log set, extracts structured events, and generates Tableau-friendly analytical outputs.
2. Top-level runtime flow
Execution starts in LogShark/Program.cs, flows into LogSharkRunner, then into the log processing engine (TableauLogsProcessor / TableauLogsExtractor). The runner takes command-line parameters plus JSON config from LogShark/Config/LogSharkConfig.json, resolves the input log set, and coordinates parsing, plugin execution, and output generation.
3. Input boundary: log sets and file discovery
LogShark.Shared/LogReading and LogShark/TableauLogsExtractor.cs define the ingestion layer. This layer handles zipped log sets, nested folders, and individual files, and normalizes them into per-log-file metadata and line streams. The shared containers (LogFileInfo, LogSetInfo, LogLine, ReadLogLineResult, etc.) are the handoff objects between discovery and parsing.
4. Log parsing layer
The repository supports multiple reader strategies under LogShark.Shared/LogReading/Readers: plain line-by-line, multiline Java logs, CSV, YAML, native JSON logs, Prep logs, and Windows netstat output. ILogReader abstracts the parsing contract; each reader converts raw text into typed line records that downstream plugins can match against.
5. Plugin architecture
The core analysis is plugin-driven. LogShark/Plugins contains one plugin per Tableau subsystem, such as Apache, Backgrounder, Bridge, Config, Hyper, Postgres, ResourceManager, Tabadmin, Vizportal, VizqlDesktop, and others. PluginFactory, PluginInitializer, and PluginManager provide the composition boundary: they select which plugins run, wire dependencies, and aggregate execution results.
6. How plugins map logs to datasets
Each plugin owns the parsing rules and output schema for its subsystem. The plugin typically transforms raw log lines into domain events and then into tabular datasets, reflected by the many model classes under each plugin folder. Shared parsers and regex utilities live under LogShark/Plugins/Shared and LogShark.Shared to avoid duplicating timestamp/process parsing logic.
7. Output generation boundary
Output is separated from analysis through the LogShark/Writers layer. Writers can emit:
- CSV
- Hyper files for Tableau
- SQL/Postgres-backed outputs via Npgsql
This separation is important: plugins produce structured data, while writers decide how that data is persisted and packaged.
8. Workbook creation and publishing
The repository includes workbook templates in LogShark/Workbooks and generators/publishers in LogShark/Writers/Csv, LogShark/Writers/Hyper, and LogShark/Writers/Sql. These components turn datasets into workbook-ready artifacts, using packaged .twbx templates and generated data sources (.csv or .hyper) as inputs. This is the main “analysis to user-facing artifact” bridge.
9. Configuration and environment handling
LogShark/LogSharkConfiguration.cs, LogSharkCommandLineParameters.cs, and the config extensions define how the tool is customized. EnvironmentController / EnvironmentControllerBase provide environment-specific behavior, while shared helpers like OutputDirTrimmer and OutputDirInitializer manage file-system output shape and cleanup.
10. Telemetry and metrics
The LogShark/Metrics subsystem is an optional side channel for run telemetry: hardware, process, user, and run-summary inspection feed a MetricUploader. This is separate from the main log-analysis pipeline and appears intended for operational monitoring rather than user output.
11. Notifications, summaries, and execution results
ProcessingNotificationsCollector, RunSummary, and the various Process*Result containers form the execution-reporting layer. They collect progress, warnings, counts, and completion status so the CLI can report what was processed and what succeeded or failed.
12. Tests and fixtures as architecture evidence
The test suite is extensive and mirrors the production boundaries: readers, plugin parsers, writers, metrics, output dir handling, and end-to-end runs. The large expected-output fixtures in LogShark.Tests/TestData/EndToEndTests/Expected show that the system is validated at the artifact level, not just unit level, across CSV, Hyper, and workbook outputs.
13. External dependencies that materially matter
The main external/runtime dependencies are the .NET runtime, Tableau Hyper output, and Npgsql/Postgres for SQL-related writing. Docker is supported through LogShark/Dockerfile, and the README shows containerized execution as a first-class deployment path for processing log sets in isolation.
14. Mental model for an architecture diagram
A clear diagram would show: input log set and config on the left; file discovery and log readers in the middle; a plugin fan-out stage per Tableau subsystem; then writer backends producing CSV/Hyper/SQL and packaged Tableau workbooks on the right. Cross-cutting boxes should cover shared parsing utilities, metrics/telemetry, and run notifications.
Main entry points: PluginManager.cs Plugin manager · LogSharkRunner.cs Runner · TableauLogsProcessor.cs Processor · IWriterFactory.cs Writer factory
Components 27
Runtime
CLI entrypoint and orchestration
Ingestion
Log discovery and line parsing
- Extractor · Finds log sets, files, and streams
LogShark.Shared/LogReading/TableauLogsExtractor.cs - Log types · Log type and file metadata containers
LogShark.Shared/LogReading/Containers/LogSetInfo.cs - Readers · Parses plain, multiline, CSV, YAML, JSON, prep, and netstat logs
LogShark.Shared/LogReading/Readers/ILogReader.cs - Shared parse · Timestamp, process, and regex parsing shared by plugins
LogShark/Plugins/Shared/TimestampParsers.cs
Plugins
Subsystem analyzers for Tableau logs
- Plugin factory · Selects and wires plugin instances
LogShark/Plugins/PluginFactory.cs - Plugin manager · Executes plugins and aggregates their results
LogShark/Plugins/PluginManager.cs - Apache · HTTP request analysis
LogShark/Plugins/Apache/ApachePlugin.cs - Backgrounder · Job and subscription analysis
LogShark/Plugins/Backgrounder/BackgrounderPlugin.cs - Bridge · Bridge activity and protocol analysis
LogShark/Plugins/Bridge/BridgePlugin.cs - ResourceMgr · CPU, memory, and threshold analysis
LogShark/Plugins/ResourceManager/ResourceManagerPlugin.cs - Vizql Desktop · Desktop session and query analysis
LogShark/Plugins/VizqlDesktop/VizqlDesktopPlugin.cs - Config · Config and process topology analysis
LogShark/Plugins/Config/ConfigPlugin.cs
Output
Writers and workbook generation
- Writer factory · Chooses CSV, Hyper, or SQL writers
LogShark/Writers/IWriterFactory.cs - CSV writer · Generates CSV datasets and workbook inputs
LogShark/Writers/Csv/CsvFileWriter.cs - Hyper writer · Builds Hyper extracts and packaged workbooks
LogShark/Writers/Hyper/HyperWriter.cs - SQL writer · Persists results through Npgsql/Postgres
LogShark/Writers/Sql/SqlWriter.cs - Workbooks · Packaged Tableau workbook templates
LogShark/Workbooks/Apache.twbx
Cross-cutting
Configuration, metrics, and reporting
- Config · CLI parameters and JSON configuration
LogShark/LogSharkConfiguration.cs - Metrics · Run telemetry and environment inspection
LogShark/Metrics/MetricUploader.cs - Notifications · Progress, warnings, counts, and completion status
LogShark/ProcessingNotificationsCollector.cs
Tests
Primary validation suites
- E2E tests · Artifact-level end-to-end regression coverage
Logshark.Tests/E2E/EndToEndTest.cs - Reader tests · Parser and extraction coverage
Logshark.Tests/LogParser/TableauLogsExtractorTests.cs - Plugin tests · Subsystem parser and transformer coverage
Logshark.Tests/Plugins/PluginManagerTests.cs - Writer tests · Output backend coverage
Logshark.Tests/Writers/Sql/NpgsqlRepositoryTests.cs
Connections 28
- Program → Runner: starts · CLI hands control to the runner
- Runner → Config: reads · Loads command-line parameters and JSON settings
- Runner → Processor: invokes · Begins the main analysis pipeline
- Processor → Extractor: discovers · Resolves log sets and file streams
- Extractor → Log types: yields · Produces normalized file and set metadata
- Extractor → Readers: feeds · Hands raw text to reader strategies
- Readers → Shared parse: uses · Relies on shared timestamp and process parsing
- Processor → Plugin factory: selects · Builds the plugin set for the run
- Plugin factory → Plugin manager: wires · Composes the plugin execution boundary
- Plugin manager → Apache: runs · Executes subsystem analyzers
- Plugin manager → Backgrounder: runs · Executes subsystem analyzers
- Plugin manager → Bridge: runs · Executes subsystem analyzers
- Plugin manager → ResourceMgr: runs · Executes subsystem analyzers
- Plugin manager → Vizql Desktop: runs · Executes subsystem analyzers
- Plugin manager → Config: runs · Executes subsystem analyzers
- Plugin manager → Writer factory: hands off · Passes structured datasets to output creation
- Writer factory → CSV writer: creates · Builds CSV output path
- Writer factory → Hyper writer: creates · Builds Hyper output path
- Writer factory → SQL writer: creates · Builds SQL/Postgres output path
- Hyper writer → Workbooks: packages · Binds extracts into Tableau workbook templates
- SQL writer → Workbooks: supports · Feeds workbook-ready outputs where applicable
- Runner → Metrics: reports · Optional operational telemetry channel
- Runner → Notifications: emits · Captures progress and completion reporting
- Processor → Notifications: updates · Publishes processing results and warnings
- E2E tests → Processor: validates · Checks full run outputs
- Reader tests → Readers: validates · Checks parsing and extraction behavior
- Plugin tests → Plugin manager: validates · Checks plugin orchestration and parsers
- Writer tests → SQL writer: validates · Checks SQL output persistence