How sujith0613/craxpert works
1. System shape
- Craxpert is a single-process Go service with two main faces: a backend job simulator and a read-only browser dashboard.
- There is no database; durability comes from an append-only write-ahead log (jobs.log) plus periodic checkpointing.
2. Entry point and server bootstrap
- main.go is the orchestration layer: it starts the HTTP server, boots any restored state from disk, and likely wires together the auto job generator and background checkpoint worker.
- The backend runs on Go’s standard net/http stack.
3. Job domain model and shared in-memory state
- job.go holds the core job representation, job states, and the global job registry/map.
- The in-memory job store is protected with sync.RWMutex, which is central to concurrency safety across handlers, workers, and reset/reload paths.
4. Worker pool and execution pipeline
- worker.go contains the concurrent worker-pool logic that consumes jobs, simulates work, and handles retries.
- Jobs transition through states such as PENDING, RUNNING, SUCCESS, FAILED, and RETRYING, with workers updating shared state as they execute.
5. Persistence boundary: WAL
- wal.go is the durability layer: every job state transition is serialized to disk before or during execution, so the simulator can recover after crashes.
- The WAL is append-only and is the authoritative recovery source on startup.
6. Checkpointing / compaction
- The readme describes a background checkpoint process that periodically snapshots active jobs and compacts the WAL.
- This likely uses a staging file such as jobs.tmp before replacing the live log, reducing replay time on restart.
7. Job ingestion paths
- Jobs enter the system from two sources: the HTTP API and an automatic generator.
- The API accepts manual job submission, while the auto-generator continuously creates simulated workload for the worker pool.
8. HTTP API and control plane
- The server exposes endpoints for job submission and operational controls such as reset, pool resizing, cancellation, and status retrieval.
- These endpoints are the main boundary between external clients and internal job state.
9. Frontend dashboard
- web/index.html is a single-page, vanilla HTML/CSS/JavaScript dashboard.
- It is not the control source of truth; it mostly polls backend status and visualizes the job queue, worker activity, and log history.
10. Observability surfaces
- The dashboard exposes live queue state, raw WAL/log viewing, and connection monitoring.
- The terminal monitor described in the README is a secondary observability surface for local runtime inspection.
11. Utility layer
- utils.go is a small support module, likely housing random generation and helper functions used by job creation or simulation timing.
- It is not a major subsystem, but it supports the workload generator and worker simulation.
12. Key architectural boundary
- The important design split is: ephemeral concurrent execution in memory versus durable state in the WAL.
- All meaningful state transitions flow from API or generator -> in-memory job registry/workers -> WAL append -> dashboard/status consumers, with startup recovery replaying the log back into memory.
Main entry points: job.go In-memory state · worker.go Workers · main.go Main · wal.go WAL
Components 16
Runtime
Process startup, HTTP serving, job execution, and shared in-memory coordination.
- Main · Starts the server, restores state, and wires background loops.
main.go - Job model · Core job types, states, and shared registry.
job.go - Workers · Concurrent worker pool that simulates job processing and retries.
worker.go - HTTP API · Endpoints for job submission and operational control.
main.go - Auto generator · Continuously creates simulated jobs to feed the workers.
main.go - In-memory state · Shared job registry protected by read/write locking.
job.go - Status view · Current queue, worker, and job-state view exposed to clients.
main.go - Cancellation · Stops or marks jobs through operational endpoints.
main.go - Pool resize · Adjusts worker concurrency at runtime.
main.go - Utils · Shared helper functions for generation and simulation support.
utils.go
Durability
Append-only log and checkpointing that restore state after restart.
- WAL · Durable record of job transitions used for crash recovery.
wal.go - Checkpointing · Snapshots active jobs and compacts the write-ahead log.
wal.go - Recovery · Boot-time log replay that rebuilds in-memory state.
wal.go
Browser UI
Read-only dashboard and operational visibility into the running simulator.
Other components
- Terminal monitor · Local runtime inspection surface mentioned by the README.
Connections 21
- Main → HTTP API: serves · Bootstraps the HTTP control plane.
- Main → Workers: starts · Launches the execution pool.
- Main → Auto generator: starts · Runs the automatic workload generator.
- Main → Recovery: restores · Rebuilds state from the durable log on startup.
- HTTP API → In-memory state: mutates · Creates, cancels, and manages jobs through the shared registry.
- Auto generator → In-memory state: creates · Injects new jobs into memory for execution.
- Workers → In-memory state: updates · Transitions jobs through execution states.
- Workers → WAL: appends · Persists state transitions for recovery.
- HTTP API → WAL: records · Operational changes are written durably.
- Recovery → In-memory state: rebuilds · Replays log entries back into memory.
- Checkpointing → WAL: compacts · Rewrites the log after snapshotting active jobs.
- Checkpointing → In-memory state: snapshots · Captures active in-memory jobs for compaction.
- Dashboard → Status view: polls · Reads live status from the backend.
- Dashboard → Log view: shows · Visualizes queue and log history.
- Status view → In-memory state: reads · Exposes the current job registry without mutation.
- Cancellation → In-memory state: marks · Changes job lifecycle state via control endpoints.
- Pool resize → Workers: reconfigures · Changes worker pool size at runtime.
- Utils → Auto generator: supports · Provides helper logic for job creation and timing.
- Utils → Workers: supports · Provides helper logic for simulation behavior.
- Terminal monitor → In-memory state: observes · Secondary local visibility into runtime activity.
- Terminal monitor → WAL: observes · Can inspect durability history.