# dspinellis/cscout architecture

> Architecture diagram of the GitHub repository dspinellis/cscout, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dspinellis/cscout
- Repository: https://github.com/dspinellis/cscout

## Overview

1. **What this repository is**
   CScout is a C source-code analysis and refactoring system. Its core job is to ingest one or more C projects, model preprocessor-expanded code back onto original files, and store the resulting relationships in a SQLite-backed workspace so they can be queried, visualized, and used for refactoring.

2. **Top-level execution model**
   The primary executable is built from the `src/` C++ code, with `src/cscout.cpp` as the main entry point. Around it are command-line helper scripts for workspace setup, split/merge/reconstruction, and reporting. This is not a server application; the architecture is a batch analysis engine plus generated outputs.

3. **Analysis pipeline**
   The central data flow is: C source files and project definitions (`*.prj`, workspace files) enter the analyzer, are preprocessed and parsed, identifiers and scopes are resolved, call/dependency relationships and metrics are computed, and the results are persisted in the workspace database. The analysis explicitly tracks both C preprocessor scopes and C-language scopes/namespaces.

4. **Front-end: preprocessing, lexing, and parsing**
   The front end is split across tokenizer/parser-related modules such as `cpp.h`, `parse.y`, `eval.y`, `token.*`, `ctoken.*`, `ptoken.*`, `pltoken.*`, `pdtoken.*`, `tokmap.*`, and `macro.*`. This layer handles macro expansion, conditional compilation, token coloring, include processing, and expression evaluation, which are essential because CScout reasons about code before and after preprocessing.

5. **Semantic model and identifier resolution**
   The identifier/scope subsystem lives in modules like `id.*`, `stab.*`, `type.*`, `eclass.*`, `globobj.*`, `initializer.*`, `obfuscate.*`, and related query helpers. This layer distinguishes identifiers across files, scopes, and namespaces, and builds the precise binding model needed for refactoring and cross-file analysis.

6. **Cross-reference and dependency analysis**
   Relationship extraction is handled by modules such as `call.*`, `fcall.*`, `mcall.*`, `fdep.*`, `fileid.*`, `functionid.*`, `funmetrics.*`, `filemetrics.*`, and `metrics.*`. These subsystems compute call graphs, file/function metrics, include relationships, and other dependency views that are later queried or rendered.

7. **Persistence layer: SQLite workspace**
   The repository includes a substantial SQL schema and merge scripts under `src/csmerge/`, plus database access code such as `sql.*`, `workdb.*`, `dbtoken.*`, and `cscout-data.sql`. This is the main persistence boundary: analysis results are normalized into tables, and `csmerge` scripts assemble or extend workspace data from multiple projects/files.

8. **Query and reporting layer**
   Query functionality is organized around `query.*`, `idquery.*`, `filequery.*`, `funquery.*`, `mquery.h`, `filedetails.*`, `dirbrowse.*`, and `gdisplay.*`. These modules expose the analyzed workspace as searchable reports and structured views, including identifier-centric, file-centric, and function-centric queries.

9. **Output generation and visualization**
   The system can emit HTML and graphical output through `html.*`, `webmap.cpp`, `color.cpp`, `pager.*`, and styling assets like `src/style.css`. The docs mention GraphViz `dot`, and the repo includes web templates under `web/` and documentation diagrams under `doc/`, so diagram generation is part of the user-facing reporting path.

10. **Reconstruction and code transformation tools**
   Several scripts support refactoring-oriented workflows: `src/csreconst.sh` reconstructs code, `src/cssplit.py` splits sources, `src/cscut.sh` and `src/csmake.pl` support preprocessing/build workflows, and `src/obfuscate.*` supports name obfuscation. These tools operate on the analyzed model and workspace rather than raw source alone.

11. **Examples, tests, and validation**
   The `example/`, `example.obf/`, and `src/test/` trees provide representative workspaces and regression tests. The tests cover preprocessing edge cases, language features, SQL queries, and csmake/workspace behavior, indicating that correctness depends heavily on exact C language and preprocessor semantics.

12. **Build and operational dependencies**
   The repository is built with GNU Make and uses Perl and Python for auxiliary tooling. SQLite is required for testing and workspace storage; modern C++ is required for the analyzer; and GraphViz is needed for diagrams. CI is configured through GitHub Actions and legacy Travis files, but those are build/validation concerns rather than runtime architecture.

13. **Architectural boundaries to draw**
   For a diagram, the key boundaries are: external C source workspaces on one side; the CScout analysis engine in the middle; the SQLite workspace/database as the persistence boundary; and query/report/reconstruction outputs on the other side. A separate tooling boundary exists for helper scripts, SQL schema/merge assets, and documentation/examples, which support the core engine but are not part of the runtime analysis loop.

Main entry points: [`src/workdb.cpp`](https://github.com/dspinellis/cscout/blob/master/src/workdb.cpp) (Workspace DB), [`src/query.cpp`](https://github.com/dspinellis/cscout/blob/master/src/query.cpp) (Query core), [`src/cscout.cpp`](https://github.com/dspinellis/cscout/blob/master/src/cscout.cpp) (cscout main), [`src/test`](https://github.com/dspinellis/cscout/tree/master/src/test) (Examples/tests)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Analysis engine"]
  node_n1["cscout main<br/>entrypoint<br/>[cscout.cpp]"]
  node_n2["Preprocessor<br/>cpp front end<br/>[cpp.h]"]
  node_n3["Parser & eval<br/>grammar layer<br/>[parse.y]"]
  node_n4["Tokens<br/>token model<br/>[token.cpp]"]
  node_n5["Identifier model<br/>scope/binding<br/>[id.h]"]
  node_n6["Cross refs<br/>relations<br/>[call.cpp]"]
  node_n7["Metrics<br/>analysis stats<br/>[metrics.cpp]"]
end

subgraph group_g2["Workspace store"]
  node_n8[("Workspace DB<br/>sqlite store<br/>[workdb.cpp]")]
  node_n9["SQL schema<br/>schema/scripts<br/>[cscout-data.sql]"]
  node_n10["csmerge<br/>merge pipeline"]
end

subgraph group_g3["Reports & views"]
  node_n11["Query core<br/>query API<br/>[query.cpp]"]
  node_n12["ID queries<br/>search views<br/>[idquery.cpp]"]
  node_n13["File views<br/>search views<br/>[filequery.cpp]"]
  node_n14["Function views<br/>search views<br/>[funquery.cpp]"]
  node_n15["HTML output<br/>reporting<br/>[html.cpp]"]
  node_n16["Visual output<br/>graph/report<br/>[gdisplay.cpp]"]
end

subgraph group_g4["Workflow tools"]
  node_n17["Reconstruction<br/>refactor tools<br/>[csreconst.sh]"]
  node_n18["Build helpers<br/>workflow scripts<br/>[csmake.pl]"]
  node_n19["Obfuscation<br/>transformation<br/>[obfuscate.cpp]"]
end

subgraph group_g5["Support assets"]
  node_n20["Examples/tests<br/>fixtures"]
end

node_n1 -->|"drives"| node_n2
node_n2 -->|"emits"| node_n4
node_n4 -->|"feeds"| node_n3
node_n3 -->|"resolves"| node_n5
node_n5 -->|"binds"| node_n6
node_n6 -->|"computes"| node_n7
node_n5 -->|"persists"| node_n8
node_n6 -->|"persists"| node_n8
node_n7 -->|"persists"| node_n8
node_n9 -->|"drives"| node_n10
node_n10 -->|"assembles"| node_n8
node_n8 -->|"queries"| node_n11
node_n11 -->|"routes"| node_n12
node_n11 -->|"routes"| node_n13
node_n11 -->|"routes"| node_n14
node_n13 -->|"renders"| node_n15
node_n14 -->|"renders"| node_n16
node_n12 -->|"supports"| node_n19
node_n17 -->|"supports"| node_n1
node_n18 -->|"supports"| node_n1
node_n20 -.->|"validates"| node_n1
node_n20 -.->|"validates"| node_n2
node_n20 -.->|"validates"| node_n8

click node_n1 "https://github.com/dspinellis/cscout/blob/master/src/cscout.cpp"
click node_n2 "https://github.com/dspinellis/cscout/blob/master/src/cpp.h"
click node_n3 "https://github.com/dspinellis/cscout/blob/master/src/parse.y"
click node_n4 "https://github.com/dspinellis/cscout/blob/master/src/token.cpp"
click node_n5 "https://github.com/dspinellis/cscout/blob/master/src/id.h"
click node_n6 "https://github.com/dspinellis/cscout/blob/master/src/call.cpp"
click node_n7 "https://github.com/dspinellis/cscout/blob/master/src/metrics.cpp"
click node_n8 "https://github.com/dspinellis/cscout/blob/master/src/workdb.cpp"
click node_n9 "https://github.com/dspinellis/cscout/blob/master/src/cscout-data.sql"
click node_n10 "https://github.com/dspinellis/cscout/tree/master/src/csmerge"
click node_n11 "https://github.com/dspinellis/cscout/blob/master/src/query.cpp"
click node_n12 "https://github.com/dspinellis/cscout/blob/master/src/idquery.cpp"
click node_n13 "https://github.com/dspinellis/cscout/blob/master/src/filequery.cpp"
click node_n14 "https://github.com/dspinellis/cscout/blob/master/src/funquery.cpp"
click node_n15 "https://github.com/dspinellis/cscout/blob/master/src/html.cpp"
click node_n16 "https://github.com/dspinellis/cscout/blob/master/src/gdisplay.cpp"
click node_n17 "https://github.com/dspinellis/cscout/blob/master/src/csreconst.sh"
click node_n18 "https://github.com/dspinellis/cscout/blob/master/src/csmake.pl"
click node_n19 "https://github.com/dspinellis/cscout/blob/master/src/obfuscate.cpp"
click node_n20 "https://github.com/dspinellis/cscout/tree/master/src/test"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3,node_n4,node_n5,node_n6,node_n7 toneBlue
class node_n8,node_n9,node_n10 toneAmber
class node_n11,node_n12,node_n13,node_n14,node_n15,node_n16 toneMint
class node_n17,node_n18,node_n19 toneRose
class node_n20 toneIndigo
```

## Components

### Analysis engine

Core CScout batch analysis pipeline from source ingestion through semantic modeling.

- [cscout main](https://github.com/dspinellis/cscout/blob/master/src/cscout.cpp): entrypoint · Primary executable entry point for batch analysis runs. · `src/cscout.cpp`
- [Preprocessor](https://github.com/dspinellis/cscout/blob/master/src/cpp.h): cpp front end · Macro expansion, conditional compilation, include handling, and token coloring. · `src/cpp.h`
- [Parser & eval](https://github.com/dspinellis/cscout/blob/master/src/parse.y): grammar layer · Yacc-based parsing and compile-time expression evaluation. · `src/parse.y`
- [Tokens](https://github.com/dspinellis/cscout/blob/master/src/token.cpp): token model · Core token representations and token mapping across preprocessing stages. · `src/token.cpp`
- [Identifier model](https://github.com/dspinellis/cscout/blob/master/src/id.h): scope/binding · Identifier, scope, type, and global-object resolution for C and preprocessor namespaces. · `src/id.h`
- [Cross refs](https://github.com/dspinellis/cscout/blob/master/src/call.cpp): relations · Call, dependency, file-id, and function-id relationship extraction. · `src/call.cpp`
- [Metrics](https://github.com/dspinellis/cscout/blob/master/src/metrics.cpp): analysis stats · File and function metrics computed from the analyzed workspace. · `src/metrics.cpp`

### Workspace store

SQLite-backed persistence and merge scripts for normalized analysis results.

- [Workspace DB](https://github.com/dspinellis/cscout/blob/master/src/workdb.cpp): sqlite store · Database access layer that persists the normalized analysis model. · `src/workdb.cpp`
- [SQL schema](https://github.com/dspinellis/cscout/blob/master/src/cscout-data.sql): schema/scripts · Workspace schema and merge/extension scripts for assembling analysis tables. · `src/cscout-data.sql`
- [csmerge](https://github.com/dspinellis/cscout/tree/master/src/csmerge): merge pipeline · SQL and script bundle that combines project outputs into workspace tables. · `src/csmerge`

### Reports & views

Query, browsing, HTML, and visualization outputs built on the workspace.

- [Query core](https://github.com/dspinellis/cscout/blob/master/src/query.cpp): query API · Common query dispatch used by identifier, file, and function queries. · `src/query.cpp`
- [ID queries](https://github.com/dspinellis/cscout/blob/master/src/idquery.cpp): search views · Identifier-centric lookup and reporting over the analyzed workspace. · `src/idquery.cpp`
- [File views](https://github.com/dspinellis/cscout/blob/master/src/filequery.cpp): search views · File-centric browsing, details, and directory exploration. · `src/filequery.cpp`
- [Function views](https://github.com/dspinellis/cscout/blob/master/src/funquery.cpp): search views · Function-centric lookup, graphs, and function detail reporting. · `src/funquery.cpp`
- [HTML output](https://github.com/dspinellis/cscout/blob/master/src/html.cpp): reporting · HTML and web-map generation for browsable analysis output. · `src/html.cpp`
- [Visual output](https://github.com/dspinellis/cscout/blob/master/src/gdisplay.cpp): graph/report · Graph and display generation for rendered dependency and browsing views. · `src/gdisplay.cpp`

### Workflow tools

Helper scripts for reconstruction, splitting, make integration, and obfuscation.

- [Reconstruction](https://github.com/dspinellis/cscout/blob/master/src/csreconst.sh): refactor tools · Code reconstruction and source-splitting workflow helpers. · `src/csreconst.sh`
- [Build helpers](https://github.com/dspinellis/cscout/blob/master/src/csmake.pl): workflow scripts · Make and preprocessing helpers used to prepare analysis workspaces. · `src/csmake.pl`
- [Obfuscation](https://github.com/dspinellis/cscout/blob/master/src/obfuscate.cpp): transformation · Name obfuscation support built on the analyzed identifier model. · `src/obfuscate.cpp`

### Support assets

Example projects, tests, docs, and generated fixtures that validate behavior.

- [Examples/tests](https://github.com/dspinellis/cscout/tree/master/src/test): fixtures · Representative workspaces, regression tests, and expected outputs. · `src/test`

## Connections

- cscout main → Preprocessor: drives · Main executable orchestrates preprocessing.
- Preprocessor → Tokens: emits · Preprocessing produces token streams and mappings.
- Tokens → Parser & eval: feeds · Tokens are parsed and evaluated.
- Parser & eval → Identifier model: resolves · Parsing populates the identifier and scope model.
- Identifier model → Cross refs: binds · Resolved identifiers support relationship extraction.
- Cross refs → Metrics: computes · Relationships feed metric calculation.
- Identifier model → Workspace DB: persists · Semantic state is written to the workspace database.
- Cross refs → Workspace DB: persists · Cross-reference results are stored in SQLite.
- Metrics → Workspace DB: persists · Metrics are normalized into workspace tables.
- SQL schema → csmerge: drives · Schema assets are consumed by merge scripts.
- csmerge → Workspace DB: assembles · Merge scripts build or extend the workspace database.
- Workspace DB → Query core: queries · Reporting layers read from the workspace store.
- Query core → ID queries: routes · Core query dispatch serves identifier queries.
- Query core → File views: routes · Core query dispatch serves file queries.
- Query core → Function views: routes · Core query dispatch serves function queries.
- File views → HTML output: renders · File views can emit HTML pages.
- Function views → Visual output: renders · Function views can emit diagrams and displays.
- ID queries → Obfuscation: supports · Obfuscation uses the identifier model.
- Reconstruction → cscout main: supports · Reconstruction tools operate on analyzed outputs.
- Build helpers → cscout main: supports · Build helpers prepare analysis runs.
- Examples/tests → cscout main: validates · Fixtures and regressions exercise the main pipeline.
- Examples/tests → Preprocessor: validates · Preprocessor tests cover macro and include behavior.
- Examples/tests → Workspace DB: validates · SQL and workspace tests validate persistence behavior.
