How dspinellis/cscout works
1. What this repository is
CScout is a C source-code analysis and refactoring system. Its core job is to ingest one or more C projects, model preprocessor-expanded code back onto original files, and store the resulting relationships in a SQLite-backed workspace so they can be queried, visualized, and used for refactoring.
2. Top-level execution model
The primary executable is built from the src/ C++ code, with src/cscout.cpp as the main entry point. Around it are command-line helper scripts for workspace setup, split/merge/reconstruction, and reporting. This is not a server application; the architecture is a batch analysis engine plus generated outputs.
3. Analysis pipeline
The central data flow is: C source files and project definitions (*.prj, workspace files) enter the analyzer, are preprocessed and parsed, identifiers and scopes are resolved, call/dependency relationships and metrics are computed, and the results are persisted in the workspace database. The analysis explicitly tracks both C preprocessor scopes and C-language scopes/namespaces.
4. Front-end: preprocessing, lexing, and parsing
The front end is split across tokenizer/parser-related modules such as cpp.h, parse.y, eval.y, token.*, ctoken.*, ptoken.*, pltoken.*, pdtoken.*, tokmap.*, and macro.*. This layer handles macro expansion, conditional compilation, token coloring, include processing, and expression evaluation, which are essential because CScout reasons about code before and after preprocessing.
5. Semantic model and identifier resolution
The identifier/scope subsystem lives in modules like id.*, stab.*, type.*, eclass.*, globobj.*, initializer.*, obfuscate.*, and related query helpers. This layer distinguishes identifiers across files, scopes, and namespaces, and builds the precise binding model needed for refactoring and cross-file analysis.
6. Cross-reference and dependency analysis
Relationship extraction is handled by modules such as call.*, fcall.*, mcall.*, fdep.*, fileid.*, functionid.*, funmetrics.*, filemetrics.*, and metrics.*. These subsystems compute call graphs, file/function metrics, include relationships, and other dependency views that are later queried or rendered.
7. Persistence layer: SQLite workspace
The repository includes a substantial SQL schema and merge scripts under src/csmerge/, plus database access code such as sql.*, workdb.*, dbtoken.*, and cscout-data.sql. This is the main persistence boundary: analysis results are normalized into tables, and csmerge scripts assemble or extend workspace data from multiple projects/files.
8. Query and reporting layer
Query functionality is organized around query.*, idquery.*, filequery.*, funquery.*, mquery.h, filedetails.*, dirbrowse.*, and gdisplay.*. These modules expose the analyzed workspace as searchable reports and structured views, including identifier-centric, file-centric, and function-centric queries.
9. Output generation and visualization
The system can emit HTML and graphical output through html.*, webmap.cpp, color.cpp, pager.*, and styling assets like src/style.css. The docs mention GraphViz dot, and the repo includes web templates under web/ and documentation diagrams under doc/, so diagram generation is part of the user-facing reporting path.
10. Reconstruction and code transformation tools
Several scripts support refactoring-oriented workflows: src/csreconst.sh reconstructs code, src/cssplit.py splits sources, src/cscut.sh and src/csmake.pl support preprocessing/build workflows, and src/obfuscate.* supports name obfuscation. These tools operate on the analyzed model and workspace rather than raw source alone.
11. Examples, tests, and validation
The example/, example.obf/, and src/test/ trees provide representative workspaces and regression tests. The tests cover preprocessing edge cases, language features, SQL queries, and csmake/workspace behavior, indicating that correctness depends heavily on exact C language and preprocessor semantics.
12. Build and operational dependencies
The repository is built with GNU Make and uses Perl and Python for auxiliary tooling. SQLite is required for testing and workspace storage; modern C++ is required for the analyzer; and GraphViz is needed for diagrams. CI is configured through GitHub Actions and legacy Travis files, but those are build/validation concerns rather than runtime architecture.
13. Architectural boundaries to draw
For a diagram, the key boundaries are: external C source workspaces on one side; the CScout analysis engine in the middle; the SQLite workspace/database as the persistence boundary; and query/report/reconstruction outputs on the other side. A separate tooling boundary exists for helper scripts, SQL schema/merge assets, and documentation/examples, which support the core engine but are not part of the runtime analysis loop.
Main entry points: workdb.cpp Workspace DB · query.cpp Query core · cscout.cpp cscout main · test Examples/tests
Components 20
Analysis engine
Core CScout batch analysis pipeline from source ingestion through semantic modeling.
- cscout main · Primary executable entry point for batch analysis runs.
src/cscout.cpp - Preprocessor · Macro expansion, conditional compilation, include handling, and token coloring.
src/cpp.h - Parser & eval · Yacc-based parsing and compile-time expression evaluation.
src/parse.y - Tokens · Core token representations and token mapping across preprocessing stages.
src/token.cpp - Identifier model · Identifier, scope, type, and global-object resolution for C and preprocessor namespaces.
src/id.h - Cross refs · Call, dependency, file-id, and function-id relationship extraction.
src/call.cpp - Metrics · File and function metrics computed from the analyzed workspace.
src/metrics.cpp
Workspace store
SQLite-backed persistence and merge scripts for normalized analysis results.
- Workspace DB · Database access layer that persists the normalized analysis model.
src/workdb.cpp - SQL schema · Workspace schema and merge/extension scripts for assembling analysis tables.
src/cscout-data.sql - csmerge · SQL and script bundle that combines project outputs into workspace tables.
src/csmerge
Reports & views
Query, browsing, HTML, and visualization outputs built on the workspace.
- Query core · Common query dispatch used by identifier, file, and function queries.
src/query.cpp - ID queries · Identifier-centric lookup and reporting over the analyzed workspace.
src/idquery.cpp - File views · File-centric browsing, details, and directory exploration.
src/filequery.cpp - Function views · Function-centric lookup, graphs, and function detail reporting.
src/funquery.cpp - HTML output · HTML and web-map generation for browsable analysis output.
src/html.cpp - Visual output · Graph and display generation for rendered dependency and browsing views.
src/gdisplay.cpp
Workflow tools
Helper scripts for reconstruction, splitting, make integration, and obfuscation.
- Reconstruction · Code reconstruction and source-splitting workflow helpers.
src/csreconst.sh - Build helpers · Make and preprocessing helpers used to prepare analysis workspaces.
src/csmake.pl - Obfuscation · Name obfuscation support built on the analyzed identifier model.
src/obfuscate.cpp
Support assets
Example projects, tests, docs, and generated fixtures that validate behavior.
- Examples/tests · Representative workspaces, regression tests, and expected outputs.
src/test
Connections 23
- cscout main → Preprocessor: drives · Main executable orchestrates preprocessing.
- Preprocessor → Tokens: emits · Preprocessing produces token streams and mappings.
- Tokens → Parser & eval: feeds · Tokens are parsed and evaluated.
- Parser & eval → Identifier model: resolves · Parsing populates the identifier and scope model.
- Identifier model → Cross refs: binds · Resolved identifiers support relationship extraction.
- Cross refs → Metrics: computes · Relationships feed metric calculation.
- Identifier model → Workspace DB: persists · Semantic state is written to the workspace database.
- Cross refs → Workspace DB: persists · Cross-reference results are stored in SQLite.
- Metrics → Workspace DB: persists · Metrics are normalized into workspace tables.
- SQL schema → csmerge: drives · Schema assets are consumed by merge scripts.
- csmerge → Workspace DB: assembles · Merge scripts build or extend the workspace database.
- Workspace DB → Query core: queries · Reporting layers read from the workspace store.
- Query core → ID queries: routes · Core query dispatch serves identifier queries.
- Query core → File views: routes · Core query dispatch serves file queries.
- Query core → Function views: routes · Core query dispatch serves function queries.
- File views → HTML output: renders · File views can emit HTML pages.
- Function views → Visual output: renders · Function views can emit diagrams and displays.
- ID queries → Obfuscation: supports · Obfuscation uses the identifier model.
- Reconstruction → cscout main: supports · Reconstruction tools operate on analyzed outputs.
- Build helpers → cscout main: supports · Build helpers prepare analysis runs.
- Examples/tests → cscout main: validates · Fixtures and regressions exercise the main pipeline.
- Examples/tests → Preprocessor: validates · Preprocessor tests cover macro and include behavior.
- Examples/tests → Workspace DB: validates · SQL and workspace tests validate persistence behavior.