How haskell/text works
Project type and purpose
• This is a Haskell library (“text”) that provides efficient, Unicode‐aware text operations. It is *not* a web application but a systems library with both Haskell and C/C++ (via FFI) components, plus test and benchmark harnesses.
File structure
• Top‐level
• src/ — Haskell source organized into public API (Data.Text, Data.Text.Lazy, etc.) and internal modules
• cbits/ — C and C++ implementations (measure_off.c, validate_utf8.cpp, and the bundled simdutf library)
• scripts/ — Haskell code generators (e.g., CaseFolding, SpecialCasing) and a regenerate.sh driver
• benchmarks/ — Benchmark definitions and data harnesses
• tests/ — Test suites (unit, QuickCheck properties, shell scripts)
• Setup.lhs + text.cabal — Cabal build instructions
• .github/workflows/… — CI pipelines for various platforms
README for additional insights
• Core feature: space/time‐efficient Unicode text operations
• Benchmarks and tests are first‐class citizens
• Uses stream fusion and UTF‐8 under the hood
Capturing essential details
• Don’t stop at “src” vs. “cbits”—break src into API vs. internal subpackages (IO, Builder, Fusion, Encoding).
• That scripts regenerate code automatically as part of the build.
• Bundled simdutf library as an external‐but‐included component.
• How benchmarks rely on external test data (benchmarks/text-test-data).
• What each part of the library does
• How they depend on each other
• Where performance is offloaded to native code
• How testing and benchmarking fit into the life cycle
• How CI/CD orchestrates builds on different targets