How intel-spark/corenlp works
1. What this repository is
Stanford CoreNLP is a large Java NLP platform organized as a library plus command-line tools, servers, demos, and web apps. The top-level architecture is not a single app, but a set of reusable NLP components wired together through a common annotation pipeline.
2. Core data model and control flow
The central abstraction is the Annotation / CoreMap object in src/edu/stanford/nlp/pipeline and src/edu/stanford/nlp/util / src/edu/stanford/nlp/ling. Raw text is converted into an Annotation, then annotators add structured outputs step by step: tokens, sentences, POS tags, lemmas, NER, parse trees, dependencies, coreference chains, sentiment, time expressions, and extracted relations.
3. Pipeline orchestration layer
StanfordCoreNLP, AnnotationPipeline, AnnotatorPool, and the Annotator implementations define the main execution boundary. This layer is responsible for ordering annotators, enforcing requirements, and passing shared document state between modules. It is the primary integration point for users of the library and for server/web entrypoints.
4. Tokenization, sentence splitting, and text normalization
The process package and several pipeline annotators handle the early text normalization stages: tokenization (PTBTokenizer, language-specific tokenizers), sentence segmentation, XML stripping, truecasing, stemming/morphology, and word-shape normalization. This is where unstructured text becomes a token sequence that downstream components can consume.
5. Sequence labeling and classification subsystem
The sequences, classify, ie/crf, tagger/maxent, and regexp packages implement the statistical sequence models used for POS tagging, NER, and related labeling tasks. These packages contain both model abstractions and training/inference code, with supporting optimization in optimization, maxent, and math. The repository also includes example datasets and property files for model configuration.
6. Parsing subsystem
Syntactic parsing is split across parser, parser/lexparser, parser/shiftreduce, parser/nndep, and parser/dvparser. This includes the legacy lexicalized parser, a shift-reduce parser, a neural dependency parser, and evaluation tooling. Parser outputs feed later annotators such as dependency conversion, coreference, and OpenIE.
7. Coreference subsystem
Coreference is represented by multiple generations of code: dcoref, hcoref, and scoref. These packages implement mention detection, sieves/rules, clustering, scoring, and document readers. They depend heavily on earlier pipeline stages, especially tokenization, parsing, NER, and speaker/entity metadata.
8. Higher-level semantic extraction
naturalli and the pipeline annotators for relation extraction and entity mentions sit above the basic linguistic layers. OpenIE/Natural Logic uses parse trees and dependencies to extract relation triples. ie/machinereading, ie/kbp*, and patterns support rule-based and statistical extraction workflows for entities, relations, and pattern mining.
9. Language-specific processing
The international and trees/international subtrees contain language-specific tokenizers, tree normalizers, head finders, grammars, and pipeline adaptations for Arabic, Chinese, French, German, Hebrew, and Spanish. These are not just data files; they define alternate processing pipelines and treebank conventions that branch from the English defaults.
10. Tree, dependency, and graph infrastructure
trees, semgraph, graph, fsm, and tregex provide the shared structural representation layer. Constituency trees, typed dependencies, semantic graphs, and graph algorithms are used across parsing, OpenIE, coreference, and tree transformation tools. tregex/tsurgeon are especially important as pattern/query and tree-editing utilities used throughout the ecosystem.
11. Temporal and numeric normalization
time and ie/qe implement specialized extraction for dates, times, holidays, and quantifiable entities. These modules behave like focused annotators built on top of tokenization and sentence boundaries, and their outputs become additional annotations on the same document model.
12. Serving, demos, and UI entrypoints
The repo includes multiple deployment surfaces: pipeline/webapp, parser/webapp, ie/ner/webapp, naturalli/demo, and time/suservlet, plus CLI scripts under scripts and docs under doc. StanfordCoreNLPServer, StanfordCoreNLPClient, servlet classes, and demo frontends show that the same core pipeline can be embedded in an HTTP service or run interactively.
13. Training, configuration, and artifacts
scripts, data, and many .properties, .txt, .prop, and .conf files define model training recipes, language configs, and test corpora. The codebase is meant to be paired with external model jars, and the repo also carries generated parser/lexer sources, implying build-time generation from JFlex/JavaCC but source-controlled outputs for convenience.
14. Build and dependency boundaries
The project is primarily Java, built with Ant and also supported by Gradle files. External libraries materially shape the architecture: servlet APIs and Tomcat for web endpoints, Lucene for indexing/pattern search, Protobuf for serialized annotations, Joda-Time/Jollyday for temporal reasoning, EJML for linear algebra, and logging via Log4j/SLF4J/Redwood.
15. Testing and regression coverage
test, itest, and test/regression provide a broad test matrix across nearly every subsystem: tokenization, parsing, NER, coref, semantic graphs, patterns, time, and utilities. The test structure mirrors the production package layout, which is useful for drawing subsystem boundaries in an architecture diagram.
16. Architectural summary
The dominant pattern is a shared-document pipeline with many pluggable annotators layered on common linguistic representations. Most subsystems either produce annotations into that shared document or consume them for higher-level inference. For a diagram, the clearest spine is: input text → normalization/tokenization → sentence/sequence labeling → syntax/dependencies → semantic extraction/coreference/time/sentiment → structured outputs, with language-specific branches and server/web/CLI entrypoints wrapping the same core pipeline.
Main entry points: TokenizerAnnotator.java Annotators · StanfordCoreNLPServer.java Entry points · StanfordCoreNLP.java Pipeline core · LexicalizedParser.java Parsing
Components 19
Core pipeline
Shared document model and annotator orchestration.
- Annotation model · Shared document and per-token/per-sentence annotation containers used throughout the system.
src/edu/stanford/nlp/pipeline/Annotation.java - Pipeline core · Annotator pipeline, pool, factories, and top-level CoreNLP driver that order and run stages.
src/edu/stanford/nlp/pipeline/StanfordCoreNLP.java - Annotators · Main built-in annotators for tokenization, sentence split, POS, parse, NER, regex, sentiment, coref, and relations.
src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java - Output formats · Annotation serializers and outputters for JSON, XML, CoNLL, CoNLLU, protobuf, and text.
src/edu/stanford/nlp/pipeline/ProtobufAnnotationSerializer.java
NLP stages
Text processing, tagging, parsing, semantics, and coreference.
- Text processing · Tokenization, sentence splitting, XML stripping, truecasing, morphology, stemming, and word shape normalization.
src/edu/stanford/nlp/process/PTBTokenizer.java - Sequence models · CRF, maxent, and sequence decoding infrastructure used by POS tagging, NER, and similar tasks.
src/edu/stanford/nlp/sequences/SequenceModel.java - Classifiers · General-purpose classifiers, datasets, objective functions, and linear/logistic model training code.
src/edu/stanford/nlp/classify/Classifier.java - Parsing · Lexicalized, shift-reduce, neural dependency, and evaluation tooling for constituency and dependency parsing.
src/edu/stanford/nlp/parser/lexparser/LexicalizedParser.java - Coreference · Deterministic, hybrid, and statistical coreference systems with mention detection, sieves, clustering, and scoring.
src/edu/stanford/nlp/dcoref/CoreferenceSystem.java - Semantics · OpenIE, natural logic, clause splitting, relation triples, and higher-level semantic inference.
src/edu/stanford/nlp/naturalli/OpenIE.java - Time/number · Temporal expression extraction, holiday reasoning, and quantifiable entity normalization.
src/edu/stanford/nlp/time/TimeAnnotator.java - Pattern mining · Surface, dependency, and tokensregex-based pattern application and scoring workflows.
src/edu/stanford/nlp/patterns/Pattern.java
Infrastructure
Shared representations, utilities, and learning/tooling support.
- Tree infra · Tree, dependency, grammar, head-finding, and tree transformation infrastructure shared by parsing and downstream analysis.
src/edu/stanford/nlp/trees/Tree.java - Graph infra · Generic graph, shortest-path, and connected-components utilities used by semantic and dependency structures.
src/edu/stanford/nlp/graph/Graph.java - Semantic graphs · SemanticGraph, Semgrex, and Ssurgeon tools for dependency-style graph analysis and rewriting.
src/edu/stanford/nlp/semgraph/SemanticGraph.java - Build/data · Ant/Gradle build files, scripts, model properties, corpora, and bundled data artifacts.
build.xml
Entry points
Servers, demos, and UI surfaces.
- Entry points · CLI tools, servers, servlets, demos, and webapp entry surfaces built on the shared pipeline.
src/edu/stanford/nlp/pipeline/StanfordCoreNLPServer.java
Language branches
Language-specific tokenization and tree/pipeline adaptations.
- Lang processing · Language-specific tokenizers, segmenters, and pipeline adaptations for Arabic, Chinese, French, and Spanish.
src/edu/stanford/nlp/international - Lang trees · Language-specific treebank language packs, head finders, and normalizers for non-English tree conventions.
src/edu/stanford/nlp/trees/international
Connections 24
- Text processing → Annotation model: creates · Raw text is turned into annotations and token sequences.
- Annotation model → Pipeline core: flows through · The shared document state is passed between annotators.
- Pipeline core → Annotators: runs · StanfordCoreNLP assembles and executes annotator stages.
- Annotators → Text processing: uses · Early annotators depend on tokenization and sentence splitting.
- Annotators → Sequence models: uses · Tagging and NER rely on sequence labeling components.
- Annotators → Parsing: uses · Parse and dependency annotators invoke parser subsystems.
- Annotators → Coreference: uses · Coreference annotators consume earlier linguistic annotations.
- Annotators → Semantics: uses · OpenIE, relation extraction, and natural logic build on syntax and dependencies.
- Annotators → Time/number: uses · Time and quantity annotators add specialized document annotations.
- Pipeline core → Output formats: emits · Pipeline results can be written through multiple output formats.
- Parsing → Tree infra: produces · Parsing centers on tree and dependency representations.
- Parsing → Semantic graphs: converts to · Dependency and semantic outputs are exposed as graph structures.
- Tree infra → Semantic graphs: feeds · Tree and dependency structures underpin semantic graph construction.
- Semantic graphs → Semantics: supports · Semgrex and dependency graphs are used by OpenIE and related extraction.
- Sequence models → Classifiers: shares math · Labeling models reuse classifier and optimization infrastructure.
- Classifiers → Build/data: trained with · Model recipes and corpora are stored with the repository.
- Pattern mining → Semantics: feeds · Pattern-based extraction contributes to higher-level semantic workflows.
- Pattern mining → Annotators: wraps · TokensRegex and regex NER are exposed as pipeline annotators.
- Lang processing → Text processing: specializes · Language-specific preprocessing extends the generic text pipeline.
- Lang trees → Tree infra: specializes · Language-specific tree conventions extend shared tree infrastructure.
- Entry points → Pipeline core: hosts · Servers and demos embed the same core pipeline for interactive use.
- Entry points → Parsing: hosts · Parser-specific web and UI entry points wrap parsing components.
- Entry points → Time/number: hosts · SUTime servlet and demo surfaces expose temporal extraction.
- Entry points → Build/data: depends on · CLI and web launches are configured via scripts and properties.