# ms20190155/measuring-corporate-culture-using-machine-learning architecture

> Architecture diagram of the GitHub repository ms20190155/measuring-corporate-culture-using-machine-learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ms20190155/measuring-corporate-culture-using-machine-learning
- Repository: https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning

## Overview

##### Main Components
a. Data Stores (persistent folders/files):
- Raw Input: data/input/{documents.txt, document_ids.txt, id2firms.csv}
- Parsed Output: data/processed/parsed/{documents.txt, document_sent_ids.txt}
- Cleaned Corpora:
- data/processed/unigram/documents_cleaned.txt
- data/processed/bigram/documents_cleaned.txt
- data/processed/trigram/documents_cleaned.txt
- Models:
- models/phrases/{bigram.mod, trigram.mod}
- models/w2v/w2v.mod
- Outputs:
- outputs/dict/expanded_dict.csv
- outputs/scores/{scores_TF.csv, scores_TFIDF.csv, scores_WFIDF.csv}
- (Optional) aggregated firm‐time scores via aggregate_firms.py
b. Processing Modules (Python scripts):
- parse.py / parse_parallel.py
- clean_and_train.py
- create_dict.py
- score.py
- aggregate_firms.py
c. Configuration:
- global_options.py (CoreNLP path, RAM, CPU cores, seed words, etc.)
d. External Service:
- Stanford CoreNLP server (Java process)

##### Relationships & Data Flow
1) Raw Input → parse.py (or parse_parallel.py)
- Calls CoreNLP server to produce sentence‐level parses → data/processed/parsed/
2) Parsed Data → clean_and_train.py
- Cleans text, removes stopwords/NERs, learns bigram/trigram phrases (gensim)
- Outputs cleaned corpora (uni/bi/tri) → trains word2vec → models/
3) Word2vec Model & Seed Words → create_dict.py
- Expands dictionary dimensions → outputs expanded_dict.csv
4) Expanded Dictionary + Cleaned Corpus → score.py
- Computes TF / TF-IDF / WFIDF scores → outputs scores_*.csv
5) Document‐level Scores + id2firms.csv → aggregate_firms.py (Optional)
- Aggregates to firm‐time level

##### Architectural Patterns & Principles
- Pipeline Pattern: Strict sequential stages, each producing data for the next.
- Modularity: Each Python script implements a single concern.
- Batch Processing: No real‐time requests; designed for parallelizable workloads.
- External Integration: Reliance on an external NLP service (Stanford CoreNLP).

##### Technologies & Frameworks
- Python 3.6+ (core language)
- Stanford CoreNLP v3.9.2 (parsing)
- gensim (phrase models & word2vec)
- pandas / numpy (data handling)
- CSV / plaintext files for storage

Section 1 – Data Ingestion & Parsing:
[data/input/*] → parse.py → [data/processed/parsed/*]
(note: parse_parallel.py is alternative path)
Section 2 – Cleaning & Model Training:
[data/processed/parsed/*] + StopWords → clean_and_train.py →
- [data/processed/unigram/*], [data/processed/bigram/*], [data/processed/trigram/*]
- [models/phrases/*], [models/w2v/w2v.mod]
Section 3 – Dictionary Expansion:
[models/w2v/w2v.mod] + seed words → create_dict.py → [outputs/dict/expanded_dict.csv]
Section 4 – Scoring:
[data/processed/trigram/*] + [expanded_dict.csv] → score.py → [outputs/scores/*.csv]
Section 5 – Aggregation (Optional):
[outputs/scores/*.csv] + [data/input/id2firms.csv] → aggregate_firms.py → final aggregated output

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Stores
    subgraph "Data Stores"
        direction TB
        raw_input["Raw Input\n(data/input/*)"]:::datastore
        parsed_output["Parsed Output\n(data/processed/parsed/*)"]:::datastore
        cleaned_unigram["Unigram Corpus\n(data/processed/unigram/*)"]:::datastore
        cleaned_bigram["Bigram Corpus\n(data/processed/bigram/*)"]:::datastore
        cleaned_trigram["Trigram Corpus\n(data/processed/trigram/*)"]:::datastore
        models_phrases["Phrase Models\n(models/phrases/*)"]:::datastore
        model_w2v["Word2Vec Model\n(models/w2v/w2v.mod)"]:::datastore
        expanded_dict["Expanded Dictionary\n(outputs/dict/expanded_dict.csv)"]:::datastore
        scores["Scores\n(outputs/scores/*.csv)"]:::datastore
        aggregated_output["Aggregated Output\n(final)"]:::datastore
    end

    %% Processing Modules
    subgraph "Processing Modules"
        direction TB
        parse["parse.py"]:::process
        parse_par["parse_parallel.py"]:::process
        clean["clean_and_train.py"]:::process
        create_dict["create_dict.py"]:::process
        score["score.py"]:::process
        aggregate["aggregate_firms.py"]:::process
        preprocess["culture/preprocess.py"]:::process
        preprocess_par["culture/preprocess_parallel.py"]:::process
        culture_models["culture/culture_models.py"]:::process
        culture_dict["culture/culture_dictionary.py"]:::process
        file_util["culture/file_util.py"]:::process
        compute_contrib["compute_score_contribution.py"]:::process
        stopwords["StopWords List\n(resources/StopWords_Generic.txt)"]:::process
    end

    %% Configuration
    global_opts["global_options.py"]:::config

    %% External Service
    corenlp["Stanford CoreNLP"]:::external

    %% Data Flow
    raw_input --> parse
    raw_input --> parse_par
    parse --> corenlp
    parse_par --> corenlp
    corenlp --> parsed_output
    parsed_output --> clean
    stopwords --> clean
    clean --> cleaned_unigram
    clean --> cleaned_bigram
    clean --> cleaned_trigram
    clean --> models_phrases
    clean --> model_w2v
    culture_models --> clean
    model_w2v --> create_dict
    culture_dict --> create_dict
    create_dict --> expanded_dict
    cleaned_trigram --> score
    expanded_dict --> score
    file_util --> parse
    file_util --> parse_par
    file_util --> clean
    file_util --> create_dict
    file_util --> score
    file_util --> aggregate
    score --> scores
    raw_input -->|"id2firms.csv"| aggregate
    scores --> aggregate
    aggregate --> aggregated_output

    %% Config connections
    global_opts -.-> parse
    global_opts -.-> parse_par
    global_opts -.-> clean
    global_opts -.-> create_dict
    global_opts -.-> score
    global_opts -.-> aggregate

    %% Click Events
    click parse "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/parse.py"
    click parse_par "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/parse_parallel.py"
    click global_opts "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/global_options.py"
    click clean "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/clean_and_train.py"
    click preprocess "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/culture/preprocess.py"
    click preprocess_par "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/culture/preprocess_parallel.py"
    click stopwords "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/resources/StopWords_Generic.txt"
    click culture_models "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/culture/culture_models.py"
    click create_dict "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/create_dict.py"
    click culture_dict "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/culture/culture_dictionary.py"
    click score "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/score.py"
    click compute_contrib "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/compute_score_contribution.py"
    click aggregate "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/aggregate_firms.py"
    click file_util "https://github.com/ms20190155/measuring-corporate-culture-using-machine-learning/blob/master/culture/file_util.py"

    %% Styles
    classDef process fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef datastore fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
```
