How ms20190155/measuring-corporate-culture-using-machine-learning works
Main Components
a. Data Stores (persistent folders/files):
• Raw Input: data/input/{documents.txt, document_ids.txt, id2firms.csv}
• Parsed Output: data/processed/parsed/{documents.txt, document_sent_ids.txt}
• Cleaned Corpora:
• data/processed/unigram/documents_cleaned.txt
• data/processed/bigram/documents_cleaned.txt
• data/processed/trigram/documents_cleaned.txt
• Models:
• models/phrases/{bigram.mod, trigram.mod}
• models/w2v/w2v.mod
• Outputs:
• outputs/dict/expanded_dict.csv
• outputs/scores/{scores_TF.csv, scores_TFIDF.csv, scores_WFIDF.csv}
• (Optional) aggregated firm‐time scores via aggregate_firms.py
b. Processing Modules (Python scripts):
• parse.py / parse_parallel.py
• clean_and_train.py
• create_dict.py
• score.py
• aggregate_firms.py
c. Configuration:
• global_options.py (CoreNLP path, RAM, CPU cores, seed words, etc.)
d. External Service:
• Stanford CoreNLP server (Java process)
Relationships & Data Flow
1) Raw Input → parse.py (or parse_parallel.py)
• Calls CoreNLP server to produce sentence‐level parses → data/processed/parsed/
2) Parsed Data → clean_and_train.py
• Cleans text, removes stopwords/NERs, learns bigram/trigram phrases (gensim)
• Outputs cleaned corpora (uni/bi/tri) → trains word2vec → models/
3) Word2vec Model & Seed Words → create_dict.py
• Expands dictionary dimensions → outputs expanded_dict.csv
4) Expanded Dictionary + Cleaned Corpus → score.py
• Computes TF / TF-IDF / WFIDF scores → outputs scores_*.csv
5) Document‐level Scores + id2firms.csv → aggregate_firms.py (Optional)
• Aggregates to firm‐time level
Architectural Patterns & Principles
• Pipeline Pattern: Strict sequential stages, each producing data for the next.
• Modularity: Each Python script implements a single concern.
• Batch Processing: No real‐time requests; designed for parallelizable workloads.
• External Integration: Reliance on an external NLP service (Stanford CoreNLP).
Technologies & Frameworks
• Python 3.6+ (core language)
• Stanford CoreNLP v3.9.2 (parsing)
• gensim (phrase models & word2vec)
• pandas / numpy (data handling)
• CSV / plaintext files for storage
Section 1 – Data Ingestion & Parsing:
[data/input/*] → parse.py → [data/processed/parsed/*]
(note: parse_parallel.py is alternative path)
Section 2 – Cleaning & Model Training:
[data/processed/parsed/*] + StopWords → clean_and_train.py →
• [data/processed/unigram/*], [data/processed/bigram/*], [data/processed/trigram/*]
• [models/phrases/*], [models/w2v/w2v.mod]
Section 3 – Dictionary Expansion:
[models/w2v/w2v.mod] + seed words → create_dict.py → [outputs/dict/expanded_dict.csv]
Section 4 – Scoring:
[data/processed/trigram/*] + [expanded_dict.csv] → score.py → [outputs/scores/*.csv]
Section 5 – Aggregation (Optional):
[outputs/scores/*.csv] + [data/input/id2firms.csv] → aggregate_firms.py → final aggregated output