How bkhmsi/cultural-trends works
Project type and purpose
• Type: Research pipeline for quantifying cultural alignment in LLMs.
• Purpose: (a) Load survey questions and metadata from datasets; (b) generate prompts; (c) query multiple LLMs (OpenAI API and Hugging Face models); (d) store raw completions; (e) post‐process to compute alignment metrics and majority‐vote; (f) enable Anthropological Prompting experiments.
Top‐level components (to represent as distinct blocks)
a. Dataset Store (folder: dataset/)
b. Prompt Generation & Orchestration (scripts/utils.py + script driver)
c. Query Engines
– OpenAI Query (wvs_query_openai.py)
– HF Transformers Query (wvs_query_hf.py)
d. Completions Storage (folder: completions/)
e. Post‐Processing & Analytics
– Majority‐Vote (wvs_majority_vote.py)
– Standard Alignment Compute (wvs_compute_alignment.py)
– Anthropological Alignment Compute (wvs_compute_alignment_anthro.py)
f. External Services
– OpenAI API
– Hugging Face Inference endpoint or local model instances
g. (Optional) Results Export (e.g., CSV/JSON metrics)
Data flows & relationships
1. Data Ingestion
– Read WVS questions and templates from dataset/ (CSV, JSON, YAML).
2. Prompt Generation
– utils.py builds prompt payloads using the loaded templates and question data.
3. Model Query
– Pass prompt payload to
• OpenAI API via wvs_query_openai.py
• HF models via wvs_query_hf.py
– Response → JSON
4. Completions Storage
– Persist raw JSON under completions/<model>/<country>/<lang>/….
5. Aggregation & Analysis
– wvs_majority_vote.py reads completions → computes consensus answers.
– wvs_compute_alignment.py reads completions + ground‐truth survey data → computes alignment scores.
– wvs_compute_alignment_anthro.py repeats alignment compute under Anthropological Prompting variations.
6. Output
– Write metrics (JSON/CSV) for further reporting or visualization.
Architectural patterns & principles
• Modular pipeline: each stage is isolated in its own script.
• Configuration‐driven: uses YAML/JSON templates for prompts and survey metadata.
• File‐based storage: completions and datasets live on disk with hierarchical naming.
• Separation of concerns: data ingest, model querying, and analysis are distinct.
Technologies & Frameworks to annotate
– Python 3.x
– OpenAI Python SDK (for GPT models)
– Hugging Face Transformers library
– JSON, CSV, YAML for data interchange
– Standard Python file I/O and OS path conventions