How kanewilliams/political_gang_discourse works
Main Components
• Data Ingestion:
• Corpus Files: The project includes a .7z archive (NZ Parliament Transcripts Gang Corpus.7z) containing the scraped transcripts, along with supporting text files (Corpus_Information.txt and webscrape-IO_sitemap.txt) and the essay PDF that provides background.
• Data Preprocessing:
• Scripts & Notebooks: The preprocess.ipynb notebook and process_txt.py script are responsible for parsing, cleaning, and structuring the raw text from the transcripts.
• Analysis Modules:
• Analysis Techniques: The ReadMe mentions various analysis approaches (keyness, concordance, collocation, and word frequency). Each of these would be represented as separate modules or stages in the analysis pipeline.
• Output & Presentation:
Relationships and Interactions
• The preprocessed data then flows into the analysis stage where different analytical processes are executed in parallel or sequentially:
• Keyness Analysis
• Concordance Analysis
• Collocation Analysis
• Word Frequency Analysis
• The outputs of these analyses are then consolidated to support the final conclusions and are presented in the final essay document along with supplementary visualizations.
Architectural Patterns & Design Principles
• Modular Design: Each major stage (data ingestion, pre-processing, analysis, and output) should be clearly separated into distinct modules.
• Extensibility: The design should allow for adding new analysis modules or modifying pre-processing steps if additional text features are needed in future iterations.
• Documentation: Include clear labels and annotations to reflect the underlying research methodology.