How agarwaldevesh374/gsoc-25-humanai-issr2-screening-test--nlp-tool works
Project Overview and Purpose
• This is a Python-based machine learning and NLP project aimed at communication analysis during driving simulations.
• It processes video files to extract audio, cleans the audio, transcribes it, performs speaker diarization and sentiment analysis, and then visualizes the results.
• Advanced techniques (e.g., speaker embedding via pretrained models) along with dynamic reporting and visualization are key features.
Main Components
a. Data Ingestion:
• Input Videos: Represented by the "files_audio_video/input_videos" directory where users add the raw video files.
b. Audio/Video Processing Pipeline:
• Audio Extraction: Use the "audio_extraction.py" script to extract audio from video.
• Noise Removal: Display "audio_noiseremoval.py" for cleaning the extracted audio.
• Speaker Diarization: Show the "speaker_diarization.py" step that leverages the pretrained model in "pretrained_models/spkrec-ecapa-voxceleb".
• Transcription: Include "transcription.py" for converting audio to text.
• Sentiment Analysis: Include "sentiment_analysis.py" to classify sentiment (Positive, Negative, Neutral).
• Process Orchestration: "process_all_videos.py" ties all these steps together.
c. Data Storage and Outputs:
• Processed Data Storage: CSV files and cleaned audio are stored in "files_audio_video/output_audios_csv" and "cleaned_audio".
• Visualization: The "visualization_plots.py" script produces output plots (targeting "Output Plots" directory).
• Additional Data: Also include the "transcript_with_speakers.csv" output.
d. External Dependencies:
• Pretrained Models: Represent the pretrained speaker recognition model files (classifier, embedding_model, hyperparams, etc.) that the diarization component relies on.
• Python Environment: Note that the setup is based on a virtual environment with Python 3.11.
Relationships and Data Flow
• Sequential processing:
• Audio Extraction → Noise Removal → Speaker Diarization → Transcription → Sentiment Analysis.
• Each of these steps feeds into a CSV generation step in the output directory.
• CSV output to the Visualization component.
• Involvement of pretrained models that are ‘called’ or ‘referenced’ by the speaker diarization functionality.
• A side note for the file storing “cleaned_audio” as an output of the noise removal step.
Architectural Patterns and Design Principles
• Pipeline Design: Represent the project as a clear sequential processing pipeline where each module is a distinct component.
• Modularity: Each processing script (audio extraction, noise removal, etc.) is a modular component that can be executed separately.
• Separation of Concerns: Emphasize that ingestion, processing, external module integration (pretrained models), and visualization are clearly separated.