How damanish/financial_analysis works
Project Type and Purpose
• This is a full-stack data science pipeline + minimal front end.
• Purpose: Ingest OHLC time series and news headlines, run multiple regressors/classifiers, fuse their outputs via a neural net, and expose results in an HTML page.
File Structure
• data/All_new.csv → Raw dataset (OHLC + news headlines).
• ohlc_regression/ → Polynomial Regression on numerical data.
• KNN_regressor/, Random_Forest_Regressor/, weighted_knn/ → Various regression algorithms.
• SVM_classifier/, SentimentAnalysis/, ANN_classifier/ → Classification models on textual or time‐series data.
• late_fusion_model/ → Multimodal neural network that fuses numerical + textual features; includes app.py for serving.
• Project_Page/Page.html → Front-end UI.
README Highlights
• 3 model stages: Regression, Classification, Fusion.
• Custom implementations (gradient descent, random forest).
• Fusion network built in a Jupyter notebook and served via app.py.
Main Components & Their Relationships
a. Data Layer
– Component: Raw Data Store
– Files: data/All_new.csv, ohlc_regression/ohlc.csv
b. Preprocessing Modules
– Numerical Preprocessor (rolling averages, polynomial features)
– Text Preprocessor (tokenization, lemmatization, TF-IDF)
c. Modeling Modules
– Regression Services
• Polynomial Regression (ohlc_regression/regression.py)
• K-NN Regression (KNN_regressor/price_predict.py)
• Random Forest Regression (Random_Forest_Regressor/price_predictor.py)
• Weighted K-NN (weighted_knn/weighted_knn.py)
– Classification Services
• SVM (SVM_classifier/svm_classifier.py)
• Random Forest Classifier (SentimentAnalysis/sentiment_analysis.py)
• ANN Classifier (ANN_classifier/monthly prediction.py)
d. Fusion Module
– late_fusion_model/final_fusion.ipynb & app.py
– Takes concatenated numerical + text feature vectors → feeds into neural net → outputs final price prediction
e. Serving & Visualization
– Backend API: late_fusion_model/app.py (Flask or similar)
– Frontend: Project_Page/Page.html
Architectural Patterns & Design Principles
• Modular “each-model-in-its-own-folder” structure (loosely a micro-pipeline).
• ETL‐style flow: Extract raw data → Transform (feature engineering) → Load into models → Fuse → Serve.
• Separation of concerns: preprocessing vs modeling vs serving vs UI.
• Manual gradient descent & backprop to show custom ML pipeline.
Technologies & Frameworks
• Python (core)
• pandas, numpy (data handling)
• scikit-learn (KNN, Random Forest, SVM)
• Custom Python scripts (regression.py, sentiment_analysis.py)
• Jupyter notebook for fusion prototyping
• Flask (or FastAPI) in app.py
• HTML/CSS for front end