How sanjay-krishna-mv/product-analytics-project works
1. Project type and overall shape
- This repository is a product analytics pipeline rather than an application service.
- The architecture is organized as a data flow from raw session data → warehouse transformation → analytics notebooks → business insights and prediction.
2. Primary data source
- The main input data lives in data/raw/cleaned_speakers_data.csv.
- Per the README, this CSV is uploaded into Google BigQuery as a raw_sessions table.
- The data/processed/events.csv file appears to be a local processed/exported artifact of the engineered event stream.
3. Warehouse and storage boundary
- BigQuery is the central system boundary for structured analytics data.
- The repository models two key warehouse layers:
- raw_sessions: the ingested source table
- events: the transformed, granular event stream used for analysis
- This separation is important for architecture because raw ingestion and analytical modeling are intentionally decoupled.
4. Transformation layer
- sql/01_feature_engineering_events.sql is the main transformation script.
- It converts session-level raw data into an event-level model that resembles product analytics platforms like Amplitude or Pendo.
- The SQL directory also contains downstream analytical queries, so SQL is both the transformation and metric-computation layer.
5. Core analytical workflows
- sql/02_product_metrics_dau_wau_mau.sql computes product health metrics such as DAU/WAU/MAU.
- sql/03_funnel_analysis_query.sql performs funnel analysis to locate drop-off points.
- sql/04_deep_dive_dropped_user_segmentation.sql and sql/05_deep_dive_dropped_users_by_device_traffic.sql support segmentation and root-cause analysis around funnel abandonment.
- These queries represent the business-intelligence layer built on top of the engineered event table.
6. Python notebook layer
- The notebooks/ directory is the primary consumer-facing analysis layer.
- exploratory_data_analysis.ipynb inspects data quality, distributions, and relationships.
- product_metrics_analysis.ipynb operationalizes metric analysis and visualization.
- conversion_prediction_model.ipynb builds the machine learning layer for purchase prediction.
7. Machine learning boundary
- The predictive model is trained in Python using the engineered dataset and product-behavior features.
- The README indicates use of scikit-learn and a Random Forest Classifier to predict conversion for signed-in users.
- This is a separate subsystem from reporting: it consumes analytic features and produces a scoring/prediction outcome.
8. Documentation as architecture map
- The documentation/ folder contains phase-by-phase design and analysis notes.
- DATA_SETUP.md defines environment and BigQuery setup, while phase1 through phase6 document the end-to-end analytical lifecycle.
- In architecture terms, documentation captures the operating model, assumptions, and interpretation of each layer.
9. Execution environment and dependencies
- The repo is designed to run in a local Python + Jupyter environment.
- requirements.txt includes data science and BigQuery integration libraries such as pandas, numpy, matplotlib, seaborn, scikit-learn, google-cloud-bigquery, and db-dtypes.
- BigQuery is the external cloud service that materially affects the architecture; Python notebooks are the local computation and visualization runtime.
10. Output and artifact layer
- screenshots/ stores visual evidence of schemas, charts, funnel results, and model evaluation outputs.
- These are presentation artifacts, not system inputs, but they help document the analytical results and are useful when diagramming the pipeline’s end states.
11. End-to-end data flow
- The practical flow is: CSV source → BigQuery raw table → SQL feature engineering → BigQuery events table → SQL metrics/funnel queries and Python notebooks → insights, recommendations, and conversion model.
- The main architectural boundary is between data warehouse processing and analysis/modeling in notebooks.
12. What to emphasize in a diagram
- Show BigQuery as the central hub.
- Distinguish raw ingestion, event transformation, metric/funnel analysis, and predictive modeling as separate stages.
- Treat notebooks and SQL scripts as distinct consumers of the engineered data, with documentation and screenshots as supporting layers around the core pipeline.
Main entry points: product_metrics_analysis.ipynb Metrics Notebook · screenshots Screenshots · 01_feature_engineering_events.sql Feature SQL · 02_product_metrics_dau_wau_mau.sql Metrics SQL
Components 17
Sources & Storage
Raw and processed data boundaries
- Raw CSV · Primary session-level input file
data/raw/cleaned_speakers_data.csv - Events CSV · Local engineered event-stream artifact
data/processed/events.csv
Warehouse & SQL
BigQuery tables and analytic SQL layers
- raw_sessions · Ingested source table in BigQuery
- events · Transformed event-level analytics table
- Feature SQL · Session-to-event feature engineering
sql/01_feature_engineering_events.sql - Metrics SQL · DAU/WAU/MAU product health metrics
sql/02_product_metrics_dau_wau_mau.sql - Funnel SQL · Conversion funnel analysis and drop-off
sql/03_funnel_analysis_query.sql - Segmentation SQL · Dropped-user segmentation analysis
sql/04_deep_dive_dropped_user_segmentation.sql - Device SQL · Dropped users by device and traffic source
sql/05_deep_dive_dropped_users_by_device_traffic.sql
Analysis & ML
Notebook-based exploration, reporting, and prediction
- EDA Notebook · Exploration of distributions and data quality
notebooks/exploratory_data_analysis.ipynb - Metrics Notebook · Visual product metrics analysis
notebooks/product_metrics_analysis.ipynb - Prediction Notebook · Conversion prediction with Random Forest
notebooks/conversion_prediction_model.ipynb
Docs & Artifacts
Documentation and presentation outputs
- README · Top-level project narrative and usage
README.md - Data Setup · BigQuery and local environment setup
documentation/DATA_SETUP.md - Phase Notes · Phase-by-phase documentation set
documentation/documentation.md - Screenshots · Charts, schemas, and result captures
screenshots - Dependencies · Runtime libraries for notebooks and BigQuery
requirements.txt
Connections 19
- Raw CSV → raw_sessions: load · Upload raw sessions into BigQuery
- raw_sessions → Feature SQL: transform · Use raw warehouse data as SQL input
- Feature SQL → events: build · Materialize engineered event stream
- events → Events CSV: export · Local processed artifact mirrors event table
- events → Metrics SQL: analyze · Compute product health metrics
- events → Funnel SQL: analyze · Run funnel drop-off analysis
- events → Segmentation SQL: analyze · Segment users for root-cause analysis
- events → Device SQL: analyze · Break down dropped users by device/traffic
- events → EDA Notebook: explore · Notebook exploration on engineered data
- events → Metrics Notebook: visualize · Notebook-driven metric analysis
- events → Prediction Notebook: train · Use engineered features for prediction
- Metrics SQL → Metrics Notebook: support · Notebook narrative may reference SQL metrics
- Funnel SQL → Metrics Notebook: inform · Funnel results feed analysis and recommendations
- Segmentation SQL → Screenshots: capture · Save result visuals and summaries
- Device SQL → Screenshots: capture · Save supporting charts and outputs
- Prediction Notebook → Screenshots: evaluate · Store model evaluation artifacts
- README → Data Setup: references · Project overview points to setup guidance
- Data Setup → Dependencies: configures · Describes local runtime dependencies
- Phase Notes → README: documents · Phase notes explain the end-to-end workflow