How rektifyai/adv-attack-analysis works
Project Type and Purpose
• This is an AI/ML-driven data-analysis and prescriptive-analytics platform for Web3 attack-vulnerability detection.
• It uses data ingestion (scraping, APIs), storage (datasets, Beneath API), processing (EDA, feature engineering), modeling (ANNs), serving (prediction scripts, microservices), and a minimal React front-end.
Main Architectural Components
A. Data Sources (External)
– Blockchain nodes / on-chain data
– Twitter API / social feeds
– Web-scraper targets (relevant Web3 news sites)
– CSV/Excel files (historic datasets in dataset/)
B. Ingestion & ETL Layer
– web-scraper module (web-scraper/)
– Twitter-API notebooks and scripts (Twitter-API/)
– CSV-to-DB conversion notebooks (dataset/CSV-to-HTML-DB.ipynb)
– Beneath API connector (Beneath-API/beneath-react.js, notebooks)
C. Data Storage
– Local files (CSV, JSON in dataset/, Beneath-API/SEAL-dataset-keyed.json)
– Beneath Data Warehouse (REAL API dataset)
D. Processing & Analysis
– Exploratory Data Analysis notebooks (*.ipynb under root & subfolders)
– Vulnerability detection research docs (sentiment-analysis/, ANN-application/)
E. ML Model Training & Serving
– Training pipeline (Beneath-API/train_model.ipynb, make_predictions.py)
– ANN application notebooks (ANN-application/GNN-app-revised.ipynb)
F. API & Microservices
– Ethereum-blocks microservice (Beneath-API/ethereum-blocks/ with Dockerfile, kube.yaml, main.py)
– Beneath-API wrapper (serving processed data/predictions)
G. Presentation / Alerting
– Minimal React front-end (beneath-react.js)
– Dashboards/visualizations on Rektify AI application (not in this repo)
Relationships & Data Flows
• ETL outputs → Data Storage (local files & Beneath warehouse)
• Data Storage → Processing & Analysis notebooks → Feature store
• Feature store → ML Model Training → Trained ANN Models
• Trained Models → Prediction Service (make_predictions.py)
• Prediction Service → API & Microservices → Front-end alerts
• Kubernetes/Docker layer encompasses Ethereum-blocks microservice
Architectural Patterns & Design Principles
• Modular monorepo separated by function (ingestion, storage, ML, service)
• ETL pipeline feeding a supervised learning loop
• Microservice for on-chain data extraction (dockerized & K8s deployed)
• Notebooks for research/EDA → production scripts for training & inference
• Clear separation of concerns: Data Ingestion vs. Data Storage vs. Model Serving
Essentials
– Clear component names (Ingestion, Storage, Processing, ML, API, UI)
– Tech stack callouts (Python, Jupyter, Docker, Kubernetes, React)
– External dependencies (Twitter API, Beneath API, SWC Registry)