How ma-da/peersai works
Project type and purpose
• This is a multi-stage data pipeline plus AI training system with a client-side search frontend.
• Purpose:
• webscraper crawls and downloads raw “peers-family” corpus material.
• preprocessing ingests that raw corpus, filters and indexes it.
• IPFS processes, cleans, and uploads the indexed corpus to a decentralized storage layer.
• train_ai uses the dataset on IPFS to perform LoRA training and inference experiments.
• farsight is a browser-based single-page app that queries the IPFS-backed index for TF-IDF + BGE re-ranked search.
File structure
• Top-level directories map to major subsystems:
– webscraper
– preprocessing
– IPFS
– train_ai
– farsight
• Each subsystem contains scripts, configs, notebooks, and utility modules.
• There are shell scripts (crawl, convert, logs) under webscraper, Python modules for filtering/indexing under preprocessing, IPFS upload and embedding tests under IPFS, Jupyter notebooks and train scripts under train_ai, and HTML/JS under farsight.
README for additional insights
• The README clearly lists four projects and their roles.
raw crawl → filter/index → decentralized store → model training & frontend search.