How naman14113114/practice_paper_generator works
1. Overall shape
- This is a Python, offline-first question-bank pipeline.
- The repository is split into two main concerns: ingestion of questions from aptitude sites and local storage / future paper generation from normalized JSON banks.
2. Primary entry point for ingestion
- scrape.py is the current operational entry point.
- It drives the scraping workflow end-to-end: choose a scraper, fetch questions from a source URL, normalize them, and merge them into the local bank.
3. Scraper subsystem
- scrapers/base.py defines the common scraper interface.
- scrapers/indiabix.py is the concrete implementation currently in use.
- scrapers/examveda.py is a placeholder for another source, showing the intended extension pattern.
- This boundary isolates website-specific parsing from the rest of the system.
4. Question normalization contract
- Scrapers emit a shared question shape with fields like question, options, answer, source, and topic_url.
- That common schema is the key integration point: different websites can feed the same storage layer without downstream code caring where a question came from.
5. Local persistence layer
- storage/question_bank.py manages the JSON-backed question repository.
- The system stores questions by topic under data/banks/*.json, such as probability.json.
- This is the durable system of record; there is no database or remote backend in the current design.
6. Merge and deduplication boundary
- The bank layer is responsible for merging new scraped questions into an existing topic bank and removing duplicates.
- This keeps ingestion idempotent and prevents repeated scraping from polluting the bank.
7. Data organization
- Each topic is isolated into its own JSON file, which makes the storage model simple and file-based.
- This means the architecture is topic-centric rather than source-centric: one bank can accumulate questions from multiple sites over time.
8. Generation subsystem
- generator/generate.py and generator/paper.py represent the practice-paper assembly layer.
- The README describes this as planned/future-facing, but the code layout already reserves a separate boundary for it.
- Its job is to consume banks, select questions, randomize them, and produce a paper artifact.
9. Planned paper-generation flow
- The intended downstream flow is: question banks → selection/filtering → randomization → paper output.
- The README calls out future features like multi-topic papers, difficulty balancing, answer keys, and printable templates.
10. Extensibility model
- New source support is added by implementing the base scraper interface and wiring it into the scraping entry point.
- New content is added by topic through additional JSON files, without changing the storage model.
- This makes the architecture modular at the source boundary and lightweight at the persistence boundary.
11. Operational characteristics
- The project is designed to be local, deterministic, and easy to maintain.
- It does not depend on a web service, queue, or database; the filesystem is the main integration surface.
- requirements.txt suggests a small Python dependency set, consistent with a simple scraping-and-storage toolchain.
12. How to draw the architecture
- Draw three layers: website scrapers on the left, question bank storage in the middle, and paper generation on the right.
- Show scrape.py as the current orchestrator feeding normalized questions into storage/question_bank.py.
- Show data/banks/*.json as the persistent output, and generator/* as a separate consumer that will read those banks to build practice papers.
Main entry points: question_bank.py Question bank · scrape.py scrape.py · base.py Base scraper · generate.py Paper builder
Components 10
Ingestion
Scraping and normalization of questions from external sites.
- scrape.py · Current end-to-end scraping entry point.
scrape.py - Base scraper · Common scraper contract for source-specific parsers.
scrapers/base.py - Indiabix · Concrete scraper currently used for question ingestion.
scrapers/indiabix.py - ExamVeda · Placeholder for adding another question source.
scrapers/examveda.py - Normalized question · Shared question shape passed from scrapers into storage.
Storage
Local topic banks and merge/dedup persistence.
- Question bank · JSON-backed topic bank with merge and dedup logic.
storage/question_bank.py - Probability bank · Representative persisted topic bank file.
data/banks/probability.json
Generation
Paper assembly from stored question banks.
- Paper builder · Selects and prepares questions for paper output.
generator/generate.py - Paper model · Paper artifact structure and rendering support.
generator/paper.py - generate.py · Root-level wrapper for generation workflows.
generate.py
Connections 12
- scrape.py → Base scraper: uses · Loads the shared scraper interface to drive ingestion.
- scrape.py → Indiabix: runs · Current operational path for fetching questions.
- scrape.py → Question bank: merges into · Writes normalized questions into the local bank.
- Base scraper → Indiabix: implements · Concrete scraper follows the common contract.
- Base scraper → ExamVeda: extends · Placeholder source also fits the same interface.
- Indiabix → Normalized question: emits · Produces the shared question schema.
- ExamVeda → Normalized question: emits · Would produce the same normalized schema.
- Normalized question → Question bank: ingested by · Storage layer accepts the shared question shape.
- Question bank → Probability bank: persists · Topic bank is stored as JSON on disk.
- Paper builder → Question bank: reads · Consumes banks as the source corpus.
- Paper builder → Paper model: builds · Assembles a paper artifact from selected questions.
- generate.py → Paper builder: delegates · Top-level generation entrypoint forwards to the generator package.