bulktornado/cbse-rdms-project

1. Project intent

This repository is a Python data-preparation and database project for CBSE school/curriculum information. It ingests curriculum PDFs and raw school datasets, converts them into structured records, and stores them in local data files and a relational database.

2. Primary pipeline

The main flow is: reference documents and raw text data in docs/ and data/raw_datasets/ → parsing/cleanup scripts in src/ → tabular/serialized outputs in data/ and backup/ → database schema and connection layer for query/use by the application.

3. Source data boundary

docs/ contains the curriculum PDFs and the project report; these are the human/reference inputs. data/raw_datasets/ holds extracted or intermediate source material, including reference notes about filtered courses and pickled data snapshots. These are not runtime logic, but they define the input domain.

4. Parsing and normalization

src/parse_affiliated_school_data.py and src/parse_courses.py are the key transformation scripts. They appear to convert raw school/course sources into structured datasets, likely handling filtering, normalization, and mapping into consistent records for downstream storage.

5. Setup and configuration boundary

src/setup.py and src/set_configuration.py suggest a one-time or repeatable initialization phase that prepares paths, configuration, and probably database/data generation. This is separate from the parsing logic and from the runtime module layer.

6. Database schema

schemas/database_schema.sql defines the relational structure for the project. The tracked dataset names strongly indicate at least two main entities: affiliated schools and courses, with the schema serving as the contract between parsed data and the database.

7. Shared runtime modules

modules/connection_module.py centralizes database connectivity, while modules/functions.py contains reusable helpers used across scripts. This is the main internal utility layer that separates low-level I/O/DB handling from parsing/setup logic.

8. Entry point

main.py is the top-level entry point for running the project. It likely coordinates setup, data loading, and/or query operations by invoking the modules and scripts above rather than containing the business logic itself.

9. Derived data and snapshots

data/ and backup/ contain generated artifacts such as .dat and .txt files for affiliated_schools and courses. These look like persistence snapshots and interchange formats, useful for reruns, debugging, or restoring state without re-parsing the original sources.

10. One-off/manual correction path

src/forbidden_scripts/manually_update_affiliated_schools_from_tabular_data.py stands out as an exceptional maintenance script. It likely exists for manual data correction or migration and should be treated as outside the normal pipeline.

11. Execution environment

The repository is a Python project managed with pyproject.toml, requirements.txt, uv.lock, and .python-version. The architecture is local/offline and file-driven; there are no obvious external network services or web framework boundaries in the repo.

12. What to draw in an architecture diagram

A clear diagram would show:

- Reference inputs: PDFs/text/raw datasets

- Transform layer: parsing and setup scripts

- Shared services: connection/utilities

- Storage layer: schema + generated data files/backups

- Entry point: main.py

The most important boundary is between raw source material and normalized structured data.