# kamwithk/gitstarred architecture

> Architecture diagram of the GitHub repository kamwithk/gitstarred, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kamwithk/gitstarred
- Repository: https://github.com/kamwithk/gitstarred

## Overview

GitStarred’s implemented workflow collects GitHub repository data, stores it in PostgreSQL, engineers features with Spark, and trains/evaluates an H2O AutoML model. The README also describes neural-network and website capabilities, but the supplied tree shows no implementation paths for those, so they are not represented as runtime components. Edges are limited to relationships visible in excerpts or documented in the README.

Main entry points: [`src/DataCollection/DateBreakdown.py`](https://github.com/kamwithk/gitstarred/blob/master/src/DataCollection/DateBreakdown.py) (Date breakdown), [`src/CollectData.py`](https://github.com/kamwithk/gitstarred/blob/master/src/CollectData.py) (Collection runner), [`src/ProcessTrain.py`](https://github.com/kamwithk/gitstarred/blob/master/src/ProcessTrain.py) (Process and train), [`src/Pipeline/Processor.py`](https://github.com/kamwithk/gitstarred/blob/master/src/Pipeline/Processor.py) (Spark processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data collection"]
  node_collector["Collection runner<br/>[CollectData.py]"]
  node_periods["Date breakdown<br/>[DateBreakdown.py]"]
  node_graphql["GraphQL client<br/>[GraphQL.py]"]
  node_query["Repository query<br/>[Collection.graphql]"]
  node_datequery["Date query"]
  node_parser["Repository parser<br/>[Parser.py]"]
end

subgraph group_engineering["Feature engineering"]
  node_training_runner["Process and train<br/>[ProcessTrain.py]"]
  node_processor["Spark processor<br/>[Processor.py]"]
  node_components["Feature transforms<br/>[Components.py]"]
  node_parquet["Train/test datasets"]
end

subgraph group_modelling["Model training"]
  node_automl["H2O AutoML<br/>[H2OSparkAutoML.py]"]
  node_model_results["Model results"]
end

node_analyst(("Analyst"))
node_github["GitHub API"]
node_postgres[("PostgreSQL")]
node_spark["Apache Spark"]

node_analyst -->|"starts collection"| node_collector
node_collector -->|"collects periods"| node_periods
node_collector -->|"creates parser"| node_parser
node_collector -->|"loads query"| node_query
node_periods -->|"loads date query"| node_datequery
node_periods -->|"runs searches"| node_graphql
node_periods -->|"pages through"| node_query
node_graphql -->|"sends requests"| node_github
node_periods -->|"processes results"| node_parser
node_parser -->|"upserts repositories"| node_postgres
node_periods -->|"tracks progress"| node_postgres
node_analyst -->|"starts training"| node_training_runner
node_training_runner -->|"processes data"| node_processor
node_processor -->|"reads repositories"| node_postgres
node_processor -->|"runs transformations"| node_spark
node_training_runner -->|"configures stages"| node_components
node_processor -->|"saves datasets"| node_parquet
node_training_runner -->|"trains models"| node_automl
node_automl -->|"fits AutoML"| node_spark
node_automl -->|"returns results"| node_model_results

click node_collector "https://github.com/kamwithk/gitstarred/blob/master/src/CollectData.py"
click node_periods "https://github.com/kamwithk/gitstarred/blob/master/src/DataCollection/DateBreakdown.py"
click node_graphql "https://github.com/kamwithk/gitstarred/blob/master/src/DataCollection/GraphQL.py"
click node_query "https://github.com/kamwithk/gitstarred/blob/master/src/Queries/Collection.graphql"
click node_datequery "https://github.com/kamwithk/gitstarred/blob/master/src/Queries/InnerDateTester.graphql"
click node_parser "https://github.com/kamwithk/gitstarred/blob/master/src/DataCollection/Parser.py"
click node_training_runner "https://github.com/kamwithk/gitstarred/blob/master/src/ProcessTrain.py"
click node_processor "https://github.com/kamwithk/gitstarred/blob/master/src/Pipeline/Processor.py"
click node_components "https://github.com/kamwithk/gitstarred/blob/master/src/Pipeline/Components.py"
click node_automl "https://github.com/kamwithk/gitstarred/blob/master/src/Modelling/H2OSparkAutoML.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_collector,node_periods,node_graphql,node_query,node_datequery,node_parser toneBlue
class node_training_runner,node_processor,node_components,node_parquet,node_postgres toneAmber
class node_automl,node_model_results,node_github toneMint
class node_analyst,node_spark toneIndigo
```

## Components

### Data collection

- [Collection runner](https://github.com/kamwithk/gitstarred/blob/master/src/CollectData.py): `src/CollectData.py`
- [Date breakdown](https://github.com/kamwithk/gitstarred/blob/master/src/DataCollection/DateBreakdown.py): `src/DataCollection/DateBreakdown.py`
- [GraphQL client](https://github.com/kamwithk/gitstarred/blob/master/src/DataCollection/GraphQL.py): `src/DataCollection/GraphQL.py`
- [Repository query](https://github.com/kamwithk/gitstarred/blob/master/src/Queries/Collection.graphql): `src/Queries/Collection.graphql`
- [Date query](https://github.com/kamwithk/gitstarred/blob/master/src/Queries/InnerDateTester.graphql): `src/Queries/InnerDateTester.graphql`
- [Repository parser](https://github.com/kamwithk/gitstarred/blob/master/src/DataCollection/Parser.py): `src/DataCollection/Parser.py`

### Feature engineering

- [Process and train](https://github.com/kamwithk/gitstarred/blob/master/src/ProcessTrain.py): `src/ProcessTrain.py`
- [Spark processor](https://github.com/kamwithk/gitstarred/blob/master/src/Pipeline/Processor.py): `src/Pipeline/Processor.py`
- [Feature transforms](https://github.com/kamwithk/gitstarred/blob/master/src/Pipeline/Components.py): `src/Pipeline/Components.py`
- **Train/test datasets**

### Model training

- [H2O AutoML](https://github.com/kamwithk/gitstarred/blob/master/src/Modelling/H2OSparkAutoML.py): `src/Modelling/H2OSparkAutoML.py`
- **Model results**

### Other components

- **Analyst**
- **GitHub API**
- **PostgreSQL**
- **Apache Spark**

## Connections

- Analyst → Collection runner: starts collection (evidence: `src/CollectData.py`)
- Collection runner → Date breakdown: collects periods (evidence: `src/CollectData.py`)
- Collection runner → Repository parser: creates parser (evidence: `src/CollectData.py`)
- Collection runner → Repository query: loads query (evidence: `src/CollectData.py`)
- Date breakdown → Date query: loads date query (evidence: `src/DataCollection/DateBreakdown.py`)
- Date breakdown → GraphQL client: runs searches (evidence: `src/DataCollection/DateBreakdown.py`)
- Date breakdown → Repository query: pages through
- GraphQL client → GitHub API: sends requests (evidence: `src/DataCollection/GraphQL.py`)
- Date breakdown → Repository parser: processes results (evidence: `src/DataCollection/DateBreakdown.py`)
- Repository parser → PostgreSQL: upserts repositories (evidence: `src/DataCollection/Parser.py`)
- Date breakdown → PostgreSQL: tracks progress (evidence: `src/DataCollection/DateBreakdown.py`)
- Analyst → Process and train: starts training (evidence: `src/ProcessTrain.py`)
- Process and train → Spark processor: processes data (evidence: `src/ProcessTrain.py`)
- Spark processor → PostgreSQL: reads repositories (evidence: `src/Pipeline/Processor.py`)
- Spark processor → Apache Spark: runs transformations (evidence: `README.md`)
- Process and train → Feature transforms: configures stages (evidence: `src/ProcessTrain.py`)
- Spark processor → Train/test datasets: saves datasets (evidence: `src/Pipeline/Processor.py`)
- Process and train → H2O AutoML: trains models (evidence: `src/ProcessTrain.py`)
- H2O AutoML → Apache Spark: fits AutoML (evidence: `README.md`)
- H2O AutoML → Model results: returns results (evidence: `src/Modelling/H2OSparkAutoML.py`)