# anish-anish/nl2flink architecture

> Architecture diagram of the GitHub repository anish-anish/nl2flink, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anish-anish/nl2flink
- Repository: https://github.com/anish-anish/nl2flink

## Overview

MegaStore Pulse turns a business question into a validated Flink pipeline, holds it for engineer approval, then deploys it and exposes live results and operational status. The main path runs from the Streamlit personas through the API, authoring agent, model and validator, into review, deployment, and the streaming stack. The graph also preserves monitoring, answer reading, and the MCP integrations documented by the README. Some subsystem links rely on README or import-index evidence because their implementation excerpts were not included.

Main entry points: [`backend/dependencies.py`](https://github.com/anish-anish/nl2flink/blob/main/backend/dependencies.py) (Shared services), [`backend/agents/mcp_client.py`](https://github.com/anish-anish/nl2flink/blob/main/backend/agents/mcp_client.py) (MCP client), [`backend/agents/pipeline_agent.py`](https://github.com/anish-anish/nl2flink/blob/main/backend/agents/pipeline_agent.py) (Authoring agent), [`backend/services/stack_clients.py`](https://github.com/anish-anish/nl2flink/blob/main/backend/services/stack_clients.py) (Stack clients)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["User experience"]
  node_streamlit["Streamlit app<br/>[streamlit_app.py]"]
  node_business_view["Business view<br/>[business_view.py]"]
  node_engineer_view["Engineer view<br/>[engineer_view.py]"]
  node_ui_components["UI components"]
  node_api_client["HTTP API client<br/>[api_client.py]"]
end

subgraph group_backend["API and workflow"]
  node_api["FastAPI app<br/>[main.py]"]
  node_business_routes["Business routes<br/>[business_routes.py]"]
  node_engineer_routes["Engineer routes<br/>[engineer_routes.py]"]
  node_monitor_routes["Monitor routes<br/>[monitor_routes.py]"]
  node_request_store[("Request state<br/>[database.py]")]
  node_dependencies["Shared services<br/>[dependencies.py]"]
end

subgraph group_pipeline["Pipeline authoring"]
  node_authoring_agent["Authoring agent<br/>[pipeline_agent.py]"]
  node_pipeline_spec["Pipeline DSL<br/>[pipeline_spec.py]"]
  node_model_generator["Model generator<br/>[model_generator.py]"]
  node_validator["Pipeline validator"]
  node_sql_builder["Flink SQL builder"]
end

subgraph group_deployment["Deployment and operations"]
  node_mcp_client["MCP client<br/>[mcp_client.py]"]
  node_deployment_agent["Deployment agent"]
  node_stack_clients["Stack clients<br/>[stack_clients.py]"]
  node_answer_reader["Answer reader<br/>[answer_reader.py]"]
end

subgraph group_integrations["External systems"]
  node_model_server["Fine-tuned model server<br/>[model_server.py]"]
  node_postgres_mcp["Postgres schema MCP"]
  node_cdc_mcp["Debezium / Kafka MCP"]
  node_flink_mcp["Flink MCP<br/>[flink_server.py]"]
  node_github_mcp["GitHub MCP<br/>[github_server.py]"]
  node_slack_mcp["Slack MCP<br/>[slack_server.py]"]
  node_snowflake_mcp["Snowflake MCP"]
  node_operational_db[("PostgreSQL")]
  node_debezium["Debezium"]
  node_kafka["Kafka"]
  node_flink["Apache Flink"]
  node_snowflake[("Snowflake warehouse")]
end

node_business_user(("Business user"))
node_engineer(("Data engineer"))

node_business_user -->|"asks question"| node_streamlit
node_engineer -->|"reviews pipeline"| node_streamlit
node_streamlit -->|"renders"| node_business_view
node_streamlit -->|"renders"| node_engineer_view
node_business_view -->|"uses"| node_ui_components
node_engineer_view -->|"uses"| node_ui_components
node_business_view -->|"requests data"| node_api_client
node_engineer_view -->|"requests data"| node_api_client
node_api_client -->|"HTTP requests"| node_api
node_api -->|"routes requests"| node_business_routes
node_api -->|"routes requests"| node_engineer_routes
node_api -->|"routes requests"| node_monitor_routes
node_business_routes -->|"reads / writes"| node_request_store
node_engineer_routes -->|"reads / writes"| node_request_store
node_business_routes -->|"uses services"| node_dependencies
node_engineer_routes -->|"uses services"| node_dependencies
node_monitor_routes -->|"uses clients"| node_dependencies
node_dependencies -->|"creates"| node_authoring_agent
node_dependencies -->|"creates"| node_deployment_agent
node_dependencies -->|"creates"| node_model_generator
node_dependencies -->|"creates"| node_validator
node_dependencies -->|"creates"| node_sql_builder
node_dependencies -->|"creates"| node_answer_reader
node_dependencies -->|"uses"| node_mcp_client
node_business_routes -->|"generates pipeline"| node_authoring_agent
node_authoring_agent -->|"generates / corrects"| node_model_generator
node_authoring_agent -->|"validates output"| node_validator
node_authoring_agent -->|"grounds schema"| node_mcp_client
node_authoring_agent -->|"parses pipeline"| node_pipeline_spec
node_validator -->|"compilation checks"| node_sql_builder
node_model_generator -->|"requests generation"| node_model_server
node_mcp_client -->|"calls schema tools"| node_postgres_mcp
node_validator -->|"checks live schema"| node_operational_db
node_engineer_routes -->|"deploys on approval"| node_deployment_agent
node_deployment_agent -->|"builds SQL"| node_sql_builder
node_deployment_agent -->|"uses specification"| node_pipeline_spec
node_deployment_agent -->|"dispatches tools"| node_mcp_client
node_mcp_client -->|"commits pipeline"| node_github_mcp
node_mcp_client -->|"configures CDC"| node_cdc_mcp
node_mcp_client -->|"submits job"| node_flink_mcp
node_mcp_client -->|"notifies requester"| node_slack_mcp
node_github_mcp -->|"uses client"| node_stack_clients
node_cdc_mcp -->|"uses clients"| node_stack_clients
node_flink_mcp -->|"uses client"| node_stack_clients
node_snowflake_mcp -->|"uses client"| node_stack_clients
node_stack_clients -->|"reads schema"| node_operational_db
node_stack_clients -->|"configures connector"| node_debezium
node_debezium -->|"streams changes"| node_kafka

click node_streamlit "https://github.com/anish-anish/nl2flink/blob/main/ui/streamlit_app.py"
click node_business_view "https://github.com/anish-anish/nl2flink/blob/main/ui/views/business_view.py"
click node_engineer_view "https://github.com/anish-anish/nl2flink/blob/main/ui/views/engineer_view.py"
click node_ui_components "https://github.com/anish-anish/nl2flink/tree/main/ui/components"
click node_api_client "https://github.com/anish-anish/nl2flink/blob/main/ui/api_client.py"
click node_api "https://github.com/anish-anish/nl2flink/blob/main/backend/main.py"
click node_business_routes "https://github.com/anish-anish/nl2flink/blob/main/backend/routers/business_routes.py"
click node_engineer_routes "https://github.com/anish-anish/nl2flink/blob/main/backend/routers/engineer_routes.py"
click node_monitor_routes "https://github.com/anish-anish/nl2flink/blob/main/backend/routers/monitor_routes.py"
click node_request_store "https://github.com/anish-anish/nl2flink/blob/main/backend/database.py"
click node_dependencies "https://github.com/anish-anish/nl2flink/blob/main/backend/dependencies.py"
click node_authoring_agent "https://github.com/anish-anish/nl2flink/blob/main/backend/agents/pipeline_agent.py"
click node_pipeline_spec "https://github.com/anish-anish/nl2flink/blob/main/backend/pipeline_spec.py"
click node_model_generator "https://github.com/anish-anish/nl2flink/blob/main/backend/services/model_generator.py"
click node_model_server "https://github.com/anish-anish/nl2flink/blob/main/model_server/model_server.py"
click node_validator "https://github.com/anish-anish/nl2flink/blob/main/backend/services/pipeline_validator.py"
click node_sql_builder "https://github.com/anish-anish/nl2flink/blob/main/backend/services/flink_sql_builder.py"
click node_mcp_client "https://github.com/anish-anish/nl2flink/blob/main/backend/agents/mcp_client.py"
click node_deployment_agent "https://github.com/anish-anish/nl2flink/blob/main/backend/agents/deployment_agent.py"
click node_stack_clients "https://github.com/anish-anish/nl2flink/blob/main/backend/services/stack_clients.py"
click node_answer_reader "https://github.com/anish-anish/nl2flink/blob/main/backend/services/answer_reader.py"
click node_postgres_mcp "https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/postgres_schema_server.py"
click node_cdc_mcp "https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/debezium_kafka_server.py"
click node_flink_mcp "https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/flink_server.py"
click node_github_mcp "https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/github_server.py"
click node_slack_mcp "https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/slack_server.py"
click node_snowflake_mcp "https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/snowflake_server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_streamlit,node_business_view,node_engineer_view,node_ui_components,node_api_client,node_business_user toneBlue
class node_api,node_business_routes,node_engineer_routes,node_monitor_routes,node_request_store,node_dependencies toneAmber
class node_authoring_agent,node_pipeline_spec,node_model_generator,node_validator,node_sql_builder toneMint
class node_mcp_client,node_deployment_agent,node_stack_clients,node_answer_reader toneRose
class node_model_server,node_postgres_mcp,node_cdc_mcp,node_flink_mcp,node_github_mcp,node_slack_mcp,node_snowflake_mcp,node_operational_db,node_debezium,node_kafka,node_flink,node_snowflake,node_engineer toneIndigo
```

## Components

### User experience

- [Streamlit app](https://github.com/anish-anish/nl2flink/blob/main/ui/streamlit_app.py): `ui/streamlit_app.py`
- [Business view](https://github.com/anish-anish/nl2flink/blob/main/ui/views/business_view.py): `ui/views/business_view.py`
- [Engineer view](https://github.com/anish-anish/nl2flink/blob/main/ui/views/engineer_view.py): `ui/views/engineer_view.py`
- [UI components](https://github.com/anish-anish/nl2flink/tree/main/ui/components): `ui/components`
- [HTTP API client](https://github.com/anish-anish/nl2flink/blob/main/ui/api_client.py): `ui/api_client.py`

### API and workflow

- [FastAPI app](https://github.com/anish-anish/nl2flink/blob/main/backend/main.py): `backend/main.py`
- [Business routes](https://github.com/anish-anish/nl2flink/blob/main/backend/routers/business_routes.py): `backend/routers/business_routes.py`
- [Engineer routes](https://github.com/anish-anish/nl2flink/blob/main/backend/routers/engineer_routes.py): `backend/routers/engineer_routes.py`
- [Monitor routes](https://github.com/anish-anish/nl2flink/blob/main/backend/routers/monitor_routes.py): `backend/routers/monitor_routes.py`
- [Request state](https://github.com/anish-anish/nl2flink/blob/main/backend/database.py): `backend/database.py`
- [Shared services](https://github.com/anish-anish/nl2flink/blob/main/backend/dependencies.py): `backend/dependencies.py`

### Pipeline authoring

- [Authoring agent](https://github.com/anish-anish/nl2flink/blob/main/backend/agents/pipeline_agent.py): `backend/agents/pipeline_agent.py`
- [Pipeline DSL](https://github.com/anish-anish/nl2flink/blob/main/backend/pipeline_spec.py): `backend/pipeline_spec.py`
- [Model generator](https://github.com/anish-anish/nl2flink/blob/main/backend/services/model_generator.py): `backend/services/model_generator.py`
- [Pipeline validator](https://github.com/anish-anish/nl2flink/blob/main/backend/services/pipeline_validator.py): `backend/services/pipeline_validator.py`
- [Flink SQL builder](https://github.com/anish-anish/nl2flink/blob/main/backend/services/flink_sql_builder.py): `backend/services/flink_sql_builder.py`

### Deployment and operations

- [MCP client](https://github.com/anish-anish/nl2flink/blob/main/backend/agents/mcp_client.py): `backend/agents/mcp_client.py`
- [Deployment agent](https://github.com/anish-anish/nl2flink/blob/main/backend/agents/deployment_agent.py): `backend/agents/deployment_agent.py`
- [Stack clients](https://github.com/anish-anish/nl2flink/blob/main/backend/services/stack_clients.py): `backend/services/stack_clients.py`
- [Answer reader](https://github.com/anish-anish/nl2flink/blob/main/backend/services/answer_reader.py): `backend/services/answer_reader.py`

### External systems

- [Fine-tuned model server](https://github.com/anish-anish/nl2flink/blob/main/model_server/model_server.py): `model_server/model_server.py`
- [Postgres schema MCP](https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/postgres_schema_server.py): `mcp_servers/postgres_schema_server.py`
- [Debezium / Kafka MCP](https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/debezium_kafka_server.py): `mcp_servers/debezium_kafka_server.py`
- [Flink MCP](https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/flink_server.py): `mcp_servers/flink_server.py`
- [GitHub MCP](https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/github_server.py): `mcp_servers/github_server.py`
- [Slack MCP](https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/slack_server.py): `mcp_servers/slack_server.py`
- [Snowflake MCP](https://github.com/anish-anish/nl2flink/blob/main/mcp_servers/snowflake_server.py): `mcp_servers/snowflake_server.py`
- **PostgreSQL**
- **Debezium**
- **Kafka**
- **Apache Flink**
- **Snowflake warehouse**

### Other components

- **Business user**
- **Data engineer**

## Connections

- Business user → Streamlit app: asks question (evidence: `README.md`)
- Data engineer → Streamlit app: reviews pipeline (evidence: `README.md`)
- Streamlit app → Business view: renders (evidence: `ui/streamlit_app.py`)
- Streamlit app → Engineer view: renders (evidence: `ui/streamlit_app.py`)
- Business view → UI components: uses (evidence: `ui/views/business_view.py`)
- Engineer view → UI components: uses (evidence: `ui/views/engineer_view.py`)
- Business view → HTTP API client: requests data (evidence: `ui/views/business_view.py`)
- Engineer view → HTTP API client: requests data (evidence: `ui/views/engineer_view.py`)
- HTTP API client → FastAPI app: HTTP requests (evidence: `README.md`)
- FastAPI app → Business routes: routes requests (evidence: `backend/main.py`)
- FastAPI app → Engineer routes: routes requests (evidence: `backend/main.py`)
- FastAPI app → Monitor routes: routes requests (evidence: `backend/main.py`)
- Business routes → Request state: reads / writes (evidence: `backend/routers/business_routes.py`)
- Engineer routes → Request state: reads / writes (evidence: `backend/routers/engineer_routes.py`)
- Business routes → Shared services: uses services (evidence: `backend/routers/business_routes.py`)
- Engineer routes → Shared services: uses services (evidence: `backend/routers/engineer_routes.py`)
- Monitor routes → Shared services: uses clients (evidence: `backend/routers/monitor_routes.py`)
- Shared services → Authoring agent: creates (evidence: `backend/dependencies.py`)
- Shared services → Deployment agent: creates (evidence: `backend/dependencies.py`)
- Shared services → Model generator: creates (evidence: `backend/dependencies.py`)
- Shared services → Pipeline validator: creates (evidence: `backend/dependencies.py`)
- Shared services → Flink SQL builder: creates (evidence: `backend/dependencies.py`)
- Shared services → Answer reader: creates (evidence: `backend/dependencies.py`)
- Shared services → MCP client: uses (evidence: `backend/dependencies.py`)
- Business routes → Authoring agent: generates pipeline (evidence: `backend/routers/business_routes.py`)
- Authoring agent → Model generator: generates / corrects (evidence: `backend/agents/pipeline_agent.py`)
- Authoring agent → Pipeline validator: validates output (evidence: `backend/agents/pipeline_agent.py`)
- Authoring agent → MCP client: grounds schema (evidence: `backend/agents/pipeline_agent.py`)
- Authoring agent → Pipeline DSL: parses pipeline
- Pipeline validator → Flink SQL builder: compilation checks (evidence: `backend/services/pipeline_validator.py`)
- Model generator → Fine-tuned model server: requests generation (evidence: `README.md`)
- MCP client → Postgres schema MCP: calls schema tools (evidence: `README.md`)
- Pipeline validator → PostgreSQL: checks live schema (evidence: `backend/services/pipeline_validator.py`)
- Engineer routes → Deployment agent: deploys on approval (evidence: `backend/routers/engineer_routes.py`)
- Deployment agent → Flink SQL builder: builds SQL (evidence: `backend/agents/deployment_agent.py`)
- Deployment agent → Pipeline DSL: uses specification (evidence: `backend/agents/deployment_agent.py`)
- Deployment agent → MCP client: dispatches tools (evidence: `README.md`)
- MCP client → GitHub MCP: commits pipeline (evidence: `README.md`)
- MCP client → Debezium / Kafka MCP: configures CDC (evidence: `README.md`)
- MCP client → Flink MCP: submits job (evidence: `README.md`)
- MCP client → Slack MCP: notifies requester (evidence: `README.md`)
- GitHub MCP → Stack clients: uses client
- Debezium / Kafka MCP → Stack clients: uses clients (evidence: `mcp_servers/debezium_kafka_server.py`)
- Flink MCP → Stack clients: uses client (evidence: `mcp_servers/flink_server.py`)
- Snowflake MCP → Stack clients: uses client (evidence: `mcp_servers/snowflake_server.py`)
- Stack clients → PostgreSQL: reads schema (evidence: `backend/services/stack_clients.py`)
- Stack clients → Debezium: configures connector (evidence: `backend/services/stack_clients.py`)
- Debezium → Kafka: streams changes
