How anish-anish/nl2flink works
MegaStore Pulse turns a business question into a validated Flink pipeline, holds it for engineer approval, then deploys it and exposes live results and operational status. The main path runs from the Streamlit personas through the API, authoring agent, model and validator, into review, deployment, and the streaming stack. The graph also preserves monitoring, answer reading, and the MCP integrations documented by the README. Some subsystem links rely on README or import-index evidence because their implementation excerpts were not included.
Main entry points: dependencies.py Shared services · mcp_client.py MCP client · pipeline_agent.py Authoring agent · stack_clients.py Stack clients
Components 34
User experience
- Streamlit app
ui/streamlit_app.py - Business view
ui/views/business_view.py - Engineer view
ui/views/engineer_view.py - UI components
ui/components - HTTP API client
ui/api_client.py
API and workflow
- FastAPI app
backend/main.py - Business routes
backend/routers/business_routes.py - Engineer routes
backend/routers/engineer_routes.py - Monitor routes
backend/routers/monitor_routes.py - Request state
backend/database.py - Shared services
backend/dependencies.py
Pipeline authoring
- Authoring agent
backend/agents/pipeline_agent.py - Pipeline DSL
backend/pipeline_spec.py - Model generator
backend/services/model_generator.py - Pipeline validator
backend/services/pipeline_validator.py - Flink SQL builder
backend/services/flink_sql_builder.py
Deployment and operations
- MCP client
backend/agents/mcp_client.py - Deployment agent
backend/agents/deployment_agent.py - Stack clients
backend/services/stack_clients.py - Answer reader
backend/services/answer_reader.py
External systems
- Fine-tuned model server
model_server/model_server.py - Postgres schema MCP
mcp_servers/postgres_schema_server.py - Debezium / Kafka MCP
mcp_servers/debezium_kafka_server.py - Flink MCP
mcp_servers/flink_server.py - GitHub MCP
mcp_servers/github_server.py - Slack MCP
mcp_servers/slack_server.py - Snowflake MCP
mcp_servers/snowflake_server.py - PostgreSQL
- Debezium
- Kafka
- Apache Flink
- Snowflake warehouse
Other components
- Business user
- Data engineer
Connections 48
- Business user → Streamlit app: asks question
README.md - Data engineer → Streamlit app: reviews pipeline
README.md - Streamlit app → Business view: renders
ui/streamlit_app.py - Streamlit app → Engineer view: renders
ui/streamlit_app.py - Business view → UI components: uses
ui/views/business_view.py - Engineer view → UI components: uses
ui/views/engineer_view.py - Business view → HTTP API client: requests data
ui/views/business_view.py - Engineer view → HTTP API client: requests data
ui/views/engineer_view.py - HTTP API client → FastAPI app: HTTP requests
README.md - FastAPI app → Business routes: routes requests
backend/main.py - FastAPI app → Engineer routes: routes requests
backend/main.py - FastAPI app → Monitor routes: routes requests
backend/main.py - Business routes → Request state: reads / writes
backend/routers/business_routes.py - Engineer routes → Request state: reads / writes
backend/routers/engineer_routes.py - Business routes → Shared services: uses services
backend/routers/business_routes.py - Engineer routes → Shared services: uses services
backend/routers/engineer_routes.py - Monitor routes → Shared services: uses clients
backend/routers/monitor_routes.py - Shared services → Authoring agent: creates
backend/dependencies.py - Shared services → Deployment agent: creates
backend/dependencies.py - Shared services → Model generator: creates
backend/dependencies.py - Shared services → Pipeline validator: creates
backend/dependencies.py - Shared services → Flink SQL builder: creates
backend/dependencies.py - Shared services → Answer reader: creates
backend/dependencies.py - Shared services → MCP client: uses
backend/dependencies.py - Business routes → Authoring agent: generates pipeline
backend/routers/business_routes.py - Authoring agent → Model generator: generates / corrects
backend/agents/pipeline_agent.py - Authoring agent → Pipeline validator: validates output
backend/agents/pipeline_agent.py - Authoring agent → MCP client: grounds schema
backend/agents/pipeline_agent.py - Authoring agent → Pipeline DSL: parses pipeline
- Pipeline validator → Flink SQL builder: compilation checks
backend/services/pipeline_validator.py - Model generator → Fine-tuned model server: requests generation
README.md - MCP client → Postgres schema MCP: calls schema tools
README.md - Pipeline validator → PostgreSQL: checks live schema
backend/services/pipeline_validator.py - Engineer routes → Deployment agent: deploys on approval
backend/routers/engineer_routes.py - Deployment agent → Flink SQL builder: builds SQL
backend/agents/deployment_agent.py - Deployment agent → Pipeline DSL: uses specification
backend/agents/deployment_agent.py - Deployment agent → MCP client: dispatches tools
README.md - MCP client → GitHub MCP: commits pipeline
README.md - MCP client → Debezium / Kafka MCP: configures CDC
README.md - MCP client → Flink MCP: submits job
README.md - MCP client → Slack MCP: notifies requester
README.md - GitHub MCP → Stack clients: uses client
- Debezium / Kafka MCP → Stack clients: uses clients
mcp_servers/debezium_kafka_server.py - Flink MCP → Stack clients: uses client
mcp_servers/flink_server.py - Snowflake MCP → Stack clients: uses client
mcp_servers/snowflake_server.py - Stack clients → PostgreSQL: reads schema
backend/services/stack_clients.py - Stack clients → Debezium: configures connector
backend/services/stack_clients.py - Debezium → Kafka: streams changes