How sinaptik-ai/pandas-ai works
1. What this repo is
PandasAI is a Python library for natural-language-to-data analysis. The primary runtime is a library API, not a server: users import pandasai, configure an LLM, load data into PandasAI dataframe abstractions, and ask questions through chat().
2. Top-level architecture
The repository is split into a core library (pandasai/) plus optional extension packages under extensions/. The core handles orchestration, prompt generation, code/SQL synthesis, execution, and response shaping. Extensions add pluggable integrations for LLMs, databases, vector stores, and sandboxing.
3. User-facing entry points
The main entry points are pandasai.DataFrame, pandasai.chat(...), pandasai.config, and the CLI under pandasai/cli/main.py. The library supports both single-dataframe and multi-dataframe conversations, and also has a “smart datalake” abstraction for broader data access.
4. Core request flow
A typical request starts with a user query and one or more data objects. The agent layer (pandasai/agent) coordinates state, wraps the query, and routes it into prompt construction. The prompt layer uses schema/data context, output-type expectations, and SQL-specific templates when needed.
5. LLM boundary
LLM integration is intentionally abstracted through pandasai/llm/base.py. The repo itself includes a fake LLM for tests, while real provider adapters live in extensions such as pandasai-openai and pandasai-litellm. This keeps the core independent of any single model vendor.
6. Code generation and validation
Generated output is not executed blindly. The pandasai/core/code_generation subsystem cleans and validates model output before execution. This is the main safety boundary between untrusted LLM text and runtime execution.
7. Execution boundary
The execution path is handled by pandasai/core/code_execution and can run either locally or inside a sandbox. The core sandbox abstraction is complemented by the Docker-based sandbox extension in extensions/sandbox/docker, which isolates generated code in a separate containerized environment.
8. Data access layer
Data ingestion is split by source type. pandasai/data_loader includes local loading, SQL loading, DuckDB connection management, view loading, and semantic-layer schema support. The query builders in pandasai/query_builders adapt those sources into executable local queries, SQL queries, or view queries, with pagination and SQL parsing/transformation support where needed.
9. Response shaping
After execution, results are converted into typed response objects under pandasai/core/response such as dataframe, number, string, chart, or error. This gives the library a stable output contract regardless of whether the model produced tabular data, text, or a visualization.
10. Safety and support subsystems
The pandasai/helpers package provides cross-cutting concerns such as logging, session/memory, file handling, telemetry, JSON encoding, environment handling, and SQL sanitization. These utilities are important architectural boundaries because they support reproducibility, observability, and risk reduction around generated queries/code.
11. Extensions ecosystem
The extensions/ tree contains independently packaged integrations. There are connectors for SQL sources, Yahoo Finance, and enterprise-grade databases like BigQuery, Databricks, Oracle, and Snowflake; vector stores such as ChromaDB, LanceDB, Milvus, Pinecone, and Qdrant; and LLM adapters for OpenAI and LiteLLM. These packages extend the core without embedding all dependencies into the base install.
12. Enterprise and skills
The pandasai/ee area and extensions/ee packages indicate a separate enterprise feature surface, including a skills manager and enterprise connectors/vector stores. This suggests a product boundary between the open-source core and paid or restricted functionality.
13. Documentation and compatibility layers
The docs/v2 and docs/v3 directories show two major API generations, including migration and compatibility guidance. That implies the architecture must preserve backward compatibility while evolving the semantic-layer, agent, and chat APIs.
14. Validation and release workflow
The repository has separate core and extension CI workflows, plus packaging metadata for multiple distributable packages. Architecturally, this reinforces that the system is a modular Python ecosystem: the core library is tested and released alongside optional integration packages with their own test suites.
Main entry points: data_loader Data loaders · base.py Agent · prompts Prompts · __init__.py PandasAI API
Components 26
Core library
Primary runtime for natural-language data analysis.
- PandasAI API · Top-level package surface for importing the library and common entry points.
pandasai/__init__.py - Config · Library configuration and runtime options.
pandasai/config.py - CLI · Command-line entry point for running the library from the shell.
pandasai/cli/main.py - Agent · Coordinates user queries, state, and the analysis flow.
pandasai/agent/base.py - State · Session and turn state used by the agent during a chat.
pandasai/agent/state.py - Prompts · Templates and prompt builders for system, SQL, and error correction prompts.
pandasai/core/prompts - Code gen · Cleans and validates model output before execution.
pandasai/core/code_generation - Executor · Runs generated code either locally or through a sandbox boundary.
pandasai/core/code_execution - Responses · Typed response objects for dataframes, numbers, strings, charts, and errors.
pandasai/core/response - Dataframe · User-facing dataframe abstractions, including virtual and smart dataframe behavior.
pandasai/dataframe - Datalake · Broader data access abstraction for conversations over multiple data sources.
pandasai/smart_datalake - Data loaders · Loads local files, SQL sources, views, DuckDB connections, and semantic schemas.
pandasai/data_loader - Query builders · Builds local, SQL, and view queries and handles pagination and SQL transforms.
pandasai/query_builders - LLM interface · Abstract LLM contract that isolates the core from provider-specific implementations.
pandasai/llm/base.py - Helpers · Logging, memory, sessions, telemetry, sanitization, file handling, and encoding support.
pandasai/helpers - Sandbox API · Core sandbox abstraction for isolating generated code.
pandasai/sandbox/sandbox.py - Skills · Enterprise skills management and related restricted surface area.
pandasai/ee/skills/manager.py
Extensions
Optional packaged integrations and sandboxes.
- OpenAI · Provider adapter package for OpenAI and Azure OpenAI.
extensions/llms/openai/pandasai_openai - LiteLLM · Provider adapter package for LiteLLM-backed model access.
extensions/llms/litellm/pandasai_litellm - SQL connector · Optional SQL source integration packaged separately from core.
extensions/connectors/sql/pandasai_sql - YFinance · Optional market-data connector packaged as an extension.
extensions/connectors/yfinance/pandasai_yfinance - Docker sandbox · Containerized sandbox implementation for isolated execution.
extensions/sandbox/docker/pandasai_docker/docker_sandbox.py - EE connectors · Enterprise database connectors such as BigQuery, Databricks, Oracle, and Snowflake.
extensions/ee/connectors - EE vectorstores · Enterprise vector store integrations such as ChromaDB, LanceDB, Milvus, Pinecone, and Qdrant.
extensions/ee/vectorstores
Connections 28
- PandasAI API → Config: configures · Library entry uses runtime configuration.
- PandasAI API → Agent: chat flow · Public chat and dataframe APIs route into the agent orchestration layer.
- PandasAI API → Dataframe: wraps data · Top-level API exposes dataframe abstractions to users.
- PandasAI API → Datalake: multi-data · Top-level API also exposes the datalake abstraction.
- CLI → Agent: invokes · CLI delegates work to the same core orchestration path.
- Agent → State: tracks · Agent maintains conversational and execution state.
- Agent → Prompts: builds prompts · Agent prepares prompt context and output expectations.
- Prompts → LLM interface: requests model · Prompts are sent through the abstract LLM boundary.
- LLM interface → Code gen: returns text · Model output is passed to code generation and validation.
- Code gen → Executor: executes · Validated generated code flows into execution.
- Executor → Responses: formats · Execution results are normalized into typed responses.
- Dataframe → Data loaders: loads from · Dataframe abstractions are backed by source-specific loaders.
- Datalake → Data loaders: ingests · Datalake composes multiple data sources through loaders.
- Data loaders → Query builders: plans queries · Loaders rely on query builders for source-specific query generation.
- Query builders → Prompts: context · Query builders supply source context used in prompt construction.
- Helpers → Agent: supports · Shared utilities support orchestration, logging, memory, and telemetry.
- Helpers → Code gen: sanitizes · Helpers provide SQL and environment safety utilities used before execution.
- Helpers → Executor: supports · Execution depends on file, session, and serialization helpers.
- Sandbox API → Executor: isolates · Core execution can delegate to a sandbox boundary.
- Docker sandbox → Sandbox API: implements · Docker extension provides a concrete sandbox implementation.
- LLM interface → OpenAI: adapter · OpenAI extension implements the model boundary.
- LLM interface → LiteLLM: adapter · LiteLLM extension implements the model boundary.
- Data loaders → SQL connector: extension path · SQL connector is an optional packaged source integration.
- Data loaders → YFinance: extension path · YFinance connector is an optional packaged source integration.
- Skills → Prompts: extends · Enterprise skills surface affects prompt and tool behavior.
- EE connectors → Data loaders: extends · Enterprise connectors add additional data source backends.
- EE vectorstores → Prompts: augments · Enterprise vector stores extend retrieval context for prompts.
- Docs v2 → Docs v3: migration · Documentation shows the transition between API generations.