How datalens-tech/datalens works
Main components
• UI Component – A Single Page Application and its supporting Node.js server. It acts as the front door for users, proxying requests and handling light data postprocessing for charts.
• Backend Services – A collection of Python applications and libraries responsible for connecting to data sources, generating queries, and processing the resulting data.
• UnitedStorage (US) Service – A Node.js service that uses PostgreSQL to persist metadata and configuration. This component is essential for saving user settings and storing metadata.
• Infrastructure and Deployment – Docker Compose files (docker-compose.yml, docker-compose-dev.yml, docker-compose.zitadel.yml), Kubernetes-related deployment setups including Helm charts (in the deploy/helm folder) and Terraform scripts (in deploy/terraform). These files illustrate the project’s containerized, orchestrated deployment strategy.
• Auxiliary Components – Additional utilities such as GitHub workflows for CI/CD, changelog generating scripts, and initialization scripts (e.g., init.sh) which configure authentication (Zitadel) and other environment settings.
Relationships and Interactions
• The UI communicates with the Backend Services via REST APIs or proxy layers, handling chart data retrieval and user interaction.
• Backend services connect to various data sources and then forward processed data to the UI.
• The UnitedStorage service manages metadata in a PostgreSQL database. The UI and Backend may indirectly interact with metadata for dynamic configuration or data persistence.
Architectural Patterns and Design Principles
• Containerization: The entire system is built as a multi-container Docker application, ensuring that each service runs in its own isolated environment.
• Microservices Separation: Logical separation of concerns is maintained across UI, backend, and metadata services. You may also indicate that services can potentially scale independently.
• CI/CD and Infrastructure as Code: With GitHub workflows, Helm templates, and Terraform configurations, the project employs modern DevOps practices.
• Cloud-Native and Orchestration: Emphasize the Kubernetes and Helm deployment strategy, which is critical for production deployments on platforms like Yandex Cloud.
• Extensibility: The services are designed to easily integrate additional connectors and features (e.g., extending frontend visualizations by replacing Highcharts with D3.js).
Relevant Technologies, Frameworks, and Libraries
• Docker and Docker Compose for container orchestration.
• Kubernetes (with Helm charts and Terraform scripts) for production deployments.
• Node.js for both the UI and UnitedStorage services.
• Python for the backend components.
• PostgreSQL as the metadata database.
• Third-party libraries: Highcharts (with license considerations) and D3.js for charting.
• Zitadel for authentication (optional, shown in the Zitadel-specific docker-compose file).