How openai/openai-realtime-solar-system works
Main Components
• Frontend UI: Represented by the NextJS app/page.tsx, layout.tsx in the app directory, and the various React components under the components folder (e.g., scene.tsx for rendering the 3D scene, charts components for data visualizations, and controls for user interactions).
• API Endpoints: Under app/api, there are two endpoints:
• /api/iss (to fetch and display the ISS position)
• /api/session (to manage realtime sessions)
• Library/Configuration Layer: Found in the lib directory (config.ts, constants.ts, components-mapping.tsx). This layer defines behavior such as tool configuration, API keys, and interactions for function calling.
• External Services & Integrations:
• OpenAI Realtime API & WebRTC: Facilitates realtime function calling and voice interaction.
• Spline: Provides the 3D scene of the solar system and is integrated via the scene component (components/scene.tsx) by loading a remote Spline scene.
Relationships and Interactions
• The user's browser (client) loads the NextJS application.
• The UI is split into different components, where:
• The Scene component integrates with Spline to render the 3D solar system.
• Chart components display data (bar chart, pie chart) based on user queries.
• Control components handle interactive elements (like starting/stopping sessions or toggling the microphone).
• Voice commands and UI interactions trigger calls to API endpoints:
• The /api/session endpoint handles starting and stopping realtime sessions.
• The /api/iss endpoint fetches ISS related data.
• The lib/config.ts and lib/constants.ts file play key roles in mapping interactions (via OpenAI’s function-calling features) to appropriate actions in the UI, such as focusing on planets, showing orbits, triggering animations, etc.
Architectural Patterns and Technologies
• The project uses a modern React/NextJS architecture, separating presentation (UI components) and business logic (API endpoints and configuration libraries).
• It leverages event-driven integrations via the OpenAI Realtime API and WebRTC for realtime updates.
• The use of external 3D rendering through Spline introduces an integration pattern where browser-based rendering and external events are coordinated.
• The API endpoints illustrate a RESTful design pattern, encapsulated within the NextJS App Router.