How shrimaliharshal/tradewise works
Project Type and Purpose
• This is a hybrid system that integrates real-time stock data ingestion, prediction, visualization, and an AI-powered news/article fetching component.
• The system features two primary workflows: one for live stock data streaming and prediction (using Kafka, external APIs, and Streamlit dashboards) and another for StockBot—which processes news article URLs using LangChain, converts content with OpenAI embeddings, and indexes vectors with FAISS, enabling interactive chat-based information retrieval.
• The project employs Docker and Docker Compose for container orchestration, which ensures scalability and ease of deployment.
Analyzing the File Structure
• You have several Python scripts handling different system responsibilities:
• Data Fetching & Streaming: ‘streaming_kafka.py’, ‘alpaca_kafka.py’, ‘alphavantage.py’, and possibly ‘firebaseSetup.py’ (for additional services).
• Data Prediction and Processing: ‘prediction.py’ for forecasting stock prices and ‘indicators_test.py’ along with ‘traditionalGraphs.py’ for visualization.
• Interactive Dashboards: ‘dashboard.py’ uses Streamlit to deliver real-time visual data.
• StockBot Module: Located in the “StockBot” directory with ‘main.py’, a requirements file, .env, and a vector index pickle file. This encapsulates capabilities for content processing and vector embedding using LangChain, FAISS, and OpenAI.
• Additional modules like ‘alerts.py’, ‘app.py’, and others are supporting components that likely contribute to routing, error handling, or additional service integrations.
• The Docker Compose file indicates that components are containerized, which reinforces that the deployment setup is modular and each service/component is isolated yet interconnected.
• Break the system into clear, distinct modules. Label and outline the following high-level components:
A. External Data Sources and APIs:
• External stock market APIs (e.g., Alpaca, AlphaVantage) as data providers.
B. Data Ingestion and Streaming Layer:
• Module “streaming_kafka.py” as the real-time data fetcher.
• “alpaca_kafka.py” and “alphavantage.py” as additional ingestion components.
C. Data Processing and Prediction Module:
• “prediction.py” as the component that subscribes to the Kafka topic and applies prediction models.
D. Interactive Dashboard Layer:
• “dashboard.py” (and potentially “app.py”) as the user interface layer powered by Streamlit.
• That this dashboard displays both real-time insights and prediction results.
E. StockBot Component:
. “main.py” for orchestration of content processing.
. The use of LangChain’s UnstructuredURL Loader for fetching article content.
. Conversion of content to vectors via OpenAI embeddings and subsequent indexing using FAISS (where “vector_index.pkl” is stored).
. A chat interface module (possibly integrated in the same container or via another microservice) for user interaction.
F. Deployment and Orchestration:
• Docker and Docker Compose as the underlying infrastructure. Represent each primary component (or group) as its own Docker container, indicating container interconnectivity.
Relationships and Communication
• Dependencies such as API calls, Kafka subscriptions, and containerized service communications within Docker.
Architectural Patterns and Technologies
• Stress the modular architecture which separates real-time data processing from AI-assisted information retrieval.
• Use of containerization to decouple services and streamline deployment.
• Mention that the architecture supports scalability by leveraging Apache Kafka for data streaming and Docker Compose for service orchestration.
• Technologies: Python, Kafka, Docker, Streamlit, LangChain, OpenAI Embeddings, FAISS, and any external API services.