How nandishns/automated-software-and-product-listing-platform works
Main Components
• Data Acquisition:
– Web Scraper Component (located in the “Scraper” directory) that uses BeautifulSoup and Selenium to gather product details.
– Crawler (the “crawler.py” file) which may complement scraping.
• Data Ingestion & Streaming:
– Apache Kafka (and Zookeeper) as defined by the “docker-compose.yml” and “kafka-zoopkeeper-deployment.yaml” files.
• Processing Components (Kafka Consumers):
– Three separate Dockerized consumers, each in its own directory under “KafkaConsumer”:
∘ Software consumer
∘ TwitterLLM consumer
∘ NewsLLM consumer
– Each consumer has its own cron job YAML file (e.g., “product-cronjob.yaml”, “twitter-cronjob.yaml”, “news-cronjob.yaml”) for scheduled execution.
• Backend API & Interface:
– Flask Application (in “Flask/app.py”) acting as a backend service.
– Frontend website (in “Frontend/website.py”) likely serving data or visualizations.
• Data Storage:
– MongoDB Atlas (as mentioned in the README, integrating persistent storage for new listings).
• External Services & Additional Processing:
– Integration points for the G2 API.
– Large Language Models (LLMs) used for advanced text analysis on streamed content.
• Containerization & Orchestration:
– Docker containers for each service.
– Kubernetes scheduled jobs (using provided YAML files) for orchestrating tasks.
Relationships and Interactions
• Data flows from the Scraper to the Kafka topics –
– The scrapers obtain initial product information and send these events or messages to corresponding Kafka topics.
• Kafka acts as the central data pipeline –
– Different topics (e.g., “software”, “news”, “x-llm”) separate data streams.
• Kafka Consumers subscribe to these topics –
– Each consumer processes the incoming data (e.g., data from Twitter or news sites) before pushing it to a datastore or further analysis.
• Processed data is then sent to MongoDB –
– New listings added or updated in the database trigger updates in the product listing.
• The Flask API and Streamlit Frontend provide user interfaces –
– These components read from MongoDB and/or Kafka to display updated product information.
• All components are containerized and scheduled –
– Docker handles the packaging, while Kubernetes cron jobs manage regular scraping and consumer tasks.
Architectural Patterns and Design Principles
• Microservices & Containerization: Each major component (scraper, each Kafka consumer, backend API) is isolated in its own container.
• Event-driven Architecture: Kafka is the backbone ensuring decoupled communication among data producers and consumers.
• Modularity & Scalability: The project is designed to easily add new data sources or processing components by following the established pattern (each with its own consumer and cron job).
Relevant Technologies, Frameworks, and Libraries
• Scraping: BeautifulSoup and Selenium.
• Data Streaming: Apache Kafka (with embedded Zookeeper).
• Containerization/Orchestration: Docker and Kubernetes.
• Web Frameworks: Flask (backend) and Streamlit (frontend).
• Data Storage: MongoDB Atlas.
• Integration with External APIs: G2 API and potential LLM services.
• Scheduling & Automated Tasks: Cron jobs defined via YAML files.