How thuantd2207/web-news-main works
Main Components
• Airflow Environment – This includes the scheduler, DAGs (in the “dags” directory), and task logs (found in the “logs” folders). Airflow orchestrates tasks such as scraping data (using the crawler scripts), processing it (using functions like database_func.py and model_func.py), and coordinating data flows.
• Flask Web Application – Located in the “web-app” folder (with app.py and index.html). This component provides the user interface which displays the news data.
• PostgreSQL Database – Even though a dedicated folder is not present, the README mentions PostgreSQL as the storage backend for news data.
• Docker / Docker-Compose Setup – With Dockerfile and docker-compose.yaml, the entire system infrastructure (Airflow, Flask, PostgreSQL) is containerized, ensuring consistent deployment.
• Environment & Configuration – The existence of a .env file and other configuration files help define environment variables and settings, which you might represent as a configuration or settings layer.
Relationships and Interactions
• Web UI: Illustrate that the Flask app queries the PostgreSQL database to fetch the latest news data and displays it to users.
• Containerization: Include the Docker container environment as an encapsulating layer around the Airflow, Flask, and PostgreSQL components. You could show these containers connected via a Docker network.
• Logging: Optionally include a note about the logging subsystem where Airflow writes out logs (via log directories) for monitoring and troubleshooting.
Architectural Patterns and Design Principles
• Modularity – The separation between components (Airflow for orchestration and scraping, Flask for presentation, and PostgreSQL for persistence) demonstrates a modular design.
• Containerization – Use of Docker demonstrates microservice principles and ensures each service can be developed, managed, and deployed independently.
• Workflow Orchestration – The use of Airflow indicates the project follows a workflow management pattern, with clear separation between tasks defined in individual DAG files (crawler_dag.py, etc.).
• Extensibility – The directory structure with clear segregation (logs, DAGs, web-app) helps in scaling and adding new functionality.