How rodrigovp/fragmentos works
Project Type and Purpose
• This is a Java-based Spring Boot application that acts as a simple web crawler (“Fragmentos”) designed to archive articles from the old blog of Phillip Calçado.
• The main goal is to fetch, parse, and store (or generate) HTML versions of blog articles. It makes use of Jsoup for reading HTML, and the README indicates the need for a MongoDB instance (set up via docker-compose) which implies persistence or temporary storage of data.
Analyzing the File Structure
• Top-level items such as pom.xml, mvnw, and mvnw.cmd indicate that the project uses Maven as its build tool.
• The presence of Docker Compose (docker-compose.yml) suggests that containerization (at least for external services like MongoDB) is part of the deployment.
• The project is divided into several classes that seem to encapsulate specific tasks:
• Artigo.java, Conteudo.java, and TemplateDeArtigo.java deal with article data and formatting.
• LeitorDeArtigos, LeitorDeArtigosPorMes, LeitorDeDataDeArtigo, and LeitorDeURL indicate specialized readers or parsers for different aspects of the content.
• GeracaoDeHtml and Importacao represent modules likely handling output generation and data import respectively.
• The test directory under src/test indicates that unit testing is in place for several modules.
a. Main Components:
• Application Layer: Represent the Spring Boot application (FragmentosApplication) that initializes and controls the workflow.
• Crawler Module: Include components like LeitorDeArtigos, LeitorDeURL, LeitorDeArtigosPorMes, and LeitorDeDataDeArtigo. This module is responsible for fetching and parsing the HTML content using Jsoup.
• Processing & Transformation Layer: Components such as TemplateDeArtigo and GeracaoDeHtml that take parsed data and generate HTML versions or apply templates.
• Integration/Runner Component: FragmentosRunner, which might coordinate the crawling and processing tasks automatically at startup.
• Configuration Management: The application.yml file under src/main/resources which holds relevant configurations (possibly for database connections, application properties, etc.).
• Docker/Deployment: The docker-compose.yml file that orchestrates a MongoDB container, indicating that deployment involves containerizing some dependencies.
b. Relationships and Interactions:
• The Spring Boot Application initiates the runner (FragmentosRunner) on start-up.
• The runner calls the appropriate crawler services (LeitorDeArtigos and related classes) to fetch web content.
• After fetching, the processing/transformation module (GeracaoDeHtml and TemplateDeArtigo) generates a formatted output.
• Once processed, the results might be stored or serve as the basis for further actions (e.g., archiving) with MongoDB.
• The data flow is directional: Start at the Application entry point → Trigger the crawler → Process/transform the data → Persist the results, with configuration and external dependencies clearly noted.
c. Important Architectural Patterns:
• Modular separation: Each domain concern (crawling, processing, persisting) is encapsulated into its own set of classes.
• Use of Spring Boot promotes dependency injection and inversion of control, leading to a loosely coupled design.
• The integration with an external service (MongoDB) via containerization (Docker) suggests an emphasis on dev/test parity and portability.
d. Relevant Technologies and Frameworks:
• Java/Spring Boot: The base framework for building and running the application.
• Maven: For build and dependency management.
• Jsoup: For HTML parsing and web content extraction.
• MongoDB: As the persistence layer (as indicated by the README and docker-compose.yml).
• Docker Compose: For container management of MongoDB during development or deployment.
Elements to be Included
• Use clear labels for each component (e.g., “Spring Boot Application Entry Point”, “Crawler Service (LeitorDeArtigos, LeitorDeURL, etc.)”, “Content Processor (GeracaoDeHtml, TemplateDeArtigo)”, “MongoDB Instance”).
• Data flow from the web (source) to the crawler module.
• The control flow from the Spring Boot Runner to its dependent services.
• The flow from processing modules to the MongoDB persistence layer.
• Application components (Spring Boot, Java classes)
• External services (MongoDB in a Docker container)
• Configuration resources (application.yml, docker-compose.yml)
Final Advice
• Use annotations or callouts if necessary to highlight particular implementation decisions (like the use of Docker Compose for MongoDB or the use of Jsoup for HTML parsing).