How seeongh/enterprise-info-crawler works
Project Type and Purpose
• This is a backend service built with Java 17 and Spring Boot 3.x. Its primary goal is to read CSV files containing information provided by the 공정거래위원회, augment that data by consuming public API data, and then save the consolidated information into an H2 in-memory database. The system also utilizes asynchronous processing for API calls using @Async and CompletableFuture, and provides a consistent API response format with detailed error handling.
• Keep in mind that the design emphasizes extensibility: configuration-driven API settings, an abstract storage wrapper (CorpMastStore) with a JPA implementation (JpaCorpMastStore), and robust exception logging and handling.
Analyzing the File Structure
• Top-Level Files: Build scripts (build.gradle, settings.gradle, gradlew, etc.) indicate usage of Gradle for build/deployment.
• /src/main/java: Contains the primary application code. The package structure reveals several layers:
a. The entry point (EnterpriseInfoCrawlerApplication.java).
b. The API layer for "coseller" – features packages for application services (CoSellerService, CorpApiService, CsvService, RegionApiService), clients (CorpApiClient, RegionApiClient), DTOs for data transfer, and request handling.
c. Domain & infrastructure: Domain models (CorpMast), repository interfaces (CorpMastRepository), and an abstraction for data storage (CorpMastStore with JpaCorpMastStore).
d. Common modules: Exception definitions, response format (ApiResponse), constants, and validation classes.
e. Configuration classes: AsyncConfig, JacksonConfig, RestTemplateConfig, ensuring customizations for asynchronous processing, JSON mapping, and API communication.
f. Utility classes (e.g., AddressUtil) to aid in data processing.
• /src/main/resources: Contains configuration (application.yml), logging configuration (logback-spring.xml), and sample CSV files.
• /src/test: Contains exhaustive tests from unit tests for controllers, services, integration of repositories, and utilities.
Major Components
a. API Controllers (e.g., CoSellerRestController):
• Handles incoming HTTP requests.
• Coordinates with application services.
b. Application Services:
• CoSellerService: Business logic orchestrator.
• CorpApiService, CsvService, RegionApiService: Each responsible for file parsing, external API calls, or region-specific processing.
c. API Clients:
• CorpApiClient and RegionApiClient: Manage communication with external public APIs using RestTemplate.
d. Domain and Persistence:
• Domain models (e.g., CorpMast) capture business data.
• Persistence layer includes CorpMastRepository and the storage abstraction (CorpMastStore—JpaCorpMastStore) to interact with the H2 in-memory DB.
e. Configuration and Cross-Cutting Concerns:
• Configuration classes like AsyncConfig, JacksonConfig, and RestTemplateConfig clearly indicate dependency injection and customization points.
• Logging (Logback configuration) and exception handling (GlobalExceptionHandler, CustomException types) ensure maintainability and observability.
f. Utility and Common Modules:
• AddressUtil and other helper classes.
• Common response and validation classes to provide uniform client responses.
Relationships and Interactions
• The Controller layer receives requests and forwards them to the corresponding Application Services.
• Application Services read CSV files (using CsvService) and initiate asynchronous API calls via API Clients.
• API Clients communicate with external public services and return responses, which are processed by their respective services (CorpApiService, RegionApiService).
• Processed data is formatted into the domain model (CorpMast) then saved via the repository layer using the storage abstraction.
• Exception handling is centralized through GlobalExceptionHandler, ensuring that errors (CSV parsing failures, external API failures) are logged and managed gracefully.
• Configuration files from application.yml and logging setup ensure that runtime behavior (API parameters, logging levels) are easily modifiable.
Architectural Patterns and Design Principles
• Layered Architecture: The structure follows a clear separation of concerns between presentation (API controllers), business logic (services), and persistence (repositories).
• Asynchronous Processing: Use of @Async and CompletableFuture helps in building scalable API calls.
• Abstraction: The separation of storage logic to support potential changes in DB technology.
• Configuration-Driven Design: External API settings are loaded from YAML, which improves adaptability when external interfaces change.
• Exception Handling and Consistent Response Patterns: Every layer uses custom exceptions and common responses (ApiResponse).