# u2u-labs/layerg-marketplace-crawler architecture

> Architecture diagram of the GitHub repository u2u-labs/layerg-marketplace-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/u2u-labs/layerg-marketplace-crawler
- Repository: https://github.com/u2u-labs/layerg-marketplace-crawler

## Overview

##### Project type and purpose
- This is a Go-based backend microservice (no UI) whose primary purpose is to crawl EVM blockchains, index on-chain asset and history data, persist it to a relational datastore, process backfills, and expose that data over an HTTP/RPC API. It also uses an asynchronous worker pipeline for long-running or backfill jobs.

##### File structure
- Top level:
- Dockerfile(s), Makefile, docker-compose.yaml → containerization & local orchestration.
- config/params.go → central configuration (DB, RPC endpoints, queue URLs).
- cmd/ → entrypoints for API, crawler, backfill, and worker processes.
- db/ → SQL migrations, query definitions, and sqlc-generated Go code for Postgres.
- docs/ → Swagger/OpenAPI spec driving API docs.
- main.go → bootstraps application.
- cmd/ subfolders reflect a layered hexagonal structure:
- controllers → HTTP handlers
- services → business logic
- libs, helpers, utils → integrations (SQS, Pusher, Redis, RPC clients)
- middlewares → auth, logging, etc.
- External tooling: AWS SQS for task queueing, Redis for caching/unique sets, Ethereum JSON-RPC endpoint(s).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
        direction TB
        Ethereum["Ethereum JSON-RPC"]:::external
        SQS["AWS SQS Queue"]:::external
    end

    %% Core Microservice
    subgraph "Core Microservice"
        direction TB
        subgraph "API Server"
            direction TB
            APIEntry["API Entrypoint"]:::core
            Config["Config Manager"]:::core
            Controllers["HTTP Controllers"]:::core
            Services["Business Services"]:::core
            Repo["sqlc Store"]:::core
            Middleware["HTTP Middleware"]:::core
            ResponseFmt["Response Formatter"]:::core
        end

        subgraph "Worker Processes"
            direction TB
            WorkerEntry["Worker Entrypoint"]:::core
            CrawlerEntry["Crawler Entrypoint"]:::core
            RPCEntry["RPC Server"]:::core
            OnchainClient["On-chain RPC Clients"]:::core
            QueueLib["SQS Integration"]:::core
            PusherLib["Pusher Notifications"]:::core
            Utils["Utility Helpers"]:::core
        end
    end

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        Postgres[(Postgres)]:::external
        Redis[(Redis Cache)]:::external
    end

    %% Dev/Ops
    subgraph "Dev/Ops"
        direction TB
        Migrations["DB Migrations"]:::infra
        DockerComp["Docker & Compose"]:::infra
        MakefileNode["Makefile"]:::infra
        APIDocs["API Documentation"]:::infra
    end

    %% Interactions
    Ethereum -->|RPC Calls| OnchainClient
    SQS -->|Tasks| WorkerEntry
    APIEntry -->|HTTP Requests| Controllers
    Controllers -->|Calls| Services
    Services -->|Queries/Writes| Repo
    Services -->|Enqueue| QueueLib
    WorkerEntry -->|Polls| QueueLib
    WorkerEntry -->|Uses| OnchainClient
    WorkerEntry -->|Writes| Repo
    WorkerEntry -->|Caches| Redis
    WorkerEntry -->|Notify| PusherLib
    RPCEntry -->|RPC Calls| Services
    APIEntry -->|Reads/Writes| Repo
    APIEntry -->|Caches| Redis

    Migrations -->|Manage Schema| Postgres
    DockerComp --> APIEntry
    DockerComp --> WorkerEntry
    DockerComp --> Postgres
    DockerComp --> Redis

    APIDocs -->|Specs| Controllers
    MakefileNode --> DockerComp

    Config --> APIEntry
    Config --> WorkerEntry

    %% Click Events
    click Config "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/config/params.go"
    click APIEntry "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/api_cmd.go"
    click APIEntry "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/api.go"
    click Controllers "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/controllers/asset.controller.go"
    click Controllers "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/controllers/backfill.controller.go"
    click Controllers "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/controllers/chain.controller.go"
    click Controllers "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/controllers/history.controller.go"
    click Services "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/services/asset.service.go"
    click Services "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/services/chain.service.go"
    click Middleware "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/middlewares/basic-auth.go"
    click ResponseFmt "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/response/response.go"
    click RPCEntry "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/rpc.go"
    click WorkerEntry "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/worker_cmd.go"
    click WorkerEntry "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/worker.go"
    click CrawlerEntry "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/crawler_cmd.go"
    click OnchainClient "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/helpers/u2u-client.go"
    click OnchainClient "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/onchain.go"
    click QueueLib "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/libs/sqs.go"
    click PusherLib "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/libs/pusher.go"
    click Utils "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/utils/asset-filter-param.go"
    click Utils "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/utils/unmarshal.util.go"
    click Utils "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/utils/unpack_log.go"
    click Utils "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/cmd/utils/util.go"
    click Migrations "https://github.com/u2u-labs/layerg-marketplace-crawler/tree/main/db/migrations/"
    click Repo "https://github.com/u2u-labs/layerg-marketplace-crawler/tree/main/db/sqlc/"
    click DockerComp "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/docker-compose.yaml"
    click MakefileNode "https://github.com/u2u-labs/layerg-marketplace-crawler/tree/main/Makefile"
    click APIDocs "https://github.com/u2u-labs/layerg-marketplace-crawler/blob/main/docs/swagger.yaml"

    %% Styles
    classDef core fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef external fill:#c8e6c9,stroke:#388e3c,color:#1b5e20
    classDef infra fill:#ffe0b2,stroke:#f57c00,color:#e65100
```
