How u2u-labs/layerg-marketplace-crawler works
Project type and purpose
• This is a Go-based backend microservice (no UI) whose primary purpose is to crawl EVM blockchains, index on-chain asset and history data, persist it to a relational datastore, process backfills, and expose that data over an HTTP/RPC API. It also uses an asynchronous worker pipeline for long-running or backfill jobs.
File structure
• Top level:
• Dockerfile(s), Makefile, docker-compose.yaml → containerization & local orchestration.
• config/params.go → central configuration (DB, RPC endpoints, queue URLs).
• cmd/ → entrypoints for API, crawler, backfill, and worker processes.
• db/ → SQL migrations, query definitions, and sqlc-generated Go code for Postgres.
• docs/ → Swagger/OpenAPI spec driving API docs.
• main.go → bootstraps application.
• cmd/ subfolders reflect a layered hexagonal structure:
• controllers → HTTP handlers
• services → business logic
• libs, helpers, utils → integrations (SQS, Pusher, Redis, RPC clients)
• middlewares → auth, logging, etc.
• External tooling: AWS SQS for task queueing, Redis for caching/unique sets, Ethereum JSON-RPC endpoint(s).