How apify/n8n-nodes-apify-content-crawler works
1. Repository purpose
- This is an n8n community node package that wraps a single Apify actor: Website Content Crawler.
- Its job is to let n8n workflows trigger a crawl, pass input options to Apify, and consume the returned content data.
2. Primary boundary: n8n vs Apify
- The core architectural boundary is between n8n’s node execution runtime and the external Apify platform.
- The node is the adapter layer: it translates n8n parameters and credentials into an Apify actor run, then maps the actor output back into n8n items.
3. Main runtime artifact
- The central implementation lives in nodes/ApifyContentCrawler/ApifyContentCrawler.node.ts.
- Its node metadata is defined separately in ApifyContentCrawler.node.json, which is typical for n8n community nodes and keeps UI/operation definitions distinct from execution logic.
4. Operation model
- The node exposes a single operation: Run Crawler.
- Input configuration is organized under resources/properties/, with separate property definitions for single URL crawl and multiple URL crawl modes.
- This indicates the node’s UI/input schema is intentionally split by crawl shape rather than by multiple unrelated features.
5. Execution path
- The execution flow is: n8n workflow input → node parameter parsing → Apify actor invocation → result collection → n8n output items.
- helpers/executeActor.ts appears to encapsulate the actual Apify execution call.
- helpers/genericFunctions.ts and helpers/consts.ts likely provide shared transformation, validation, or naming logic used by the node implementation.
6. Authentication subsystem
- The repository supports two credential types in credentials/:
- ApifyApi.credentials.ts for API key authentication
- ApifyOAuth2Api.credentials.ts for OAuth2
- This is an important architectural choice: the node is not tied to one auth model, and the credential layer is separated from the node logic, following n8n’s credential boundary.
7. Input/UX structure
- The resources/resources.ts and resources/properties/* files define how the node is presented in the n8n editor.
- The README’s screenshots and the separation into resource/property files suggest the node uses n8n’s standard declarative configuration model rather than custom UI code.
8. Test architecture
- Tests live alongside the node under nodes/ApifyContentCrawler/__tests__/.
- The test suite includes workflow-level fixtures and utilities such as:
- workflow execution helpers
- credential helpers
- result extraction helpers
- workflow JSON fixtures
- This suggests the node is verified through integration-style n8n execution tests, not just unit tests, which is important for a node that depends on credential handling and workflow wiring.
9. Packaging and release pipeline
- The repo is set up as a distributable npm package, with release automation in .github/workflows/publish.yml.
- Release flow described in the README indicates the package version is updated and published to npm from GitHub Actions after a tagged GitHub Release.
- package-lock.json, package.json, and CI workflows show this is meant to be shipped as a versioned integration artifact, not run as a standalone service.
10. Quality gates and developer tooling
- The project uses tsconfig.json, eslint.config.mjs, .prettierrc.js, jest.config.js, and gulpfile.js.
- This implies a TypeScript-first codebase with linting, formatting, test execution, and a build step integrated into CI.
- The .github/workflows/ci.yml likely runs validation before release, while claude-md-maintenance.yml is repo maintenance automation.
11. Documentation and operational assumptions
- The README documents n8n setup, local development, credential setup, and the need for a public WEBHOOK_URL when running locally with external triggers.
- Even though this node is not itself a trigger, that note reflects the broader n8n deployment environment this package assumes.
12. Diagram-friendly mental model
- A useful architecture diagram would show:
- n8n workflow / AI Agent tool on the left
- ApifyContentCrawler node as the adapter in the middle
- Apify API / Website Content Crawler actor on the right
- Credentials as a separate trusted input into the node
- Test/CI/release pipeline as a secondary support track around the package
- The key data boundary is the actor input/output payload; everything else is adapter, config, or packaging around that flow.
Main entry points: ApifyContentCrawler.node.ts Crawler Node · ci.yml CI · resources.ts Resources · Apify.node.spec.ts Node Tests
Components 19
Runtime
The n8n node, its inputs, helpers, and external Apify boundary.
- Crawler Node · Main execution entry point that runs the Apify crawler from n8n workflows.
nodes/ApifyContentCrawler/ApifyContentCrawler.node.ts - Node Meta · Declarative node metadata used by the n8n editor and runtime.
nodes/ApifyContentCrawler/ApifyContentCrawler.node.json - Resources · Resource registry that organizes the node's editor-facing input model.
nodes/ApifyContentCrawler/resources/resources.ts - Single URL · Properties for crawling one URL at a time.
nodes/ApifyContentCrawler/resources/properties/scrapeSingle.ts - Multiple URLs · Properties for crawling a URL set or batch.
nodes/ApifyContentCrawler/resources/properties/scrapeMultiple.ts - Actor Exec · Encapsulates the actor run and result retrieval against Apify.
nodes/ApifyContentCrawler/helpers/executeActor.ts - Helpers · Common constants and transformation functions used by the node.
nodes/ApifyContentCrawler/helpers/genericFunctions.ts - Constants · Shared names and fixed values for the crawler integration.
nodes/ApifyContentCrawler/helpers/consts.ts
Support
Credentials, tests, build, CI, and release tooling around the package.
- Package · Distributable n8n community node package for the Website Content Crawler actor.
package.json - API Key · API key credential for Apify authentication.
credentials/ApifyApi.credentials.ts - OAuth2 · OAuth2 credential for Apify authentication.
credentials/ApifyOAuth2Api.credentials.ts - Node Tests · n8n execution tests for the crawler node and its workflow wiring.
nodes/ApifyContentCrawler/__tests__/Apify.node.spec.ts - Fixtures · Workflow JSON and helper fixtures used by the integration tests.
nodes/ApifyContentCrawler/__tests__/workflows/actors/run-actor-advanced.workflow.json - CI · Validation workflow for build, lint, and tests.
.github/workflows/ci.yml - Publish · Release workflow that publishes the package to npm.
.github/workflows/publish.yml - Build · TypeScript build and package task runner.
gulpfile.js - Lint/Format · Project linting and formatting configuration.
eslint.config.mjs - README · Usage, setup, and release guidance for the package.
README.md
Other components
- Apify API · External Apify platform used to run the Website Content Crawler actor.
Connections 18
- Package → Crawler Node: ships · The npm package delivers the n8n node implementation.
- Crawler Node → Node Meta: uses · Node runtime is paired with separate editor/runtime metadata.
- Crawler Node → Resources: loads · Execution logic consumes the declarative resource model.
- Resources → Single URL: defines · Single-URL crawl mode properties are registered here.
- Resources → Multiple URLs: defines · Multiple-URL crawl mode properties are registered here.
- Crawler Node → Helpers: uses · Node logic reuses shared transformation and validation helpers.
- Helpers → Constants: shares · Shared constants support the node's internal conventions.
- Crawler Node → Actor Exec: calls · The node delegates the Apify run and result collection to the executor.
- Actor Exec → Apify API: invokes · Executor sends the crawl request to the external Apify platform.
- API Key → Crawler Node: auth · API key credentials are consumed by the node when configured.
- OAuth2 → Crawler Node: auth · OAuth2 credentials are consumed by the node when configured.
- Node Tests → Crawler Node: verifies · Integration tests exercise the node through n8n workflow execution.
- Node Tests → Fixtures: uses · Workflow fixtures provide realistic execution scenarios.
- CI → Node Tests: runs · CI validates the node through automated test execution.
- CI → Build: runs · CI includes build validation for the distributable package.
- CI → Lint/Format: runs · CI applies linting and formatting checks.
- Publish → Package: releases · Release workflow publishes the versioned npm package.
- README → Crawler Node: explains · Documentation describes how to configure and run the node.