Getting started with Fetcher

Two paths to a first Fetcher extraction: the in-memory engine harness with no infrastructure, or the full Manager and Worker stack with Docker Compose.

There are two ways to see Fetcher work. Pick one.

Path	What it costs	What it proves
A — Embedded Engine	One go get. No infrastructure.	Extraction rules, planning, limits, and the direct-mode result.
B — Standalone services	Docker Compose. MongoDB, RabbitMQ, storage, Valkey.	The REST API, asynchronous jobs, and stored results.

Path A is the shortest route to a first success. Start there if you only want to evaluate the product.

Path A — Run the Engine with no infrastructure

---

The Engine ships an in-memory harness at pkg/engine/memory. It covers the storage-facing ports: the connector registry, the connection store, the schema cache, the result sink, and the execution store. You need no MongoDB, no RabbitMQ, and no object storage. It ships no CredentialProtector, so turning encrypted persistence on means supplying your own.

1. Add the module

   go get github.com/LerianStudio/fetcher/pkg/engine
   

   The Engine is a separate module from the services. It has no third-party dependencies, so this import pulls in nothing else.

2. Construct, plan, execute

   package main
   
   import (
   	"context"
   	"fmt"
   	"log"
   
   	"github.com/LerianStudio/fetcher/pkg/engine"
   	"github.com/LerianStudio/fetcher/pkg/engine/memory"
   )
   
   func main() {
   	ctx := context.Background()
   
   	store := memory.NewConnectionStore()
   	registry := memory.NewConnectorRegistry()
   
   	// WithConnectorRegistry is the only required option.
   	eng, err := engine.New(
   		engine.WithConnectorRegistry(registry),
   		engine.WithConnectionStore(store),
   	)
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	// Every operation is scoped to a tenant.
   	tenant, err := engine.NewTenantContext("tenant-123")
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	conn := memory.NewTemplateConnector(memory.ConnectorBehavior{
   		Schema: engine.SchemaSnapshot{
   			ConfigName: "pg-main",
   			Tables:     []engine.TableSnapshot{{Name: "public.users", Fields: []string{"id", "email"}}},
   		},
   		Rows: map[string][]map[string]any{
   			"public.users": {{"id": 1, "email": "[e-mail]"}},
   		},
   	})
   	registry.Register("postgres", memory.NewConnectorFactory(conn))
   
   	if _, err = eng.CreateConnection(ctx, tenant, engine.NewConnectionInput(engine.ConnectionInputParams{
   		ConfigName: "pg-main",
   		Type:       "postgres",
   		Host:       "localhost",
   		Port:       5432,
   	})); err != nil {
   		log.Fatal(err)
   	}
   
   // Plan validates the request against a cache-first schema snapshot and enforces limits.
   	plan, err := eng.PlanExtraction(ctx, tenant, engine.ExtractionRequest{
   		MappedFields: map[string]engine.FieldSelection{
   			"pg-main": {"public.users": {"id", "email"}},
   		},
   	})
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	result, err := eng.ExecuteExtraction(ctx, plan)
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	fmt.Printf("rows=%d bytes=%d\n", result.Direct.RowCount, len(result.Direct.Data))
   }
   
3. Read the result

   No result sink is wired here, so the Engine runs in direct mode. It returns the rows inline as indented JSON, plus a SHA-256 digest over those exact bytes. The bytes are deterministic: the same input always produces the same digest.

To move to production, swap the memory harness for your own adapters. Fetcher’s own Manager and Worker are the reference implementation.

Path B — Run the standalone services

---

This path gives you the REST API and asynchronous jobs. Everything runs locally under Docker Compose.

Prerequisites

• Docker and Docker Compose
• Make
• Go, for development only. The toolchain version lives in the repo’s go.mod.

Set up and run

1. Clone the repository

   git clone https://github.com/LerianStudio/fetcher.git
   cd fetcher
   
2. Create the environment files

   make set-env
   

   This copies each component’s .env.example to .env.

3. Generate the master encryption key

   make generate-master-key
   

   Copy the key into APP_ENC_KEY in both components/manager/.env and components/worker/.env. Both services need the same value. The Worker uses it to decrypt credentials and to check message signatures.

   Replace the placeholder before startup: use a valid Base64-encoded 32-byte key. The placeholder created by make set-env fails while decoding an invalid Base64 master key. The master key too short: got 0 bytes, minimum 32 required message applies to an empty or short value that does decode. Fetcher has no plaintext fallback mode.

4. Start everything

   make up
   
5. Check that the API answers

   • REST API: http://localhost:4006
   • Scalar API reference, when SWAGGER_ENABLED=true: http://localhost:4006/swagger/docs
   • RabbitMQ management: http://localhost:3008

Run your first extraction

An extraction has three moves. Register a connection, create a job, then poll the job.

1. Register a database connection

curl -X POST http://localhost:4006/v1/management/connections \
  -H "Content-Type: application/json" \
  -H "X-Product-Name: quickstart" \
  -d '{
    "configName": "my_postgres",
    "type": "POSTGRESQL",
    "host": "host.docker.internal",
    "port": 5432,
    "databaseName": "mydb",
    "userName": "postgres",
    "password": "postgres"
  }'

The X-Product-Name header names the product that owns the connection. Use the same value in metadata.source on the job in step 2, because Fetcher compares the two.

Fetcher encrypts the password before it stores the record. Test the connection before you use it:

curl -X POST http://localhost:4006/v1/management/connections/{id}/test

2. Create an extraction job

Name the fields you want, per table, per datasource:

curl -X POST http://localhost:4006/v1/fetcher \
  -H "Content-Type: application/json" \
  -d '{
    "dataRequest": {
      "mappedFields": {
        "my_postgres": {
          "accounts": ["id", "email", "created_at"]
        }
      }
    },
    "metadata": {
      "source": "quickstart"
    }
  }'

The API answers 202 Accepted with a job ID. Send the same request twice within 5 minutes and you get 200 OK with the first job instead of a second one. A failed job does not block a retry.

3. Poll the job

curl http://localhost:4006/v1/fetcher/{id}

A job ends in one of two terminal states: completed or failed. On completion the Worker has encrypted the result into object storage and published a job.completed event. The two states before that are pending and processing. Extraction jobs gives the full four-state lifecycle.

Turn on authentication with PLUGIN_AUTH_ENABLED=true. Requests then carry an Authorization: Bearer <token> header. This quickstart runs with authentication off.

Next steps

---

Core concepts

Connections, schema discovery, jobs, filters, and results.

Configuration

Every environment variable, per component.

Deployment

Dependencies, queues, scaling, and the fail-closed startup checks.

Security

Key derivation, rotation, message signing, and host validation.