> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# Getting started with Fetcher

> Two paths to a first Fetcher extraction: the in-memory engine harness with no infrastructure, or the full Manager and Worker stack with Docker Compose.

There are two ways to see Fetcher work. Pick one.

| Path                        | What it costs                                       | What it proves                                                  |
| --------------------------- | --------------------------------------------------- | --------------------------------------------------------------- |
| **A — Embedded Engine**     | One `go get`. No infrastructure.                    | Extraction rules, planning, limits, and the direct-mode result. |
| **B — Standalone services** | Docker Compose. MongoDB, RabbitMQ, storage, Valkey. | The REST API, asynchronous jobs, and stored results.            |

Path A is the shortest route to a first success. Start there if you only want to evaluate the product.

## Path A — Run the Engine with no infrastructure

***

The Engine ships an in-memory harness at `pkg/engine/memory`. It covers the storage-facing ports: the connector registry, the connection store, the schema cache, the result sink, and the execution store. You need no MongoDB, no RabbitMQ, and no object storage. It ships no `CredentialProtector`, so turning encrypted persistence on means supplying your own.

<Steps>
  <Step title="Add the module">
    ```bash theme={null}
    go get github.com/LerianStudio/fetcher/pkg/engine
    ```

    The Engine is a separate module from the services. It has no third-party dependencies, so this import pulls in nothing else.
  </Step>

  <Step title="Construct, plan, execute">
    ```go theme={null}
    package main

    import (
    	"context"
    	"fmt"
    	"log"

    	"github.com/LerianStudio/fetcher/pkg/engine"
    	"github.com/LerianStudio/fetcher/pkg/engine/memory"
    )

    func main() {
    	ctx := context.Background()

    	store := memory.NewConnectionStore()
    	registry := memory.NewConnectorRegistry()

    	// WithConnectorRegistry is the only required option.
    	eng, err := engine.New(
    		engine.WithConnectorRegistry(registry),
    		engine.WithConnectionStore(store),
    	)
    	if err != nil {
    		log.Fatal(err)
    	}

    	// Every operation is scoped to a tenant.
    	tenant, err := engine.NewTenantContext("tenant-123")
    	if err != nil {
    		log.Fatal(err)
    	}

    	conn := memory.NewTemplateConnector(memory.ConnectorBehavior{
    		Schema: engine.SchemaSnapshot{
    			ConfigName: "pg-main",
    			Tables:     []engine.TableSnapshot{{Name: "public.users", Fields: []string{"id", "email"}}},
    		},
    		Rows: map[string][]map[string]any{
    			"public.users": {{"id": 1, "email": "a@example.com"}},
    		},
    	})
    	registry.Register("postgres", memory.NewConnectorFactory(conn))

    	if _, err = eng.CreateConnection(ctx, tenant, engine.NewConnectionInput(engine.ConnectionInputParams{
    		ConfigName: "pg-main",
    		Type:       "postgres",
    		Host:       "localhost",
    		Port:       5432,
    	})); err != nil {
    		log.Fatal(err)
    	}

    // Plan validates the request against a cache-first schema snapshot and enforces limits.
    	plan, err := eng.PlanExtraction(ctx, tenant, engine.ExtractionRequest{
    		MappedFields: map[string]engine.FieldSelection{
    			"pg-main": {"public.users": {"id", "email"}},
    		},
    	})
    	if err != nil {
    		log.Fatal(err)
    	}

    	result, err := eng.ExecuteExtraction(ctx, plan)
    	if err != nil {
    		log.Fatal(err)
    	}

    	fmt.Printf("rows=%d bytes=%d\n", result.Direct.RowCount, len(result.Direct.Data))
    }
    ```
  </Step>

  <Step title="Read the result">
    No result sink is wired here, so the Engine runs in **direct mode**. It returns the rows inline as indented JSON, plus a SHA-256 digest over those exact bytes. The bytes are deterministic: the same input always produces the same digest.
  </Step>
</Steps>

To move to production, swap the memory harness for your own adapters. Fetcher's own Manager and Worker are the reference implementation.

## Path B — Run the standalone services

***

This path gives you the REST API and asynchronous jobs. Everything runs locally under Docker Compose.

### Prerequisites

* [ ] **Docker** and **Docker Compose**
* [ ] **Make**
* [ ] **Go**, for development only. The toolchain version lives in the repo's `go.mod`.

### Set up and run

<Steps>
  <Step title="Clone the repository">
    ```bash theme={null}
    git clone https://github.com/LerianStudio/fetcher.git
    cd fetcher
    ```
  </Step>

  <Step title="Create the environment files">
    ```bash theme={null}
    make set-env
    ```

    This copies each component's `.env.example` to `.env`.
  </Step>

  <Step title="Generate the master encryption key">
    ```bash theme={null}
    make generate-master-key
    ```

    Copy the key into `APP_ENC_KEY` in **both** `components/manager/.env` and `components/worker/.env`. Both services need the same value. The Worker uses it to decrypt credentials and to check message signatures.

    <Warning>
      **Replace the placeholder before startup:** use a valid Base64-encoded 32-byte key. The placeholder created by `make set-env` fails while decoding an invalid Base64 master key. The `master key too short: got 0 bytes, minimum 32 required` message applies to an empty or short value that does decode. Fetcher has no plaintext fallback mode.
    </Warning>
  </Step>

  <Step title="Start everything">
    ```bash theme={null}
    make up
    ```
  </Step>

  <Step title="Check that the API answers">
    * REST API: `http://localhost:4006`
    * Scalar API reference, when `SWAGGER_ENABLED=true`: `http://localhost:4006/swagger/docs`
    * RabbitMQ management: `http://localhost:3008`
  </Step>
</Steps>

### Run your first extraction

An extraction has three moves. Register a connection, create a job, then poll the job.

#### 1. Register a database connection

```bash theme={null}
curl -X POST http://localhost:4006/v1/management/connections \
  -H "Content-Type: application/json" \
  -H "X-Product-Name: quickstart" \
  -d '{
    "configName": "my_postgres",
    "type": "POSTGRESQL",
    "host": "host.docker.internal",
    "port": 5432,
    "databaseName": "mydb",
    "userName": "postgres",
    "password": "postgres"
  }'
```

The `X-Product-Name` header names the product that owns the connection. Use the same value in `metadata.source` on the job in step 2, because Fetcher compares the two.

Fetcher encrypts the password before it stores the record. Test the connection before you use it:

```bash theme={null}
curl -X POST http://localhost:4006/v1/management/connections/{id}/test
```

#### 2. Create an extraction job

Name the fields you want, per table, per datasource:

```bash theme={null}
curl -X POST http://localhost:4006/v1/fetcher \
  -H "Content-Type: application/json" \
  -d '{
    "dataRequest": {
      "mappedFields": {
        "my_postgres": {
          "accounts": ["id", "email", "created_at"]
        }
      }
    },
    "metadata": {
      "source": "quickstart"
    }
  }'
```

The API answers `202 Accepted` with a job ID. Send the same request twice within 5 minutes and you get `200 OK` with the first job instead of a second one. A failed job does not block a retry.

#### 3. Poll the job

```bash theme={null}
curl http://localhost:4006/v1/fetcher/{id}
```

A job ends in one of two terminal states: `completed` or `failed`. On completion the Worker has encrypted the result into object storage and published a `job.completed` event. The two states before that are `pending` and `processing`. [Extraction jobs](/en/fetcher/fetcher-extraction-jobs) gives the full four-state lifecycle.

<Note>
  Turn on authentication with `PLUGIN_AUTH_ENABLED=true`. Requests then carry an `Authorization: Bearer <token>` header. This quickstart runs with authentication off.
</Note>

## Next steps

***

<CardGroup cols={2}>
  <Card title="Core concepts" icon="book" href="/en/fetcher/fetcher-core-concepts">
    Connections, schema discovery, jobs, filters, and results.
  </Card>

  <Card title="Configuration" icon="gear" href="/en/fetcher/fetcher-configuration">
    Every environment variable, per component.
  </Card>

  <Card title="Deployment" icon="server" href="/en/fetcher/fetcher-deployment">
    Dependencies, queues, scaling, and the fail-closed startup checks.
  </Card>

  <Card title="Security" icon="shield" href="/en/fetcher/fetcher-security">
    Key derivation, rotation, message signing, and host validation.
  </Card>
</CardGroup>