What is Fetcher?

Fetcher extracts data from PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB through one API. Run it as two services, or embed its engine in your own application.

Fetcher reads data out of databases that your application does not own. It connects to PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB, discovers tables and fields, and extracts the rows you request. Standalone Fetcher supplies the HTTP API and connection storage. The embedded Engine is a Go API whose host supplies the connector registry and any needed connection or credential ports.

For generic extraction, Fetcher validates the selection against a cache-first schema snapshot, discovering live on a cache miss. Embedded direct mode returns plaintext JSON with a SHA-256 digest when no ResultSink is wired; the unkeyed digest detects accidental corruption, not tampering. Store-mode persistence and protection are host/sink-defined; the standalone Worker owns encryption, storage, and HMAC.

Source-available

---

Fetcher is source-available under the Elastic License 2.0. The full source code lives on GitHub, and every engine source file carries an SPDX-License-Identifier: Elastic-2.0 header. Fetcher and Midaz are the two Lerian products.

Two ways to run it

---

The duality below is the shape of the product. Pick the row that matches your problem.

Shape	What you deploy	When to pick it
Standalone services	A Manager (HTTP API) and a Worker (queue consumer), plus MongoDB, RabbitMQ, object storage, and Valkey/Redis.	You want a shared extraction service with a REST API, asynchronous jobs, and stored results.
Embedded Engine	Nothing. Your Go application imports github.com/LerianStudio/fetcher/pkg/engine and runs extraction in-process.	You want extraction inside a service you already operate, with no new infrastructure.

The Engine is a separate Go module from the services module (github.com/LerianStudio/fetcher/v2). It currently declares zero third-party dependencies. CI enforces the dependency boundary by rejecting non-stdlib and non-module-local dependency graphs, not by checking syntactically for a require block. Your host application supplies the parts the Engine needs through small interfaces, called ports.

The Manager and the Worker use that same Engine for generic extraction. The Worker retains a separate compatibility path for MongoDB plugin_crm extraction.

The Engine ships with an in-memory harness (pkg/engine/memory) that covers the storage-facing ports: the connector registry, the connection store, the schema cache, the result sink, and the execution store. You can run a real extraction with no MongoDB, no RabbitMQ, and no object storage. The harness does not cover credential protection: encrypted persistence is off by default, and turning it on requires your host application to supply a CredentialProtector. See Getting started.

What Fetcher gives you

---

1. Connection management. Store, validate, and test database connections. Fetcher encrypts each password with AES-256-GCM before it writes the record.
2. Schema discovery. Fetcher detects tables, columns, and data types across all five database types, with a cache and an always-fresh read.
3. Data extraction. One query interface with field projection, ten filter operators, and multi-table, multi-schema, multi-datasource requests.
4. Job orchestration. Asynchronous processing with a 5-minute duplicate window, status tracking, and job.completed and job.failed events.

Datasources

---

Fetcher accepts five datasource types. Send the identifier upper-case on the API — request validation matches the five values exactly, and postgresql is rejected with 400. Operator-declared internal datasources (DATASOURCE_{NAME}_TYPE) and the type query filter on the connection list do accept any case.

Type identifier	What is distinctive
POSTGRESQL	Multi-schema. Fetcher parses JSONB and JSON columns into objects, arrays, or strings. Tables outside public come back schema-qualified.
MYSQL	Fetcher parses JSON columns the same way it does for PostgreSQL.
ORACLE	Owner namespaces. Tables owned by the connected user come back bare, and other owners come back as OWNER.TABLE.
SQL_SERVER	Multi-schema, with dbo as the default. Tables outside dbo come back schema-qualified.
MONGODB	Schemaless inference. Fetcher discovers the field set with an aggregation, then infers types from a document sample.

Safe by construction

---

• Tenant scoping. Every engine operation carries a tenant ID, and that ID is the only isolation boundary. A malformed tenant ID fails before Fetcher touches any resource.
• Redacted errors. Fetcher discards the raw driver error at its boundary, so a DSN, a credential, or a driver internal cannot reach a caller.
• Fail-fast extraction. The first failing step stops the run. Fetcher never returns a partial result.
• Default limits. Defaults are 10 datasources, 20 tables per datasource, 50 fields per table, 4 concurrent datasource workers, 5 minutes, and 256 MiB. Hosts configure the ceiling; a request can only lower it.
• Host validation. In multi-tenant mode, Fetcher rejects a tenant-supplied host that resolves to a private, loopback, or cloud-metadata address.

Next steps

---

Use cases

Concrete problems Fetcher solves, and what changes when you adopt it.

Getting started

Two paths to a first extraction: zero infrastructure, or full Docker stack.

Core concepts

Connections, schema discovery, extraction jobs, filters, and results.

Security

Master key, derived keys, message signing, and host validation.