> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# What is Fetcher?
> Fetcher extracts data from PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB through one API. Run it as two services, or embed its engine in your own application.
export const GSourceAvailable = ({children}) =>
{children}
;
**Fetcher** reads data out of databases that your application does not own. It connects to PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB, discovers tables and fields, and extracts the rows you request. Standalone Fetcher supplies the HTTP API and connection storage. The embedded Engine is a Go API whose host supplies the connector registry and any needed connection or credential ports.
For generic extraction, Fetcher validates the selection against a cache-first schema snapshot, discovering live on a cache miss. Embedded direct mode returns plaintext JSON with a SHA-256 digest when no `ResultSink` is wired; the unkeyed digest detects accidental corruption, not tampering. Store-mode persistence and protection are host/sink-defined; the standalone Worker owns encryption, storage, and HMAC.
## Source-available
***
Fetcher is source-available under the [Elastic License 2.0](https://github.com/LerianStudio/fetcher/blob/main/LICENSE). The full source code lives on [GitHub](https://github.com/LerianStudio/fetcher), and every engine source file carries an `SPDX-License-Identifier: Elastic-2.0` header. Fetcher and Midaz are the two source-available Lerian products.
## Two ways to run it
***
The duality below is the shape of the product. Pick the row that matches your problem.
| Shape | What you deploy | When to pick it |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| **Standalone services** | A **Manager** (HTTP API) and a **Worker** (queue consumer), plus MongoDB, RabbitMQ, object storage, and Valkey/Redis. | You want a shared extraction service with a REST API, asynchronous jobs, and stored results. |
| **Embedded Engine** | Nothing. Your Go application imports `github.com/LerianStudio/fetcher/pkg/engine` and runs extraction in-process. | You want extraction inside a service you already operate, with no new infrastructure. |
The Engine is a separate Go module from the services module (`github.com/LerianStudio/fetcher/v2`). It currently declares **zero third-party dependencies**. CI enforces the dependency boundary by rejecting non-stdlib and non-module-local dependency graphs, not by checking syntactically for a `require` block. Your host application supplies the parts the Engine needs through small interfaces, called ports.
The Manager and the Worker use that same Engine for generic extraction. The Worker retains a separate compatibility path for MongoDB `plugin_crm` extraction.
The Engine ships with an in-memory harness (`pkg/engine/memory`) that covers the storage-facing ports: the connector registry, the connection store, the schema cache, the result sink, and the execution store. You can run a real extraction with no MongoDB, no RabbitMQ, and no object storage. The harness does not cover credential protection: encrypted persistence is off by default, and turning it on requires your host application to supply a `CredentialProtector`. See [Getting started](/en/fetcher/fetcher-getting-started).
## What Fetcher gives you
***
1. **Connection management.** Store, validate, and test database connections. Fetcher encrypts each password with AES-256-GCM before it writes the record.
2. **Schema discovery.** Fetcher detects tables, columns, and data types across all five database types, with a cache and an always-fresh read.
3. **Data extraction.** One query interface with field projection, ten filter operators, and multi-table, multi-schema, multi-datasource requests.
4. **Job orchestration.** Asynchronous processing with a 5-minute duplicate window, status tracking, and `job.completed` and `job.failed` events.
## Datasources
***
Fetcher accepts five datasource types. Send the identifier upper-case on the API — request validation matches the five values exactly, and `postgresql` is rejected with `400`. Operator-declared internal datasources (`DATASOURCE_{NAME}_TYPE`) and the `type` query filter on the connection list do accept any case.
| Type identifier | What is distinctive |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `POSTGRESQL` | Multi-schema. Fetcher parses `JSONB` and `JSON` columns into objects, arrays, or strings. Tables outside `public` come back schema-qualified. |
| `MYSQL` | Fetcher parses JSON columns the same way it does for PostgreSQL. |
| `ORACLE` | Owner namespaces. Tables owned by the connected user come back bare, and other owners come back as `OWNER.TABLE`. |
| `SQL_SERVER` | Multi-schema, with `dbo` as the default. Tables outside `dbo` come back schema-qualified. |
| `MONGODB` | Schemaless inference. Fetcher discovers the field set with an aggregation, then infers types from a document sample. |
## Safe by construction
***
* **Tenant scoping.** Every engine operation carries a tenant ID, and that ID is the only isolation boundary. A malformed tenant ID fails before Fetcher touches any resource.
* **Redacted errors.** Fetcher discards the raw driver error at its boundary, so a DSN, a credential, or a driver internal cannot reach a caller.
* **Fail-fast extraction.** The first failing step stops the run. Fetcher never returns a partial result.
* **Default limits.** Defaults are 10 datasources, 20 tables per datasource, 50 fields per table, 4 concurrent datasource workers, 5 minutes, and 256 MiB. Hosts configure the ceiling; a request can only lower it.
* **Host validation.** In multi-tenant mode, Fetcher rejects a tenant-supplied host that resolves to a private, loopback, or cloud-metadata address.
## Next steps
***
Concrete problems Fetcher solves, and what changes when you adopt it.
Two paths to a first extraction: zero infrastructure, or full Docker stack.
Connections, schema discovery, extraction jobs, filters, and results.
Master key, derived keys, message signing, and host validation.