setting-up-a-custom-rest-source

Before drafting anything, call external-data-sources-wizard to list the native source types and check whether the target API is among them, matching on the service name. A Custom source is the fallback for APIs with no native connector; do not skip this check on the assumption that a well-known API isn't supported — most are.

If a native connector matches, stop and tell the user the built-in path is simpler and battle-tested (it handles auth, pagination, schema, and incremental sync for you), and hand off to setting-up-a-data-warehouse-source. Only continue with this skill when the user has no matching native connector, or explicitly wants to exercise the custom REST path despite one existing.

Get either a docs URL (fetch it and read the auth scheme, the list endpoints, their response envelopes, and any pagination) or a natural-language description of the endpoints. You need, per resource you'll import:

• the path (relative to a common base_url) and method (GET, or POST for query-style read endpoints),
• the auth scheme (bearer token / API key in header or query / HTTP basic / OAuth2 with a customer-owned client — client_credentials or a pre-obtained refresh token; the interactive authorization_code flow is not supported),
• the record path — where the array of records sits in the JSON response (e.g. data, results, items),
• how the API paginates (next-URL, link header, cursor, offset, page number, or single page),
• a primary key field, and
• optionally an incremental cursor field (updated_at-style) so re-syncs only fetch new/changed rows.

Ask the user for the credential value, but tell them you'll only ever place it in the auth_* payload key, never in the manifest.

Author the RESTAPIConfig from references/manifest-reference.md. Match the auth block to the scheme, pick the paginator that matches the docs, set data_selector to the record path, and add an incremental block when the API has an updated_at-style cursor and a matching query param. For an endpoint whose rows must be fetched per parent (e.g. /forms/{form_id}/responses), use a parent/child resolve param — see the fan-out example. Keep it to one level of nesting.

Call external-data-sources-db-schema with { source_type: "Custom", manifest_json: "<stringified manifest>", auth_token: "<credential>" }. The credential key is not literally auth_* — use the one for your auth type: auth_token (bearer), auth_api_key (api_key), auth_password (http_basic), or auth_oauth2_client_secret (+ auth_oauth2_refresh_token for the refresh-token grant). It validates the manifest structure, the fan-out graph, and the credential (a bounded live probe), then returns one table entry per resource with detected_primary_keys and incremental_fields. If it returns a 400, the message is plain English (e.g. resources[0].endpoint.path: must not be empty) — fix the manifest and retry. Loop here until it validates.

For each resource, call external-data-sources-preview-resource with { source_type: "Custom", payload: { manifest_json, auth_token }, resource_name: "<name>", limit: 10 } (the auth_token key varies by auth type, as in Step 3). It returns up to limit real rows plus the inferred columns. Check that:

• data_selector is right — rows are the records you expect, not a wrapper object. If rows looks like [{ "data": [...] }] you pointed at the envelope, not the array; fix data_selector.
• primary_key exists in the rows and is unique.
• the incremental cursor_path field is present in the rows and looks like a sortable timestamp/id.

A live failure (unreachable host, auth rejected) comes back as error with empty rows — fix credentials or the URL and retry. Iterate Steps 2–4 until the sample looks right.

Call data-warehouse-source-setup with { source_type: "Custom", payload: { manifest_json, auth_token }, prefix: "<short_name>" } (the auth_token key varies by auth type, as in Step 3). It enables every resource in the manifest with sensible sync defaults (incremental where the manifest declares a cursor, else full refresh) and creates the source. If the user only wants a subset of resources, use external-data-sources-create with a schemas array instead (see setting-up-a-data-warehouse-source for the schemas shape). Pick a short lowercase prefix — tables become {prefix}_{resource_name} in HogQL.

After creation, call external-data-schemas-list to show the user the initial sync status, and tell them how to query: SELECT * FROM {prefix}_{resource_name} LIMIT 10.