# Canonical Client Onboarding

This guide shows a client how to publish canonical observations through the
public DataHub API. The source of truth for the API contract remains
[`README.md`](../README.md) and
[`api-gateway/openapi.yaml`](../api-gateway/openapi.yaml).

Share the DataHub-hosted onboarding or API reference link with external users,
not a repository path.

## Quick Start

1. Choose a stable `dataset_id`, for example `client.operating_metrics`.
2. Request `DATAHUB_API_KEY` and a `DATAHUB_WRITER_TOKEN`.
3. Register the dataset: `PUT /api/v1/canonical/datasets/{dataset_id}`.
4. Upsert entities: `POST /api/v1/canonical/datasets/{dataset_id}/entities`.
5. Upsert series: `POST /api/v1/canonical/datasets/{dataset_id}/series`.
6. Ingest points: `POST /api/v1/canonical/datasets/{dataset_id}/points:ingest`.
7. Validate reads through `/api/v1/timeseries/datasets`,
   `/api/v1/timeseries/selections/search`, `options`, and `data`.

Every write request needs both `x-api-key` and `x-datahub-writer-token`. Store
returned keys and do not send `validTo`, `sourceId`, or `value: null`.

## Public Surface

Use these public canonical write routes. Static-value replacement is global;
the remaining resource routes are dataset-scoped:

- `POST /api/v1/canonical/static-values:replace`
- `PUT /api/v1/canonical/datasets/{dataset_id}`
- `POST /api/v1/canonical/datasets/{dataset_id}/entities`
- `GET /api/v1/canonical/datasets/{dataset_id}/entities`
- `POST /api/v1/canonical/datasets/{dataset_id}/relationships`
- `GET /api/v1/canonical/datasets/{dataset_id}/relationships`
- `POST /api/v1/canonical/datasets/{dataset_id}/series`
- `GET /api/v1/canonical/datasets/{dataset_id}/series`
- `POST /api/v1/canonical/datasets/{dataset_id}/series:resolve`
- `POST /api/v1/canonical/datasets/{dataset_id}/points:ingest`

Request-local `clientRef`, `entityRef`, and `seriesRef` fields are not part of
the public write contract. Create entities and series first, then write points
by canonical `seriesKey`.

## Concepts

`dataset_id` identifies the public dataset surface, for example
`client.operating_metrics`. It can contain multiple metrics; individual
measures belong on each series as `metric`.

`entity` is the addressable subject for data, such as a company, app, domain,
ticker, product, region, or business unit.

For company data, resolve to a real `identity.company.<company_id>` when that
mapping is available. DataHub stores that as one shared global company entity,
and dataset series should point to it. If no real company can be resolved, keep
the entity dataset-local; private sources can use IDs such as
`private_company.<source>.<source_id>`.

`series` is a descriptor for one metric on one entity at one frequency with a
stable dimension set. DataHub returns a stable `seriesKey` for repeat writes.

`relationship` is an entity-to-entity descriptor such as publisher-to-app or
company-to-product. DataHub returns a stable `relationshipKey`; clients do not
submit one.

`point` is an observed value for a series on an observation `date`, with
`knownDate` representing the as-of date used for current/latest and PIT reads.

## Authentication

Every canonical write request needs both headers:

- `x-api-key: $DATAHUB_API_KEY`
- `x-datahub-writer-token: $DATAHUB_WRITER_TOKEN`

The writer token is separate from the read API key. It is scoped only to the
verbs needed by the client, not to a dataset:

- `entities:write`
- `entities:read`
- `datasets:write`
- `relationships:write`
- `relationships:read`
- `series:write`
- `series:read`
- `points:write`
- `static-values:write`

Use only the verbs required for the integration. Dataset registration requires
`datasets:write`. Entity, relationship, and series upsert routes require write
verbs. Recovery and descriptor resolve routes require read verbs. Point
ingestion requires `points:write`.

The static-value replacement route is global rather than dataset-scoped. It
requires `static-values:write` and only accepts existing canonical
`entityKey` values.

```bash
export BASE_URL="https://datahub-api-gateway-4uk36woq.ew.gateway.dev"
export DATAHUB_API_KEY="YOUR_API_KEY"
export DATAHUB_WRITER_TOKEN="YOUR_WRITER_TOKEN"
```

## Recommended Lifecycle

1. Request an API key and a writer token with the required verbs.
2. Register dataset metadata with temporal mode and read enablement.
3. Upsert or resolve an entity by stable natural identity.
4. Upsert or resolve any entity-to-entity relationship using existing
   `entityKey` values.
5. Upsert or resolve a series by entity plus metric descriptor.
6. Store returned `entityKey`, `relationshipKey`, and `seriesKey` values in the
   client.
7. Send repeat point batches using only `seriesKey`, `date`, `knownDate`,
   `value`, and optional point metadata such as `currency`.
8. Validate the written data through `/api/v1/timeseries/*`.

Read enablement is global publication. Once enabled, the dataset can appear in
`/api/v1/timeseries/datasets`, `/api/v1/timeseries/selections/search`,
`/api/v1/timeseries/options`, and `/api/v1/timeseries/data`.

## Replace Entity Static Values

Use the static-value route for slowly changing scalar properties that belong
to canonical entities rather than a timeseries. A request is the complete
current state for every field named in `fields`; omitting an existing
entity-field value closes it. Do not split one field replacement into chunks.

Each value must supply exactly one member: `textValue` or `numberValue` (the
only two static shapes). Writers may introduce a plain
field name such as `lead` without registering it first. Repeating an identical
request is a no-op and does not create another history row.

```bash
curl -sS -X POST \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/canonical/static-values:replace" \
  -d '{
    "fields": ["profile.headquarters_country", "profile.employee_count"],
    "values": [
      {
        "entityKey": "entity:identity.company:company:identity.company.123",
        "field": "profile.headquarters_country",
        "textValue": "United Kingdom"
      },
      {
        "entityKey": "entity:identity.company:company:identity.company.123",
        "field": "profile.employee_count",
        "numberValue": 1000
      }
    ]
  }'
```

The response reports only the transition counts:

```json
{"inserted": 2, "closed": 0, "unchanged": 0}
```

## Options Metadata

Options metadata is presentation, not permission — there is no list of legal
dimension names. On a read, every param that is not a reserved control param
(`entity_id`, `entity_type`, `metric`, `frequency`, `start_date`, `end_date`,
`vintage_date`) is a dimension filter. A dimension the read does not state is
not constrained: it reads every stored value and sums them, the same as `*`. A
stated key no series carries describes a shape nothing has and the read returns
no rows rather than an error — build requests from an options response. When a
read comes back empty, `matchedSeries` in the response metadata says which fault
it is: `0` means the requested shape matched no series, while a non-zero count
means the shape is right and the window simply holds no points. Options and
search derive the keys they offer from `canonical.series.dimensions`, merged
under the declared rows.

The one thing a dimension declares is whether it is **additive** — whether its
values partition what they measure, so that summing across them gives the whole
(`channel`, `country_code`). A key with no row, or `additive` NULL, is additive:
multi-select, and an omitted or `*` value reads everything. `additive = false`
says the values are alternative views of the same thing (CE `type` is three
panel methodologies over the same transactions; mscience `company` lists a
parent beside its own parts): the key is single-select with no All, and a read
whose matched series span several of its values is refused
(`Params do not aggregate and must name one value: type`) instead of summed.
Cardinality, transport, the UI control and searchability are derived from that
bit and the param's `kind`; they are not stored. Beyond additivity a row may
carry a label, a description, hidden visibility, a default or a dependency.

Canonical writes store durable identity and measurement descriptors. They do
not require every selector value to be human-readable. When a series dimension
uses a stable source ID, keep that ID in `dimensions` and add the display label
through `canonical.dataset_read_options_metadata`.

Use value-level options metadata for labels that should appear in
`/api/v1/timeseries/options` without changing canonical identity. For example,
SensorTower stores the stable vendor app ID as the dimension `app_id`; the
dropdown label for that ID belongs in
`canonical.dataset_read_options_metadata`, not in `canonical.series.dimensions`
and not by replacing the ID with a display name.

```sql
INSERT INTO canonical.dataset_read_options_metadata (
  dataset_id,
  param_key,
  option_value,
  label
)
VALUES (
  'sensortower.entity_metric',
  'app_id',
  '5eaef95d5c6ace6c5a733828',
  '1v1.LOL - Battle Royale Game'
)
ON CONFLICT (dataset_id, param_key, option_value)
WHERE option_value IS NOT NULL
DO UPDATE SET
  label = EXCLUDED.label,
  updated_at = now();
```

`dataset_id` is the canonical dataset, `param_key` is the request/option
parameter, `option_value` is the exact stored selector value, and `label` is the
human-facing text. Generic canonical reads return the stored value as the
selector and enrich the option with the label when metadata is present.

Rows with `option_value IS NULL` describe a parameter itself; rows with
`option_value IS NOT NULL` describe one concrete option value. Producers should
write labels from authoritative source metadata and keep them in sync during
backfills or incremental refreshes. External client integrations should
coordinate options metadata with the DataHub team; it is database-side read
metadata, not a public canonical write endpoint.

## Register A Dataset

```bash
cat > dataset.json <<'JSON'
{
  "label": "Client Operating Metrics",
  "description": "Client supplied operating metrics",
  "readEnabled": true
}
JSON

curl -sS -X PUT \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics" \
  --data @dataset.json
```

Registration is idempotent. It creates or updates the dataset metadata row
used by generic canonical reads and by write validation. It does not create
entities, series, relationships, or points.

## Upsert An Entity

```bash
cat > entity.json <<'JSON'
{
  "entityType": "company",
  "entityId": "ACME",
  "displayName": "ACME Corp",
  "metadata": {
    "country": "US"
  }
}
JSON

curl -sS -X POST \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/entities" \
  --data @entity.json
```

Response:

```json
{
  "datasetId": "client.operating_metrics",
  "entityKey": "entity:client.operating_metrics:company:ACME",
  "entityType": "company",
  "entityId": "ACME",
  "displayName": "ACME Corp",
  "sourceParams": {},
  "metadata": {"country": "US"},
  "created": true
}
```

Entity upsert is deterministic by `(datasetId, entityType, entityId)`.
Repeating the same natural identity returns the same `entityKey` and updates
mutable fields such as `displayName`, `sourceParams`, and allowed `metadata`.
Clients may provide `entityKey`; DataHub derives the expected key and rejects a
mismatch before mutation.

## Recover Entities

Exact lookup:

```bash
curl -sS \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/entities?entityType=company&entityId=ACME"
```

Paginated recovery:

```bash
curl -sS \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/entities?entityType=company&limit=100"
```

Responses are ordered by stable `entityKey` and include `nextPageToken` when
additional rows are available.

## Upsert A Relationship

Relationships connect two existing canonical entities. The endpoint does not
create parent or child entities; upsert the entities first and pass their
returned keys.

```bash
cat > relationship.json <<'JSON'
{
  "relationshipType": "owns",
  "parentEntityKey": "entity:client.operating_metrics:company:ACME",
  "childEntityKey": "entity:client.operating_metrics:product:WIDGET",
  "validFrom": "2026-01-01",
  "metadata": {
    "source": "client_upload"
  }
}
JSON

curl -sS -X POST \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/relationships" \
  --data @relationship.json
```

Response:

```json
{
  "datasetId": "client.operating_metrics",
  "relationshipKey": "relationship:client.operating_metrics:abc123...",
  "relationshipType": "owns",
  "parentEntityKey": "entity:client.operating_metrics:company:ACME",
  "childEntityKey": "entity:client.operating_metrics:product:WIDGET",
  "validFrom": "2026-01-01",
  "metadata": {"source": "client_upload"},
  "created": true
}
```

DataHub derives `relationshipKey` from `(datasetId, relationshipType,
parentEntityKey, childEntityKey, validFrom)`. Do not send `relationshipKey`,
`validTo`, `parentEntityRef`, `childEntityRef`, nested entities, nested series,
or `datasetId` in the body.

## Recover Relationships

Known key lookup:

```bash
curl -sS \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/relationships?relationshipKey=relationship%3Aclient.operating_metrics%3Aabc123"
```

Descriptor lookup:

```bash
curl -sS \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/relationships?relationshipType=owns&parentEntityKey=entity%3Aclient.operating_metrics%3Acompany%3AACME&childEntityKey=entity%3Aclient.operating_metrics%3Aproduct%3AWIDGET"
```

Relationship list responses are ordered by stable `relationshipKey` and include
`nextPageToken` when additional rows are available.

## Upsert A Series

```bash
cat > series.json <<'JSON'
{
  "entityKey": "entity:client.operating_metrics:company:ACME",
  "metric": "revenue",
  "frequency": "monthly",
  "dimensions": {
    "region": "US"
  }
}
JSON

curl -sS -X POST \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/series" \
  --data @series.json
```

Response:

```json
{
  "datasetId": "client.operating_metrics",
  "entityKey": "entity:client.operating_metrics:company:ACME",
  "seriesKey": "series:client.operating_metrics:abc123...",
  "metric": "revenue",
  "frequency": "monthly",
  "dimensions": {"region": "US"},
  "created": true
}
```

Series upsert is deterministic by `(datasetId, entityKey, metric, frequency,
dimensions)`. Dimension canonicalization is server-owned. Clients may also send
`entityType` and `entityId` instead of `entityKey`; `/series` resolves an
existing entity but does not create a missing entity as a side effect. Clients
may provide `seriesKey`; DataHub derives the expected key and rejects a
mismatch before mutation. Currency belongs on point rows only, not series.

## Recover Series

Known key lookup:

```bash
curl -sS \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/series?seriesKey=series%3Aclient.operating_metrics%3Aabc123"
```

Paginated list for an entity:

```bash
curl -sS \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/series?entityKey=entity%3Aclient.operating_metrics%3Acompany%3AACME&limit=100"
```

Descriptor resolve:

```bash
curl -sS -X POST \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/series:resolve" \
  --data @series.json
```

Series list responses are ordered by stable `seriesKey` and include
`nextPageToken` when additional rows are available.

## Ingest Points

Committed point batches require `idempotencyKey`. Use a stable key for a
logical upload, such as a source file checksum, export run ID, or date
partition. Retry the exact same body with the same key if the client times out.

```bash
cat > points.json <<'JSON'
{
  "idempotencyKey": "client-operating-metrics-2026-01-v1",
  "dryRun": false,
  "points": [
    {
      "seriesKey": "series:client.operating_metrics:abc123...",
      "date": "2026-01-31",
      "knownDate": "2026-02-02",
      "value": 123.4,
      "currency": "USD"
    }
  ]
}
JSON

curl -sS -X POST \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "x-datahub-writer-token: $DATAHUB_WRITER_TOKEN" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/canonical/datasets/client.operating_metrics/points:ingest" \
  --data @points.json
```

Response:

```json
{
  "datasetId": "client.operating_metrics",
  "dryRun": false,
  "entitiesProcessed": 0,
  "seriesProcessed": 0,
  "relationshipsProcessed": 0,
  "pointsProcessed": 1,
  "idempotencyStatus": "created"
}
```

Point rows must not include `entityKey`, `clientRef`, `entityRef`, or
`seriesRef`. Do not send `validTo` or `value: null`; DataHub derives SCD-2
windows from `knownDate`. A point batch must not repeat the same `seriesKey`,
`date`, and `knownDate` tuple.

Malformed stateful resource bodies, unknown fields, missing required fields,
and empty point batches return FastAPI `422` validation details. Descriptor
conflicts, unknown series keys, idempotency conflicts, and authorization
failures return the explicit route-level status codes documented in the API
Gateway spec.

Idempotency behavior:

- Same `idempotencyKey` and same request body returns
  `idempotencyStatus: "replayed"`.
- Same `idempotencyKey` and different request body returns `409`.
- `dryRun=true` validates existing `seriesKey` rows without writing points and
  returns `idempotencyStatus: "validated"`.

## Client-Supplied Keys

DataHub remains the source of truth for canonical keys. Long-running producers
that already compute deterministic descriptor keys may include `entityKey` or
`seriesKey` during resource upsert as a guardrail. DataHub still derives the
expected key from the descriptor and rejects any mismatch.

Treat returned keys as stable opaque identifiers. If a client loses local
storage, recover state through the entity, relationship, and series recovery
endpoints instead of rebuilding request-local labels.

## Metadata Controls

Client-controlled JSON fields use generic DataHub safety checks. Sensitive key
names such as `token`, `secret`, `password`, `authorization`, `apiKey`,
`privateKey`, `access_key`, `credentials`, `headers`, and `cookie` are rejected
recursively. Metadata, source params, dimensions, and nested values are bounded
by size, depth, and item-count limits.
List responses are intended for state recovery and should not be used to store
secrets.

## Fetch Written Data

After points are written and the dataset is read-enabled, clients fetch through
the normal timeseries contract:

```bash
curl -sS -X POST \
  -H "x-api-key: $DATAHUB_API_KEY" \
  -H "Content-Type: application/json" \
  "$BASE_URL/api/v1/timeseries/data" \
  -d '{
    "start_date": "2026-01-01",
    "end_date": "2026-01-31",
    "series": [
      {
        "seriesId": "acme-revenue",
        "datasetId": "client.operating_metrics",
        "params": {
          "entity_type": "company",
          "entity_id": "ACME",
          "metric": "revenue",
          "frequency": "monthly",
          "region": "US"
        }
      }
    ]
  }'
```

Current/latest reads are the default. Add `as_of` in series
params to select the revision available at that cutoff. Inputs use `known_date`
when present, otherwise their observation date. Later-known values never fill
historical gaps. Targets require a publication date, with no observation-date proxy.

## Removed Graph Batch

The older request-local graph-batch workflow is no longer part of the public
API. Use first-class entity, relationship, series, and point routes.