Skip to main content
Data Feeds delivers each dataset, one set of official records such as a county’s building permits, as numbered versions. Each time Ploti reads a new state of the source, it publishes a new version. A version has:
  • a snapshot: the whole dataset as of that version
  • a changes file: what changed since previousVersion, unless the version has resyncRequired
  • its schema and a manifest with every file’s size and checksum
  • an id map, only when the version replaced record ids (recordIdsChanged)
  • its attachments: the files Ploti holds for the dataset, such as an ordinance’s PDF, a scanned zoning map or a source’s archived copy
A version appears only once it has been built and released, meaning sent to subscriptions. That holds for GET /v1/feeds/datasets/{dataset}/versions, GET /v1/feeds/datasets/{dataset}/versions/{version} and the dataset’s current, so pulling versions and following events see the same chain. A version’s publishedAt is when it was released, and the updatedSince filter on GET /v1/feeds/datasets reads the same time. A version held for review never appears.

The version chain

Version numbers increase but can skip. A held version that was discarded or replaced by a newer one leaves a gap, and so does a version rebuilt under a new number. Each version names the version its changes apply to in previousVersion. Always follow that chain, not the numbers:
  • Apply a version’s changes only when its previousVersion is the version you hold.
  • When it is any other version, reload that dataset from its current version: read current.version from GET /v1/feeds/datasets/{dataset}, then load that version’s snapshot.
sourceUrl and fetchedAt can be null. counts is null when resyncRequired is true.

Versions released again

Ploti sometimes releases a version of a dataset again, and rebuilds it under a new number. The old number then answers 410 expired, with a message that says so. If subscribers already had the old version, the dataset’s next version sent is a resync_required, even when a newer release arrives before the rebuild. Its reason is previous_unavailable, with these exceptions:
  • schema_changed or key_columns_changed when it also changes the schema
  • record_ids_changed when it also changes ids you hold
Until that version is sent, the dataset’s current can be the version before the one released again. That version’s records are matched to the key values and content Ploti kept for the version released again (see how records are matched), and its previousVersion names that version. A record it matches keeps the id you hold. When it kept records but none could be compared, the version is held for review rather than sent. If the version released again replaced record ids and had not been sent yet, its rebuild replaces them again and carries its own id map.

Getting a version’s files

GET /v1/feeds/datasets/{dataset}/versions/{version} returns the version with signed links to its files under files:
  • snapshot and changes: GeoParquet parts of about 256 MB at most
  • idMap: Parquet parts, empty unless the version replaced record ids
  • schema, manifest and attachments
Links last one hour and expire at files.expiresAt. Ask again for fresh ones at any time, including part way through a large download. Check each file’s sha256.

Attachments

files.attachments lists the files Ploti holds for the dataset as of the version. Each is named by file, the same name GET /v1/datasets/{dataset}/files/{file} takes:
  • source:<id>: the archived copy of one source in the dataset’s provenance, with its id from GET /v1/datasets/{dataset}/provenance
  • original: the document or map file Ploti downloaded for the dataset
  • document:<sha256>: a file stored per row, such as a permit’s PDF. The row’s cell holds this name, so join a row to its file by the cell’s value. Several rows naming one document share one attachment.
sha256 is the checksum of the bytes you download.
  • Where Ploti’s record of a file names a checksum, Ploti checked the copy against it before delivering it. A file whose copy did not match, or that Ploti could not copy, is left out of that version.
  • The same file in several versions has the same sha256, so you only need to download it again when the checksum changes.
  • A dataset with no stored files has no attachments.
  • A version’s manifest.json lists them under attachments, without a path.
POST /v1/feeds/datasets/{dataset}/versions/{version}/links signs fresh links to the files you name, without reading the whole version again:
Name an attachment by its file, and any other file by its path in the version’s manifest.json, up to 200 files at a time. The answer has:
  • files: each file as {"file", "filename", "contentType", "sizeBytes", "sha256", "url"}, with expiresAt
  • missing: the names the version has no file for, or none left past retention
It is refused like GET /v1/feeds/datasets/{dataset}/versions/{version}:
  • 409 not_subscribed for a dataset none of your subscriptions delivers
  • 404 not_found for a version that was never released
  • 410 expired for one released again and rebuilt under a new number
Links are signed only for a dataset one of the workspace’s subscriptions delivers. For any other dataset the answer is 409 not_subscribed, so once a dataset leaves your subscriptions, or you delete the subscription that delivered it, no new links are issued for it. Links already issued keep working until they expire, up to an hour.

Envelope columns

Every snapshot and changes file carries these columns ahead of the dataset’s own: Changes files add these columns: See applying changes. Because every dataset carries the same envelope, you can load many places’ datasets side by side and ask what changed since Tuesday without knowing any dataset’s columns.

Applying changes

A changes file holds one row per change, with the envelope columns and the record as of this version: Apply every upsert as insert-or-replace by _ploti_record_id, in keyed and unkeyed datasets alike. A delete row carries only the envelope columns; its record columns are null. A changed row also carries:
  • _ploti_changed_columns: the columns whose values differ from the version before, compared over the columns both versions have. A column added by a MINOR schema change is not compared: the row carries its value, and records that did not change get it when you reload.
  • _ploti_match: how the record was matched to the version before, key (the same key value), content (otherwise identical content) or agent (Ploti’s matching agent judged it the same record, edited or renumbered).
  • _ploti_match_score: on an agent match, the agent’s confidence, from 0.85 to 1; otherwise null.
A column holding one value in every row of both versions, such as the time the source was read, is not compared either: a version that only moves that value has no changes, and its snapshot carries the new value. Every delete carries _ploti_delete_reason removed_at_source: the record was not matched to any record in this version. Before a record is deleted, Ploti tries to match it:
  • A record that no key or identical content matched goes to Ploti’s matching agent when a record of the new version is a plausible partner: it shares values with it, or lies within about 50 m of it. It keeps its id when the agent judges them the same record.
  • Otherwise it is deleted. An edited record the agent could not match arrives as a delete of the old copy and an added row for the new one.
  • At most 200 records a version are judged one by one, chosen among the first 20,000 left over, and none when more than 250,000 are left over. The rest of a large change arrives as deletes and adds.
  • A version waits for the agent’s matching before it is released, up to 24 hours for each ask. The agent is asked at most twice, so a version can wait up to 48 hours before it is released or held for review.
See the hard cases for what an unmatched record, a merge or a split looks like in a changes file. Apply versions in chain order, and skip any version at or below the one you already hold. After you reload a dataset to a version, skip its updated, resync_required and schema_changed events at or below that version. Only loading a snapshot or applying a changes file changes the version you hold. That makes every change safe to apply twice.

When to reload instead

A version with resyncRequired: true has no changes files, so you reload the dataset from its snapshot. A version is one when:
  • it is the dataset’s first version
  • it makes a MAJOR schema change: a column removed or retyped, or the key changed or dropped
  • the snapshot of the version before it is past retention
  • a version subscribers already had was released again
  • it replaced record ids
A MINOR schema change (an added column, or a key pinned for the first time) still has changes to apply. Reload the dataset from its snapshot when:
  • the version has resyncRequired: true
  • you receive feed.dataset.resync_required
  • the version you need answers 410: it is past retention, or was released again and rebuilt
  • an updated names a previousVersion other than the one you hold

When record ids are replaced

Record ids carry across versions, so this is rare. A version that changes ids you hold:
  • has resyncRequired: true and recordIdsChanged: true
  • arrives as feed.dataset.resync_required with reason record_ids_changed
  • lists, in its files.idMap, _ploti_previous_record_id, _ploti_record_id and _ploti_dataset_id for every record carried from the version you hold
Ploti sends one when its staff replace a dataset’s ids, and also when Ploti’s matching agent found that a record you hold under one id continues one under another. So it can change a few ids or all of them, and many of the map’s rows can map an id to itself. Reload the dataset from the snapshot as usual, and rewrite any ids your own systems store with the map. A record not in the map is new in that version. An old id not in the map belongs to a record that version no longer has. See when ids are replaced for a worked example.

Retention

File retention counts from when a version was built, and event retention from when each event was sent. A version can be released some time after it was built (one held for review, for example), so its files can expire before its event does, and a gap shorter than 90 days can still need a reload. Plan on a version past retention answering 410 expired, unless it is the dataset’s newest, or the last of its MAJOR schema and has no changes to apply. On a 410:
  • From a version: reload that dataset from its current version. Read current.version from GET /v1/feeds/datasets/{dataset}, load that version’s snapshot, and continue from there.
  • From the current version too: its files are being rebuilt. Try again in a few minutes.
  • From a cursor whose events are gone: reload from the subscription’s manifest.