ploti feeds sync downloads every dataset your Data Feeds subscription delivers into a folder, then keeps that folder current. Data Feeds is Ploti’s subscription for keeping a copy of its official county and city records in your own systems. Run sync again, by hand, from cron or when Ploti tells you something changed, and it downloads only the datasets that have a new version. When nothing changed, it changes nothing.
- A subscription is your standing order with Ploti: the places you serve, such as Orange County, Florida, and optionally the topics you want, such as permits or zoning. Its id looks like
sub_8f3k2m. Datasets Ploti adds to those places later join it on their own. - A dataset is one set of public records, such as a county’s building permits.
- A version is a numbered release of a dataset. Ploti makes a new one each time the county publishes new or changed records.
- A snapshot is the whole dataset as of one version, delivered as GeoParquet files: Parquet tables with a geometry column, which DuckDB, GeoPandas, QGIS and most warehouses read directly.
Before you start
Sync needs the Data Feeds permission, which bothploti login and every new API key hold. On your own machine, ploti login is enough. On a server, give the CLI an API key.
1
Create the key
In Ploti, a workspace owner or admin opens Settings, API keys and creates a key. Every new key can read the workspace’s subscriptions.A key belongs to the member who created it and stops working when that member leaves the workspace, so create it from an account that will stay, such as a shared service account.
2
Give it to the CLI
npx @ploti/cli, or install it with npm install -g @ploti/cli to get the ploti command.3
Find your subscription's id
Run it
The first run downloads everything. It prints a line per dataset as it finishes, then a summary:What ends up in the folder
ploti-data/ds_orange_fl_building_permits/snapshot/*.parquet and the path never has to change:
_ploti_record_id, an id that stays with the record from version to version, and _ploti_version, the version it came from. Envelope columns lists every column Ploti adds.
Files are named as the version’s manifest.json names them, so a path on disk is the same path Ploti’s API uses.
Attachments
Some datasets come with files: the archived copy of each source Ploti read the records from, and the document or map file it downloaded, such as a zoning ordinance’s PDF. A dataset that keeps a file per record, such as a permit’s scanned application, names it in the record’s cell asdocument:<sha256>. These are the dataset’s attachments.
Sync downloads each one into the dataset’s attachments/ folder, named by its sha256 checksum, and checks the bytes against it. Two records citing the same document share one file, and a file already on disk is never downloaded again, even when the dataset moves to a new version. .ploti-sync.json lists each attachment under its dataset with the name Ploti gives it (original, source:<id> or document:<sha256>), its original filename and its path on disk, so you can join a record’s document:<sha256> cell to its file.
An attachment that a new version no longer lists stays on disk, marked "listed": false in the state file, until you run sync with --prune.
Options
Each folder keeps one subscription. Sync refuses to write a second subscription into a folder that already holds another.
Run it on a schedule
Sync is safe to run as often as you like, since a run with nothing new downloads nothing.From cron
Run it every hour, or every night:When Ploti tells you
A webhook on the subscription has Ploti POST each event to you as it happens, so your copy updates within moments of a new version instead of on the next cron run.PLOTI_WEBHOOK_SECRET beside the handler.
The handler verifies the POST, answers at once, and runs the sync in the background. The event itself only says that something changed: ploti feeds sync brings every dataset up to its current version, so a missed or repeated POST does no harm.
ploti webhooks test <webhook_id> sends a signed ping to check the handler. Keep the cron job as well, at a slower rate: if the handler is down when a version lands, the next scheduled run still catches up.
Two runs on the same folder take turns: the second waits for the first, so a run started by an event never misses the version that event announced.
How sync keeps your copy whole
- No half versions. A new version downloads into
.ploti-sync/inside the folder, gets checked, and then replaces the dataset’s folder in one step. A reader, or a crash part way, never finds a folder holding half of one version and half of another. - Checked bytes. Every file is checked against the sha256 checksum Ploti gives for it, and downloaded again when it does not match.
- Links that run out. Download links last an hour. When one runs out during a long run, sync asks Ploti for a fresh link to the same file of the same version and carries on.
- Datasets that leave. A dataset that leaves the subscription, because you changed its places or your plan no longer covers it, keeps its folder and is marked
removedin the state file.--prunedeletes it. - Datasets not ready yet. A dataset whose first version is still being built, or whose files are being rebuilt, is listed as
waitingwith the reason. What is on disk stays until its next version is ready.
What sync does not do
- It loads snapshots, not changes. Each new version replaces the dataset’s folder with the full snapshot. If you load records into a database of your own and want only what was added, changed and removed, apply each version’s changes file instead: read the events after the
eventCursorthat.ploti-sync.jsonrecords, withploti feeds subscriptions events, as Run the loop yourself shows. - It writes files, not tables. Sync writes to a folder. To load a database, point your loader at the folder after each run.
- It syncs the whole subscription. To sync fewer datasets, narrow the subscription itself.
From Python
Ploti’s Python SDK has the same command, and leaves the same folder and state file:PLOTI_API_KEY too, and takes dry_run=True and prune=True for the options above. It returns the summary --raw prints, as a dict.
Run the loop yourself
To apply change files to your own tables, or to load somewhere the sync does not write, run the same loop with the REST API. Create the key and the subscription as above (see Subscriptions), then:Load the snapshots
The manifest first brings the subscription’s datasets up to date, then lists every one it still delivers that has a released version and that your workspace’s coverage and plan still open, with links to that version’s snapshot, and the event cursor it is current as of.- Download each snapshot’s parts (GeoParquet) and load them into a table keyed on
_ploti_record_id. - Record each dataset’s
versionas the version you hold. - Save
eventCursor: it is the last event already reflected in these snapshots.
expiresAt, an hour after the manifest was made. For fresh links to the same versions, get the manifest again with ?asOf=<eventCursor>, or get each version from GET /v1/feeds/datasets/{dataset}/versions/{version}. attachments lists the files the dataset’s records cite, as on a version.
Two kinds of dataset are listed under unavailable instead:
- Files being rebuilt, with reason
rebuilding. Ask for the manifest again in a few minutes to load it. Its events aftereventCursorstill apply. - No released version yet, with no
versionand itsstatusas the reason:building,held,failedorawaiting_release. Its first version arrives asfeed.dataset.resync_required.
503 plan_unavailable means Ploti could not read your workspace’s plan just now: try again in a minute.
Apply events
Read the events after your cursor, apply them in order, and savenextCursor once a page is applied. Only loading a snapshot or applying a changes file changes the version you hold.
- Skip an
updated,resync_requiredorschema_changedfor a version at or below the one you hold. - Act on a
schema_changedby changing your table to match, without recording its version as held. It names the same version as theupdated(MINOR change) orresync_required(MAJOR change) that follows it. - Apply an
updatedonly when itspreviousVersionis the version you hold.
previousVersion is any other version, reload that dataset from its current version instead: read current.version from GET /v1/feeds/datasets/{dataset}, then load that version’s snapshot from GET /v1/feeds/datasets/{dataset}/versions/{version}. You now hold that version, so skip the dataset’s later events at or below it.
Each version’s links come back signed for an hour from
GET /v1/feeds/datasets/{dataset}/versions/{version}, which data.links.version names. It signs links only for a dataset one of the workspace’s subscriptions delivers. For any other, it answers 409 not_subscribed.
A 409 (not_subscribed, outside_coverage or not_deliverable) or a 404 while you apply older events means the dataset has since left your subscriptions. Skip that dataset’s events up to its feed.dataset.removed.
A version past retention answers 410 expired, and so does a version Ploti released again and rebuilt under a new number. Reload that dataset from its current version, as above.
Add a webhook for the subscription’s events, as above, and use it as the signal to read the log. The log stays the source of truth. See Events and webhooks.