Ctrlk
Service statusLog in
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Create a Sync from SFTP to a Batch Catalog

If you're interested to sync events and attributes via SFTP, contact your CSM.

Create a Sync from SFTP to a Batch Catalog

Before you start

To create an SFTP → Batch sync, you'll need:

  • Access to the Batch dashboard

  • An SFTP server Batch can reach, and a user with read access to the synced files

  • Credentials to connect: either a password or a private key

  • An existing Batch catalog, created beforehand with the Create catalog API

  • One or more CSV files containing your complete catalog, following the Cloud Sync input format (see below)

1) Prepare your catalog files

A catalog sync is a full replace: at each run, Cloud Sync builds a complete new version of the catalog from the files it reads, and replaces the previous version atomically (see section 3).

This means your files must always contain the entire catalog, never a delta: any item absent from the new version disappears from the catalog.

1.1 File naming and location

  • Only .csv files are synced

  • A file is synced when its name contains the catalog name: for a catalog named products, files like products.csv, products_part_2.csv or 2026-08-06_products.csv are all picked up

  • Files are searched from the root folder of your SFTP user, including all subfolders

  • Each file is limited to 1 GB

  • You can split your catalog into as many files as necessary: for example, a 7 GB catalog can be split into products_v1_part1.csv, products_v1_part2.csv, products_v1_part3.csv, ... as long as each file stays under 1 GB

Batch uses the files' creation/modification date to detect what to sync: a run only reads the files that are new or modified since the last run (see section 3.1).

Always upload the complete catalog. The files read by a run completely overwrite the previous version of the catalog, they are never merged with it: if your catalog spans several files and you only update one of them, the next version will only contain that file's items. When updating your catalog, re-upload all its files.

1.2 One row per item

Your files must contain one row per catalog item. Each row becomes one item of the new catalog version.

Column
Required
Description

id

✅

The item identifier in the catalog. Max 250 characters, unique across all files

All other columns are item attributes. Each row must carry at least one attribute value besides its id.

Good to know:

  • If two rows carry the same id, the first one wins: treat duplicates as a bug in your export

  • A row with an id but no attribute values is skipped

  • An empty cell means the attribute is simply not set on the item in this version

1.3 Attribute columns and your catalog schema

Your Batch catalog declares a schema at creation time: a list of typed fields (string, integer, double, boolean, date, URL, array of strings).

At each run, Cloud Sync reads that schema and maps your CSV columns onto it:

  • A column is synced when its name matches a schema field (names use a-z, 0-9 and _, max 30 characters)

  • Columns that don't match any schema field are ignored

  • Values are automatically converted to the declared type: 2021 is read as a number for an integer field, true as a boolean, and so on

  • A value that cannot be converted (e.g. twenty-one for an integer field) makes the run fail: nothing is published and the live catalog keeps its current items

For date and url fields, simply name the column after the schema field (e.g. release_date). The date__ / url__ prefixes (e.g. date__release_date) are also accepted.

1.4 Value formats

Type
Format in CSV
Example

String

Plain text, max 10,000 characters

Trail running shoes

Integer / Float

Plain number

2021, 49.99

Boolean

true / false, 1 / 0, yes / no

true

Date

RFC 3339 UTC string or Unix timestamp

2026-04-17T04:25:00Z

URL

Absolute URL, max 2,048 characters

https://cdn.example.com/shoe.jpg

Array

Comma-separated values. Double-quote elements containing commas. Max 15 elements of 64 characters each

"running,shoes,men"

1.5 Example file (e-commerce)

For a products catalog whose schema declares name (string), brand (string), price (double), in_stock (boolean), image (URL), release_date (date) and tags (array):

How this maps in Batch:

  • id identifies the catalog item

  • name and brand become string attributes, price a number, in_stock a boolean

  • image is interpreted as a URL attribute, release_date as a date attribute

  • tags becomes an array of strings (["running", "shoes"])

1.6 Best practices

  • Always export the complete catalog: any item missing from the new version is removed from the live catalog

  • Upload new exports under a temporary name that does not contain the catalog name, then rename them once the upload is complete, so a run never reads a half-written file

  • Keep id values stable across runs: they are how items are matched between versions

2) Create the Sync in the Batch dashboard

Cloud Sync is configured from the dashboard via a dedicated Sync module.

  1. Open the Batch dashboard

  2. Go to Data → Cloud Sync

  3. Click Create Sync

  4. Select SFTP as the source

  5. Select your catalog as the destination: every existing catalog appears in the destination list

  6. Pick the sync frequency (from every hour to every 24 hours)

Your catalog doesn't appear in the destination list? Create it first with the Create catalog API: Cloud Sync fills an existing catalog, it does not create one.

2.1 Configure your SFTP connection

Enter:

Field
Description

Host

Hostname or IP address of your SFTP server

Port

The port your SFTP server listens on (22 unless it was configured otherwise)

Username

The user Batch connects with. It needs read access to the synced files

Authentication

A password, or a private key whose public counterpart is installed on your server

Delimiter

The character separating values in your CSV files: comma, semicolon, tab, or pipe

Batch validates the connection before continuing.

3) How a catalog sync works

3.1 Only new or modified files are read

Batch uses each file's creation/modification date as a cursor:

  • A file is read when it is new or was modified since the last successful run

  • Files already synced and untouched since then are skipped: old exports left on the server are not picked up again

  • The files a run reads become the entire new catalog: they completely overwrite the previous version, they are not merged into it

  • If no matching file changed, the run reads nothing and the catalog keeps its current version

3.2 Atomic replace

When a run finds data, Cloud Sync stages a complete new version of the catalog, then switches to it atomically:

  • During the run, your live catalog keeps serving its current items, unchanged

  • The new version becomes visible all at once, at the end of the run, and only if the whole run succeeded

  • If anything fails mid-run (unreadable file, value that cannot be converted, connection error), the live catalog is untouched

3.3 Inserts, updates, and deletes

Because each run replaces the whole catalog, all changes are captured naturally:

  • ✅ inserts: new rows become new items

  • ✅ updates: changed rows update existing items

  • ✅ deletes: rows removed from your files disappear from the catalog

Omitting a row deletes the item: this is why every upload must carry the complete catalog.

3.4 The empty-run guard

A run that finds zero rows never replaces anything: the catalog keeps its current items. An empty source is indistinguishable from a broken one, so emptying a catalog is done deliberately through the Catalogs API, never as a side effect of missing files.

4) Test and enable your Sync

Before enabling the schedule:

  1. Run a test sync

  2. Verify in Data → Catalogs:

    • The item count matches your files

    • Attributes carry the expected types (numbers, dates, URLs, arrays)

    • Items removed from your files are gone from the catalog

Once enabled, Batch automatically handles:

  • batching

  • retries

  • atomic promotion of each new catalog version

PreviousCreate a Sync from Databricks to Batch Profile Events
NextTroubleshooting

Last updated