configuring-imports

Imported from celigo/ai (skills/configuring-imports/SKILL.md) via skills.sh. Install upstream with npx skills add celigo/ai --skill configuring-imports. Copyright stays with the author.

Configuring Imports

An import is the data destination in a Celigo integration. It takes records from an upstream step and writes them to an external system -- REST APIs, databases, ERPs, file servers, or AI models. Every import has one adaptor type and, apart from the connection-less kinds in Minimum Required Fields, is bound to exactly one connection.

Imports handle six concerns:

• Field mapping -- transforming source fields into the destination system's expected format (including value resolution via static maps and lookup tables). Uses Mapper 2.0 (mappings[] array) by default, including NetSuiteHTTPImport. SalesforceImport and WrapperImport support only Mapper 1.0 (mapping.fields[] / mapping.lists[]), and NetSuiteDistributedImport uses its own Mapper 1.0 dialect at netsuite_da.mapping
• Operation logic -- create, update, upsert, delete, attach/detach
• Hooks -- JavaScript pre/post processing at various pipeline stages (preMap, postMap, postSubmit). File-based imports that generate files from records also support postAggregate; an HTTP records import refuses it (422 invalid_hook)
• One-to-many -- fan out a child array already on the input record. Set oneToMany: true and pathToMany to that path (e.g., "lineItems") when one source record should create one import operation per child. After fan-out the child is record ({{record.sku}}) and the parent (without the array) is record._PARENT ({{record._PARENT.id}}). Leave pathToMany blank when the arriving record is itself an array (grouped rows from a groupByFields export or a file group) -- each element then becomes the record, with no _PARENT. A path on an array record processes zero children and still reports success; a blank path on an object record fails with oneToMany_error. oneToMany doesn't batch by itself: templates see batch_of_records only when http.batchSize is above 1 (the default varies by API) or an RDBMS query runs per_page
• Response mapping -- extract fields from the import's API response back into the record for downstream steps. Configured on the flow's pageProcessors[] entry, but planned when building the import. Extracts read the response envelope: id (the destination-assigned record ID), statusCode, ignored, errors, _headers, dataURI (the record's link, when the import sets dataURITemplate), and _json (the response body after any response transform, whose top-level fields are also flattened onto the envelope). Use id for a created record's ID and _json.fieldName for other response fields (e.g., _json.output.1.content.0.text for OpenAI responses) -- _json.id works only when the body itself has a top-level id. Flow test runs never send imports, so id is empty there. Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms
• postResponseMap hook -- JavaScript processing after response mapping merges the response back into the record. Configured on the flow's pageProcessors[] entry, but planned when building the import. Use to transform or enrich the merged record before downstream steps

Imports are used across flows, APIs, and tools.

Import Execution Pipeline

When records arrive at an import step, this pipeline executes in strict order:

1. Input filter (optional) -- records that don't match skip this import (see Gotchas). Configured as filter on the import resource itself -- an inputFilter key on an import is silently discarded. See configuring-filters
2. preMap hook (optional) -- JavaScript processing before field mapping
3. Field mapping -- Mapper 2.0 or 1.0 maps source fields to destination fields, including lookups and hardcoded values
4. postMap hook (optional) -- JavaScript processing after field mapping, before submission
5. Submit to destination -- writes the mapped record to the external system
6. postSubmit hook (optional) -- JavaScript processing after the destination responds (access response data, log results, trigger side effects)
7. Response transform (optional, responseTransform) -- reshapes the response body (_json) after postSubmit. Its expression rules are Transformation 1.0 only (expression.version: "1"; "2" is refused with 422), and it keeps only the fields it maps, so a field it drops is gone for response mapping. Its execution-log stage is responseTransformation
8. Response mapping (optional) -- carries data from the destination response back into the record for downstream steps. Configured on the flow's pageProcessors[] entry, not on the import itself
9. postResponseMap hook (optional) -- JavaScript processing after response mapping merges response data back into the record

Key distinction: Response mapping lives on the flow's pageProcessors[] entry, not on the import resource. preMap, the mapping and postMap shape only the request this import sends: the next step receives the record as it arrived plus what response mapping and postResponseMap add, so a field added in steps 2-4 never reaches it. When building an import that needs to pass data downstream, plan the response mapping at flow design time.

What an HTTP import's templates see. http.relativeURI and header values render against the record before mapping, and an undefined path fails the request: a field the record lacks in the URI -- or a {{lookup.<name>}} whose lookup found nothing -- fails the record before anything is sent, with handlebars_template_parse_error (Failed to generate request url from template: … Details: "<field>" not defined in the model.). That is missing data, not a template bug, so a retry fails the same way; an update import that some records reach without the destination id needs a composite UPDATE/CREATE or ignoreMissing (Common Errors). A field holding "" is defined: it renders empty, and the request goes out without the id. A field holding null renders the text null instead (the request goes to …/null), and a field under an object that is null fails the record like a missing one ("<field>" not defined in null) -- see writing-handlebars, gotcha 16. http.body renders against the mapped record, where an undefined path renders empty and the record still succeeds (strictHandlebarEvaluation doesn't change that). Leave http.body unset to send the mapped record; when you template it, use the mapped (destination) field names, and double braces for string values in a JSON body (gotcha 11).

Categories of Import

Record-Based Imports

Submit structured records to APIs, databases, or ERPs. The vast majority of imports.

• NetSuiteDistributedImport -- high-performance SuiteApp writes (add, update, addupdate, delete, attach, detach). The default for NetSuite when the Celigo SuiteApp or bundle is installed. netsuite_da.restletVersion is fixed when the step is created, so author new steps on suiteapp2.0; a hook's SuiteScript version must match the step's
• NetSuiteHTTPImport -- NetSuite's REST APIs on a netsuite connection (nsDomainType: "suitetalk": POST /services/rest/record/v1/<type> to insert, PATCH .../<type>/{id} to update; "restlet" for a custom RESTlet). Use it when neither the SuiteApp nor the bundle is installed, or the user asks for REST
• HTTPImport -- REST/GraphQL APIs (POST, PUT, PATCH, DELETE). Supports connector-assisted (formType: "assistant") and GraphQL (graph_ql, assistant_graphql) modes; see GraphQL imports
• SalesforceImport -- Salesforce writes through salesforce.api: soap (the default), rest, or compositerecord (Composite Graph, a tree of related records in one request); metadata deploys configuration, not records. salesforce.api has no Bulk value
• RDBMSImport -- SQL databases (Snowflake, PostgreSQL, MySQL, SQL Server, Oracle). rdbms.queryType is a one-element array whose value fixes the companion field: per_record, per_page (renders against batch_of_records) and first_page (runs once, on the first page) take rdbms.query, an array of SQL strings; bulk_insert takes rdbms.bulkInsert (pure INSERT, with no upsert or ignore-existing logic); bulk_load takes rdbms.bulkLoad (a staged load that becomes a MERGE with primaryKeys, or runs your own SQL with overrideMergeQuery). Prefer bulk_load where the database supports it, then bulk_insert for pure INSERTs, then per_page, and per_record only when the logic can't be batched. The legacy uppercase INSERT (and the retired MERGE, COPY and BULK INSERT) fail any save with 422 "Import query types … are deprecated"; UPDATE is legacy too, so don't use it. Other strings, such as lowercase insert, fail as an invalid enum value. A bulk_load import's GET also returns a file block describing its staging file; the platform derives it on every read, so a save that changes it is accepted and discarded. The SQL itself: writing-sql
• MongodbImport, DynamodbImport, JDBCImport -- other databases

GraphQL imports

A GraphQL import is an HTTPImport with http.formType graph_ql (or assistant_graphql, the connector-assisted GraphQL form). Each http.body[] entry is a string holding the {"query", "operationName", "variables"} envelope, one per operation on an update/create pair -- never an object, which is stored as-is and doesn't render -- and the mutation's input comes from the record through Handlebars in variables.

• Endpoint. The import posts to the connection's base URI -- its form has no relative-URI field and writes relativeURI: ["/"] on every save -- so the connection must supply the GraphQL endpoint: a graph_ql connection, or a connector connection bound to the app's GraphQL API. On a connection bound to the app's REST API every record fails (HTTP 406 on Shopify); add a second connection bound to the GraphQL API, re-point the GraphQL steps to it, and leave the shared REST connection unchanged. The form type is set on the step, so a plain http connection whose base URI is the endpoint works too.
• Variables. Under graph_ql or assistant_graphql, a variables value stored as a JSON string is sent as an object; under http it goes out as a string. An object takes expressions inside string values only: "variables": {"input": {"email": "{{record.email}}"}}. A string can also carry bare values, but its string values sit two JSON levels deep, so give each one jsonEncode in double braces: "variables": "{\"input\": {\"note\": \"{{jsonEncode record.note}}\", \"quantity\": {{record.qty}} } }". An object or array goes in as {{jsonSerialize record.items}}, also in double braces. With {{record.note}}, or triple braces on either helper, a value holding a quote fails its record with 422 invalid_json before anything is sent.
• Errors come back as HTTP 200. A GraphQL server usually answers a failed mutation with 200 and an errors array, and with no response settings the import counts the record as written. Set http.response.resourceIdPath to the id only a successful mutation returns (["data.customerCreate.customer.id"]): any other answer then fails the record with missing_id, top-level errors and the mutation's own userErrors alike. For the server's message on top-level errors, also set failPath: ["errors[0].message"] and errorPath: "errors[0].message"; failPath: ["errors"] catches a response with exactly one error and lets two or more through.

File-Based Imports

Write or upload files to remote storage. Require the file{} configuration block. Two modes:

• Record-to-file -- aggregates incoming records into a file (CSV, JSON, XML, XLSX). The file{} block defines the output format, and file.aggregation.type how a run's records become files: all (one file per run), page (one per page) or key (EDI file definitions only, with B2B Manager). Send the legacy file.skipAggregation as its pair -- true with page, false with all or key -- because validation reads the value you send: an EDI file definition on page without true, or a file.batchSize without it, fails with 422 not_supported. The format is file.type (csv, json, xml, xlsx), with its settings under that name: a CSV's file.csv.includeHeader defaults to true, so set it false for a file with no header row. The columns are the import's mappings rows, in order, each row's generate naming its column; there is no file.columns.
• Blob passthrough -- transfers a file as-is from an upstream blob export or blob lookup. Set blob: true, and blobKeyPath to the record field holding the blobKey that step put there (FTP and Google Drive imports fall back to the record's root blobKey when it's unset). Base64 content in a record is record data, not a blobKey. AI agent, guardrail and tool imports reject both fields with 422 not_supported.

Adaptor types:

• HTTPImport with http.type: "file" -- the folder mode of the cloud file providers (Google Drive, Box, Dropbox, Azure Blob Storage, Google Cloud Storage, Azure Data Lake Storage Gen2, Celigo Storage). Address the folder by path in http.relativeURI (a folder, never an API endpoint) or by the provider's folder id in file.directory (pathMode: "directoryId"), and name the file in file.fileName; provider-specific location rules are in file.yml
• FTPImport -- CSV, XML, JSON, XLSX, EDI files to FTP/SFTP. The folder goes in ftp.directoryPath and the file's name in ftp.fileName, which takes no directory separators (the UI stores the same name in file.fileName too); a dated name is a Handlebars template (writing-handlebars > File paths and names)
• S3Import -- objects to Amazon S3
• AS2Import -- AS2 EDI file transmission
• FileSystemImport -- local/on-premise filesystem writes

AI Imports

Invoke AI models for classification, extraction, or safety checks. No _connectionId required unless using BYOK.

• AiAgentImport -- LLM invocations (any provider in the aiAgent.provider enum) with structured output, tool use, and reasoning
• GuardrailImport -- PII detection, content moderation, or custom AI-based validation

Stack and Tool Imports

• WrapperImport -- custom connector code, typically a function on a stack (wrapper.function)
• ToolImport -- runs the Celigo Tool named by tool._toolId. It needs no _connectionId; instead it must bind each connection the tool's steps use in tool.overrides.connections as {_abstractId, _id} -- set _id equal to _abstractId to keep the tool's own connection. A connection left unbound fails the run with connection_not_overridden

Topic Imports (Event Streams)

• TopicImport -- publishes each record as a message to an Event Streams topic, which topic listeners (TopicExport) read. Set topic._topicId to the topic's _id, never its name (celigo topics list; an unknown id fails the save with 422 invalid_ref), and optionally topic.batchSize (default 1000). There's no connection and no create/update logic: each record becomes one message, reshaped by the mapping if there is one; with one-to-many, each element is its own message carrying the parent under _PARENT. Register the topic on the flow's integration so its flows can bind to it (celigo integrations register-topics <integrationId> <topicId>); the account needs the Event Streams entitlement. Test runs publish nothing; an invoke publishes for real

Import Matching (create / update / upsert)

The core decision on most record-based imports is the operation -- what happens to each record at the destination. Create requires no match key; update and delete require one; upsert checks first and does whichever applies. Users describe the intent in business terms ("look up the customer and update them", "match by email and upsert", "skip the ones that already exist") that resolve into five behaviors:

• Always create -- every record is submitted as new. No matching, no checks.
• Create only if missing -- match-check first; submit as create if not found, skip silently if found.
• Update only if exists -- match-check first; submit as update if found, skip silently if not.
• Update only if exists, fail if missing -- strict variant; no-match records error out instead of skipping.
• Upsert -- submit a create when no match is found, an update when matched. The catch-all, and the right default when the user is vague.

When matching applies, decide four things: the matching behavior, the match key field(s) (email, external_id, customer.id -- required for anything other than always-create), the on-match action (update, skip, fail), and the on-no-match action (create, skip, fail). A lookup on a cross-reference field (NetSuite externalid, or a custom field that holds the source id) finds only records whose field holds that id, so the same import must write the source id into the field when it creates them; without that mapping no later run finds them, and each run adds the record again.

How a destination implements matching is adaptor-specific -- there is no single "matching mode" field. A destination might expose: a native upsert keyed off an external ID (Salesforce upsert, NetSuite upsert, RDBMS ON CONFLICT); a distinct addupdate operation that handles both paths in one call; an ignoreExisting flag paired with a lookup that probes before writing (lock it when repeats can arrive concurrently -- see Writing Exactly Once); two separate create and update endpoints with no upsert variant (see composite imports below); or a lookup endpoint plus separate create and update endpoints, where the lookup runs pre-write to drive the create-vs-update decision. Some destinations have no matching concept at all -- writing a CSV to FTP, sending an email, posting to a webhook -- so every record goes out as-is.

Prefer import-level matching over a separate lookup step. Imports natively support this pre-write check, so a single import that does the matching and the write together means fewer steps, fewer round-trips, and no glue logic to maintain. A standalone lookup earns its place only when the looked-up data has a consumer beyond the write -- a router branching on something other than "does this exist", an AI agent reasoning over the result, or multiple downstream steps reading different fields. If the only consumer is the destination call itself, the work belongs inside the import. It's also the only check that runs under the import's concurrency lock, which matters whenever the same record can arrive twice (Writing Exactly Once). An import's own lookups refuse an ambiguous match only when told to: an HTTP lookup that finds more than one match fails the record only with allowFailures: false, and otherwise sends the first match with no error (gotcha 14); a NetSuite DA lookup with allowFailures: true also resolves to one of the matching records unless useDefaultOnMultipleMatches sends it to the default (netsuitedistributed.yml). So search on a key that identifies one record. When an update's key must come from a separate lookup that can match several records, guard it -- fail or skip on multiple matches, or check for exactly one (configuring-lookup-caches > Safe Lookup-to-Update).

Composite (two-endpoint) imports

When an HTTP destination has no native upsert but exposes separate create and update endpoints, a composite import pins both endpoints on a single import node, role-tagged create and update. The runtime picks per record via a match-key check -- the same way a native upsert would -- so the flow stays one step. Prefer this over two separate imports driven by an upstream lookup or router; reach for separate imports only when the create and update paths must diverge beyond endpoint selection (different mappings, different downstream consumers, or different hook chains).

The operations sit in positional arrays, one entry per operation: http.requestType (["UPDATE", "CREATE"]), http.method and http.relativeURI. Exactly one decider picks per record: http.existingExtract, a field the record already carries (non-empty means UPDATE, whose URI reads it as {{{data.0.<field>}}}), or http.existingLookupName, a lookup in http.lookups that probes the destination (the UPDATE URI reads {{lookup.<name>}}). Never add ignoreExisting to a composite import, and give every http.response.* array one entry per operation.

existingExtract must hold the destination's id (put there by an upstream step's response mapping, or kept as a cross-reference), never the source record's own id: every record carries one, so every record would go to UPDATE. With existingLookupName, a lookup that comes back empty sends the record to CREATE, with no error -- and so does one whose extract resolves nothing, so a $.-prefixed extract sends every record to CREATE. A templated http.body has one entry per operation too; left unset, both requests send the mapped record. A search-driven upsert against an API that finds contacts by email:

"http": {
  "requestType": ["UPDATE", "CREATE"],
  "method": ["PATCH", "POST"],
  "relativeURI": ["/contacts/{{lookup.findContact}}", "/contacts"],
  "existingLookupName": "findContact",
  "lookups": [{
    "name": "findContact",
    "method": "POST",
    "relativeURI": "/contacts/search",
    "body": "{\"email\": \"{{email}}\"}",
    "extract": "results[0].id"
  }]
}

The lookup sits in http.lookups (a lookups array sent only at the top level is dropped on save), its request template is in body, and the template names the record's fields at the top level ({{email}}, never {{record.email}}, which renders empty, so every search misses and every record is created again; gotchas 10 and 11). Its extract is the id itself, because that is what the UPDATE URI receives. An HTTP lookup's extract is a path into the response written without $. -- results[0].id, or ids for an array of ids -- unlike a Mapper 2.0 row's extract, which needs $.. A $. prefix, [*], or a dot path through an array (results.id) resolves nothing, so the lookup always comes back empty.

Writing Exactly Once

Every source can hand an import the same record twice: a sender fires a webhook again, a failed or canceled run is re-run over the same delta window, an error is retried, a record that wrote this step but failed a later one comes back on the next run, or another flow writes the same destination. A trace key never de-duplicates -- it correlates a record's errors and is the default lock key below; a record that arrives twice is written twice. Make the write itself idempotent, in this order of preference:

An insert on a unique key refuses a repeat as an error (Salesforce DUPLICATE_VALUE, a unique-constraint violation). That keeps a repeated record out, but it refuses a changed one too: when changes to existing records should arrive, upsert on that key.

A match that is really a search run before the write -- a lookup-driven composite create/update, an addupdate found by search -- is a check-then-write, not rank 1, unless the destination also rejects the duplicate key. A natural id is unique only within its source: when two sources can produce the same one (two stores, two accounts), namespace the key per source ({{record.storeId}}-{{record.orderId}}), or one source's record collides with or overwrites the other's. Rank 2 is only as good as its lookup: a lookup that fails counts as "not found", so the record is written (gotcha 12).

The concurrency ID lock. idLockTemplate renders a key per record; records that render the same key are never submitted at the same time, and different keys run in parallel.

• Default: the record's trace key (a source's traceKeyTemplate arrives on the record as io_custom_trace_key). Without a template, the trace key is the first top-level id field the record has -- id, internalId, _id, internal_id or identifier, in a few spellings each -- and a nested id is never used. A blank trace key means no lock, so set idLockTemplate when the identity is nested or composite. A template replaces the default rather than adding to it.
• Render time: from the record as it arrives, before this import's own lookups and mapping. A key built from {{lookup.<name>}} or a mapped field renders empty -- and an empty key is no lock at all. Key it on a field the record carries ({{{record.orderId}}}), or one an upstream lookup step added through response mapping.
• Scope: this import across every page, every concurrent run, and every flow that uses it (celigo imports dependencies <id>) -- not the connection: two imports rendering the same key still run in parallel. So two imports that both create the same entity (an order flow that creates missing customers, and a separate customer sync) can each miss the other's write and create it twice: create the entity through one import, or schedule the flows so they don't overlap. Records within one page are sent one at a time anyway, so a one-page test never shows the race.
• Coverage: the ignoreExisting existence check runs inside the lock. Nothing documents the same for ignoreMissing, the composite create/update decision, or one-to-many children -- where those must write once, connection concurrencyLevel: 1 closes the check-then-write race (the import's pages run one at a time), at the cost of serializing everything on that connection.

Quick Reference

Adaptor Decision Matrix

Raw HTTP is the fallback, not the default. Pick the most specific match, in order:

1. Native adaptor -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an HTTPImport against that app's REST API.
2. Pre-built HTTP connector -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see Check for a pre-built connector). The step is still an HTTPImport, but it runs on a connector-backed connection and takes its endpoint config from the connector.
3. Manual HTTP -- hand-write the config from public API docs only when no connector exists or it doesn't cover the operation you need.

For NetSuite, run celigo connections suiteapp-status <connectionId>: suiteapp.success or bundle.success true means NetSuiteDistributedImport; both false, each with statusCode 404 and an install URL, means neither is installed, so use NetSuiteHTTPImport. A different statusCode means the check itself failed, not that the components are missing: fix the connection and re-run it before choosing. NetSuiteImport is legacy -- don't use it unless asked.

adaptorType is case-sensitive: NetSuiteDistributedImport, not netsuitedistributedimport.

Minimum Required Fields

Every import needs at minimum: name, adaptorType, and the adaptor config block (http{}, netsuite_da{}, salesforce{}, tool{}, topic{}, etc.). _connectionId is required except on the connection-less imports: ToolImport (connections are bound per tool step in tool.overrides), TopicImport (must not carry one -- with a connection, the save is validated as that connection's adaptor and fails), and AiAgentImport / GuardrailImport unless they use a BYOK connection.

Which Schemas to Read

1. Always: request.yml (base fields)
2. Plus: the adaptor-specific file from the decision matrix
3. If file-based: also file.yml
4. If cloning: clone-request.yml, clone-response.yml

Schema Index

All schemas are in references/schemas/:

• Base fields (all imports): request.yml
• Response shape: response.yml
• Adaptor-specific config:
  • http.yml -- HTTP/REST/GraphQL (methods, URIs, headers, response parsing, lookups, composite upsert via existingExtract or existingLookupName)
  • netsuitedistributed.yml -- NetSuite SuiteApp (operation, recordType, internalIdLookup, mapping, lookups)
  • netsuite.yml -- NetSuite legacy
  • salesforce.yml -- Salesforce (sObjectType, operation, api, idLookup)
  • rdbms.yml -- SQL databases (queryType, query, bulkInsert, bulkLoad)
  • ftp.yml -- FTP/SFTP
  • s3.yml -- Amazon S3
  • mongodb.yml -- MongoDB (method, collection, filter, upsert)
  • dynamodb.yml -- DynamoDB
  • jdbc.yml -- JDBC databases
  • as2.yml -- AS2 EDI
  • wrapper.yml -- custom stack connectors
  • tool.yml -- Tool imports (_toolId, connection overrides)
  • topic.yml -- Event Streams publisher (_topicId, batchSize)
  • filesystem.yml -- local filesystem
• AI config:
  • aiagent.yml -- AI agent (provider, model, instructions, tools, structured output)
  • guardrail.yml -- guardrails (PII, moderation, AI-agent validation)
• File output: file.yml (CSV, XML, JSON, XLSX config for file-based imports)
• Clone: clone-request.yml, clone-response.yml

Related Skills

• configuring-connections > Quick Reference -- connection types and auth for import destinations
• managing-file-storage > Use stored files in a flow -- Celigo Storage folders as a file destination
• writing-mappings > Mapper 2.0 Workflow -- field mappings on imports
• writing-scripts > Data Pipeline Hooks -- preMap, postMap, postSubmit, postAggregate hooks
• writing-handlebars > Quick Reference -- dynamic values in URIs, HTTP bodies, SQL queries
• building-flows > How to Build a Flow -- wiring imports into flow pipelines as page processors
• configuring-topics > How to Build an Event Stream -- the topic a publisher writes to: registration, retention, messages
• troubleshooting-flows > Diagnostic Workflow -- diagnosing import-related failures

How to Build an Import

1. Identify the target application

What system are you writing data to? This determines adaptor type, connection type, and configuration shape.

2. Check for existing patterns

Before building from scratch, look at what already exists:

# Search your account (fast, uses local index)
celigo account search "<keyword>"

# Show what an existing import uses (connection) and what uses it (flows)
celigo account dependencies import <id>

# Find orphaned imports not referenced by any flow
celigo account lint

# Search marketplace templates
celigo templates marketplace

# Extract just imports from a template
celigo templates preview <id> --model Import
celigo templates preview <id> --summary

The account index auto-refreshes when stale (older than 15 minutes by default). Force a fresh snapshot with celigo account snapshot.

3. Check for a pre-built connector

Always run this check before writing any HTTP config. Celigo maintains 550+ HTTP connectors with pre-configured auth, endpoints, and resources. Hand-write a manual HTTPImport from public API docs only when this search comes up empty or the connector doesn't cover the operation you need.

# Search HTTP connectors
celigo http-connectors list | grep -i "<application-name>"
celigo http-connectors get <id> --full    # see endpoints, resources, auth config

# Drill into the endpoints the connector defines for imports
celigo http-connectors catalog <id> --resource-type import --published-only
celigo http-connectors endpoint-detail <id> --resource-type import --resource-id <rid> --endpoint-id <epid>
celigo http-connectors endpoint-detail <id> --resource-type import --resource-id <rid> --endpoint-ids <createId>,<updateId>   # a create/update pair

# Search trading partner connectors (EDI, AS2)
celigo tp-connectors list

If a connector exists, create the connection from it (http._httpConnectorId -- see configuring-connections > Check for a pre-built connector and global iClient) and take the import's relativeURI, method, and body/response shapes from the connector's endpoint metadata rather than reconstructing them from public API docs. Write each operation's relativeURI and method exactly as the endpoint defines them: saving links the import to those endpoints by setting http._httpConnectorVersionId, http._httpConnectorResourceId and http._httpConnectorEndpointIds[] (one per operation; imports drop the singular _httpConnectorEndpointId), but pointers you send for the same connector are kept as sent. Read the import back: missing pointers mean a URI matched no endpoint. When retargeting an import, update or drop the old pointers in the same save -- a GET-modify-PUT that changes only the URI leaves the import naming the old endpoint.

endpoint-detail on the import plane returns the resource, whose resourceFields catalog the record's fields, and each endpoint's supportedBy: what the connector's own form applies. preConfiguredFields are values it sets, fieldsToUnset are fields it clears, and fieldsUserMustSet is what it asks for -- on Shopify's create-customer endpoint, ignoreExisting and the mapping targets mapping.fields.generate.customer.email, …first_name and …last_name. The paths are the form's, not the import's. On create and update endpoints, lookupToIdentifyExisting is the existence lookup to build (the endpoint to call and the extract that reads the existing id, such as customers[0].id), and pathParameterToIdentifyExisting names the update URI's id parameter.

4. Query metadata for the target system

For NetSuite, Salesforce, and RDBMS connections, discover available record types and fields:

celigo metadata types <connectionId>          # List record types / sObjects / tables
celigo metadata fields <connectionId> <type>  # List fields for an entity type
celigo metadata fields <connectionId> <type> --refresh   # Bypass the cache after a field is created in the app

Metadata is cached on the server, so a field or record type created since the last read stays missing until you re-read with --refresh.

• NetSuite: metadata fields returns field IDs, types, and groups. On NetSuiteDistributedImport, use the IDs in netsuite_da.mapping.fields[].generate and netsuite_da.mapping.lists[].fields[].generate; sublist names (groups such as "item", "addressbook") go in netsuite_da.mapping.lists[].generate. NetSuiteHTTPImport maps with Mapper 2.0 mappings[]. Lookup field IDs here are the searchField/resultField values for netsuite_da.lookups[].
• Salesforce: metadata fields returns field API names, types, and relationship info. Use field API names for salesforce.sObjectType lookups and for discovering which fields are createable/updateable.
• RDBMS: metadata fields returns column names and types for a table — use these to write SQL queries (see writing-sql) and verify column names before building bulkInsert.tableName or bulkLoad.tableName.

5. Determine the category

Is this a record-based import (submit records to an API/database/ERP), a file-based import (write files to storage), or an AI import (invoke a model)?

6. Choose the right adaptor type

Refer to the Adaptor Decision Matrix in the Quick Reference above. For NetSuite, Diagnosing NetSuite Steps has the account-side checks (the SuiteApp and bundle, the account's concurrency limit) and the errors NetSuite steps return.

7. Build the import JSON

Reference the Schema Index for the exact fields needed. Use the Which Schemas to Read decision rule to determine which files to consult.

File Uploads over HTTP (multipart/form-data)

Some destination APIs accept files only as multipart/form-data POSTs (Jira attachments, QuickBooks attachables, OpenAI file uploads). This is an HTTPImport with blob: true and blobKeyPath, posting to the API's upload endpoint, and it works nothing like a JSON record write. Do not set http.type: "file": that is the folder mode of file-storage connectors (Google Drive, Box, Dropbox, Azure Blob), which treats relativeURI as a folder path -- combined with an API endpoint URL it saves cleanly and fails every record with "Directory is not found." Four pieces have to line up:

1. Where the bytes come from. The import never carries the file itself. A preceding blob export or blob lookup pulls the file into blob storage, and that step's response mapping puts the reference onto the record -- the idiom is {"extract": "data.0.blobKey", "generate": "blobKey"}. The record then carries a blobKey (a storage pointer, not content).
2. The media type. The connection's media type (or the import's request media type override) is multipart/form-data; success/error response media types usually override to JSON so replies parse normally.
3. The request body. Not raw MIME and not a normal handlebars payload -- a JSON array of parts, each {name, value, type} plus optional filename (include the extension) and mime-headers. The file part is "type": "attachment" with "value": "{{blob}}" -- {{blob}} is the only accepted value for an attachment (anything else 422s). Fields the API wants alongside the file ride as "type": "inline" parts; an inline part whose value is a JSON object must be serialized. One file reference per import.
4. blob: true + blobKeyPath (Advanced settings) -- blobKeyPath is the JSON path in the record where the blobKey lives (blobKey, or file.blobKey if nested). At send time the platform follows it into blob storage and streams the real bytes into the attachment part.

The platform assembles the final MIME body itself -- it generates the boundary (never hardcode one), writes each part's Content-Disposition, and substitutes the attachment part with the raw file bytes. The parts array is a build recipe, not the payload.

Not every multipart API is form-data. Some upload endpoints expect multipart/related instead (e.g. Google Drive's /upload/drive/v3/files?uploadType=multipart), and the parts-array machinery does NOT apply. There the request body is the literal MIME document: explicit boundary, a JSON metadata part, and a content part referencing {{blob}} (double braces). Check which flavor the destination API documents before building -- mixing them produces an import that saves cleanly and fails at runtime.

Async Destinations (submit, poll, confirm)

Some destination APIs only acknowledge a write (an HTTP 202, a job ticket) and finish it in the background -- bulk loads, file ingestion, document conversion. Attach an async helper to the import (http._asyncHelperId) so the step submits, polls a status export until the external work completes, and only then resolves. The mechanics and constraints (a status export with done/error value lists and poll intervals; no transform, output filter, or hook on the async-configured step; helpers cannot nest) are identical to the export side -- see configuring-exports > Async APIs (submit, poll, fetch). Only add one when the API genuinely cannot confirm the write synchronously.

Testing an Import Without Writing

Most ways of trying an import send the request to the destination for real. Know which is which before you point one at a production system:

Invoking by id or as a virtual import runs the step with the import's own settings only: hooks get no integration or flow settings, and {{settings.integration.*}} and {{settings.flow.*}} render empty. To run it as a step of its flow, tool or API, add --integration <id> and --flow, --tool or --api <id> to the --connection form.

In the flow builder's import editor, Preview builds the request without sending it (HTTP imports only), while Send is a real write. No CLI command offers the compose-only preview. From the CLI, test-run the flow and read the import's stages with celigo flows test-run-step-results: the mapping stage's output is the record after mapping, while the request stage shows the body template unrendered, and its input is the record before mapping. Database imports have no compose-only surface at all; the SQL editor's evaluation is their only preview.

Save a realistic mockResponse. In test runs and flow-builder previews, an import with no saved mock answers with the placeholder {"id": "1234567890", "mockResponse": "replace me"} (numbered per record on a page: 1234567890-1, 1234567890-2), so response mappings extract nothing real and downstream steps see empty data. Save an array of records shaped like the destination's answer -- statusCode, the created record's id, and the response body under _json:

celigo imports set <importId> 'mockResponse=[{"statusCode": 200, "id": "SO-1001", "_json": {"id": "SO-1001", "status": "created"}}]'

A saved mockResponse substitutes only in test runs and flow-builder previews -- imports invoke, real runs, and retries always send. sampleResponseData only seeds the response-mapping editor; test runs and previews never use it.

CLI Commands

# CRUD
celigo imports list
celigo imports get <id>
celigo imports create < import.json
celigo imports update <id> < import.json
celigo imports set <id> key=value [key2=value2 ...]
celigo imports delete <id> [-y]
celigo imports audit <id>          # change history
celigo imports dependencies <id>   # the flows, APIs and tools that use it

# Invoke -- a real write: each record is sent to the destination (no job is created).
# HTTP 200 even when records fail; check each result's statusCode and errors
echo '[{"name":"test"}]' | celigo imports invoke <id>
# Virtual import: an unsaved or modified document through a connection (also a real write)
echo '{"import": {...}, "data": [{"name":"test"}]}' | celigo imports invoke --connection <connectionId>

# Mock response for test runs and flow-builder previews (never used by invoke or real runs)
celigo imports set <id> mockResponse=file://./mock-response.json

# Clone and connection management
echo '{"connectionMap":{"oldConnId":"newConnId"}}' | celigo imports clone <id>
celigo imports replace-connection <id> <newConnectionId>

# Discovery
celigo templates marketplace
celigo http-connectors list
celigo http-connectors catalog <id> --resource-type import --published-only
celigo http-connectors endpoint-detail <id> --resource-type import --resource-id <rid> --endpoint-id <epid>
celigo tp-connectors list
celigo metadata types <connectionId>
celigo metadata fields <connectionId> <entityType>

# Debug -- a window of at most 60 minutes (debugUntil, set by PATCH; a PUT doesn't set it).
# It records each request the import sends while open: not earlier ones, and nothing from test runs
celigo imports enable-debug <id> [--duration <minutes>]
celigo imports disable-debug <id>
celigo flows debug-requests <flowId> <importId>   # read them from the consumer (or apis / tools debug-requests)

Pre-Submit Checklist

Required (all imports)

• name is set
• adaptorType exact case matches connection type (request.yml > adaptorType)
• _connectionId references a valid, online connection (skip for ToolImport, TopicImport, and AI imports without BYOK)
• Adaptor config block name matches adaptorType (http{} for HTTPImport, netsuite_da{} for NetSuiteDistributedImport, etc.)

Adaptor-specific

• HTTP: pre-built connector was checked (celigo http-connectors list) -- hand-written config only because no connector covers the app or operation
• HTTP: http.method and http.relativeURI are set (http.yml)
• HTTP: string values in a JSON http.body or lookup body are double-braced inside quotes; triple braces only for numbers, booleans and serialized JSON (gotcha 11)
• HTTP lookups: every lookup is in http.lookups (the save keeps that array and mirrors it to the top-level lookups; one sent only at the top level is dropped), each request template is in body, not postBody, and names the record's fields without record. (gotcha 10), and each extract is a path without $. to the id itself or an array of ids, from a search on a key that matches one record (gotcha 14)
• NetSuite DA: netsuite_da.operation and netsuite_da.recordType are set, and update, addupdate and delete carry netsuite_da.internalIdLookup -- the lookup, not a mapped internalid, picks the record (on update it overrides a mapped one). When the record already carries the internal id, use the expression form ["internalid","is","{{{id}}}"] rather than dropping the lookup (netsuitedistributed.yml)
• NetSuite DA fields set at record initialization (NetSuite preloads them when it creates or transforms the record, before the other fields): one body row {"generate": "celigo_initializeValues", "hardCodedValue": "entity,subsidiary"} listing their ids, comma-separated and without .internalid, next to the rows that map those fields. It's what the mapping editor's "Use this field for NetSuite record initialization" saves; a useAsAnInitializeValue key on a row isn't stored
• NetSuite DA serialized items: the SuiteApp and bundle take a line's serialnumbers as <serial>(<quantity>) text when the account's Advanced Bin / Numbered Inventory Management feature is off; with it on, give the serials through the line's inventory detail subrecord (celigo_inventorydetail, netsuitedistributed.yml)
• NetSuite DA keyed on externalid (["externalid","is","{{{id}}}"]): a mapping writes that same source id to externalid ({"extract": "id", "generate": "externalid"}). The lookup finds only records whose externalid was set, so without the mapping an addupdate adds the record again on every run
• RDBMS: rdbms.queryType is one current value (bulk_load, bulk_insert, per_page, per_record or first_page) with its companion field set -- never the legacy uppercase INSERT or UPDATE (rdbms.yml)
• Salesforce: salesforce.sObjectType and salesforce.operation are set. upsert also needs salesforce.upsert.externalIdField (the Salesforce external id field) and salesforce.idLookup.extract (the record field matched against it); update, delete, addupdate and an insert with ignoreExisting need idLookup.whereClause instead (salesforce.yml). The mapping is Mapper 1.0: mapping.fields[] rows of {extract, generate} (or hardCodedValue), never mappings[]

Cross-resource consistency

• Connection type matches the import's adaptorType
• If response mapping needed: configured on the flow's pageProcessors[] entry, not on the import itself
• If one-to-many: oneToMany: true, with pathToMany set to the child array path for an object record (child fields then use record.<field> and parent fields record._PARENT.<field>) and left blank for an array (grouped) record
• If downstream steps read this import's response: a realistic mockResponse is saved, so test runs carry real-looking data past this step
• If a repeated record would duplicate data: the write is keyed (a destination-enforced key, or ignoreExisting + an existence check) and idLockTemplate renders a non-empty key from a field the record carries on arrival -- never {{lookup.*}} (Writing Exactly Once), and its existence lookup has been checked against a record that exists (gotcha 12)
• Mapper 1.0 only where required: mapping.fields[] / mapping.lists[] on Salesforce and wrapper imports, netsuite_da.mapping on NetSuite DA -- never mappings[] there. NetSuiteHTTPImport uses mappings[]

Gotchas

1. PUT erases omitted fields. Always GET first, modify, then PUT. The set command handles this. The import PATCH route accepts only debugUntil (any other path is refused with 422), so a change to a mapping, body or lookup is always this full PUT, never a reason to send the user to the UI. Leave the read-only fields a GET adds (_id, lastModified, createdAt) out of the body.
2. Including a rest: block creates a legacy RESTImport. Use only http: for new imports.
3. Input filter skips that import, not the record. Filtered records skip the current import step but continue to subsequent steps in the flow, without this import's response-mapped fields, so a later step that writes those fields back needs its own filter on them. Check numIgnore on the job if records seem to bypass a step.
4. Multipart file parts only accept {{blob}}. In a multipart/form-data parts array, the file part must be "type": "attachment" with "value": "{{blob}}" -- any other value 422s. Never hardcode the MIME boundary; the platform generates it. Fields the API wants alongside the file ride as "type": "inline" parts.
5. bodyKey/blobKey in logs is an artifact, not a payload field. Audit and debug logs never show the assembled multipart body -- an internal storage pointer appears where the body would be. But if the destination actually received the literal string bodyKey or blobKey, the file part is misconfigured (an inline part where an attachment belongs, or a blobKeyPath that doesn't resolve).
6. imports invoke is a real write. "No job created" doesn't mean nothing is sent: every form executes the request against the live destination, and a saved mockResponse doesn't stop them. It also runs the import's hooks, postSubmit included, and its response transform (the editor's Send stops before postSubmit). A 200 can still carry per-record failures in each result's errors, and on a file import (FTP, S3, file-mode HTTP) it means the file was composed, not delivered: an FTP import to an unreachable host still answers {name, fileLocation}. Point it at a sandbox, or test-run the flow when you only need to check mapping output.
7. A trace key is not duplicate detection. Repeated keys are normal (delta re-emits, retries) and never skip, merge, or reject a record. Only a keyed write stops the second copy -- see Writing Exactly Once.
8. A lock keyed on this import's own lookup is no lock. idLockTemplate renders before the import's lookups and mapping run, so {{lookup.<name>}} renders empty. Check the key against a real record (an error's retry data: celigo flows error <flowId> <importId> <errorId> --retry-data), not a sample that may lack the field.
9. Lookups can't chain. A lookup's own URI, body and headers render before any lookup has run, so {{lookup.<name>}} there is undefined: it either fails the render (handlebars_template_parse_error: Unable to read "<name>" from undefined) or renders empty, and either way that lookup fails or asks the wrong question for every record. Test runs never execute imports, so the first real run is where it shows. Resolve the value in one query, or add an upstream lookup step and reference the field it adds to the record.
10. An HTTP import's lookups send body, not postBody, and read the record's fields at the top level. Put a lookup's request template in http.lookups[].body. A template saved as postBody, the field http.yml documents, saves without complaint, but the lookup goes out with an empty body, so it asks the wrong question for every record and nothing errors. The CLI's schema check calls body undocumented; it's still the field that's sent. Inside that template the record's fields sit at the top level: {{email}} renders the record's value, and {{record.email}} renders empty.
11. In a JSON body, string values take double braces. {"name": "{{record.name}}"} is JSON-escaped at runtime, so a quote, backslash, newline or tab in the value arrives intact. Triple braces insert the raw text: such a value fails its record with 422 invalid_json before anything is sent, and in a lookup's body the lookup quietly finds nothing (gotcha 12). Keep triple braces for numbers, booleans and serialized JSON ({{{jsonSerialize record}}}).
12. A failing existence lookup counts as "not found". With ignoreExisting, an http.ignoreLookupName lookup that fails -- a wrong URI answering 404, a body broken by a raw quote -- extracts nothing, so the record is written as new and no error is raised. Before trusting the check, invoke the import in a sandbox with a record that already exists there and confirm the result comes back ignored.
13. A 200 isn't always success. When an API answers 200 for a failed write, set http.response.successPath + successValues or failPath + failValues (one entry per operation; values compare as strings). A record that trips them fails: an invoke reports statusCode 422, and a run records an error whose code is the destination's status (200), source application, with the response body as its message -- or response_failure ("response failed using path") when a successPath value doesn't match.
14. An ambiguous HTTP lookup fails only with allowFailures: false. When a lookup's extract resolves to two or more values (an array of ids or of result objects), the record fails wit

Truncated - read the full file at https://www.skills.sh/celigo/ai/configuring-imports.