Data team manager here, qq: how do you handle type conversions between different systems? Nice work overall!
replies(1):
Connect is an API you can use in your own products to either pull your customers’ data into your own systems, or push data from your own systems to your customers’, or both. We have first-class support for data warehouses too. And there's a self-hosted deployment option.
You can set up automatic data syncs in any direction between data warehouses, databases (including CDC streaming), cloud applications (e.g. Salesforce, Zendesk, etc), spreadsheets, cloud storage (S3 etc), and arbitrary APIs and webhooks. We take care of authentication, automatically pushing updates, type conversions, rate limits, scaling to handle large volumes, monitoring, and alerting.
You can also sync updates from custom queries powered by data models you define as well join data across disparate systems (e.g. HubSpot + Stripe + Airtable) to sync fields from them in one payload to any destination.
You can see demo videos and code examples for a small subset of use cases here: https://apidocs.polytomic.com/guides/code-examples/overview.
We’d love to take your comments and feedback! Happy to answer questions too.
We automatically do the right thing depending on the situation. The type engine is highly-precise when we generate our own schema. For example, JSON objects and arrays get converted to native struct and array types in data warehouses (e.g. Databricks, BigQuery) rather than strings.
But we're forgiving in other situations where we don't control schemas. For example, we'll automatically convert a source string type to a date when mapping to a Salesforce or Zendesk (or other cloud app) date field.