Fivetran Function Connector

A Fivetran function connector enables building a custom data connector as an extension of Fivetran.

fivetran.com/docs/functions

Hashmap SAM Application Accelerator

Hashmap uses AWS SAM template definitions (an IaC approach) to streamline the provisioning of resources in AWS for a Fivetran custom connector.

Lambda functions | Roles | Policies

Accelerate Your Build and Save $

With our pre-built AWS SAM template definitions, time to value for Fivetran Custom Connectors using Cloud Functions is accelerated by ~ 50%

Here's our Flow for a Custom Fivetran Connector 

Validate & Plan

• Assessing, analyzing, and validating (access) to the data sources

• Design project execution plan, identify user stories, create project backlog, and prioritize user stories into sprint logs

• User and service account provisioning

• Sprint iterations to build connector/s

Connector Build

• Leveraging Python app template, develop AWS Lambda functions and configure as Fivetran ‘Function’ connectors

• Leveraging Hashmap SAM templates, provision services and features as required such as Lambda Functions, IAM

• Promote and validate the functions and associated Fivetran connectors in QA environment

• Deploy (productionalize) to PROD environment and perform data quality tests to confirm

Deliverables

• AWS Lambda function app (preferably in a Git controlled repository)

• Additional AWS resources (IaC code/templates) (in a Git controlled repository)

• Test scripts used to validate data quality post extract and load into Snowflake

• Project documentation

Dependencies

• Client Business SME provides input for post-load DQ validation tests

• Client Tech Admin SME configures user/service accounts on AWS,
  Fivetran and Snowflake (as required)

• Project stakeholder to assess and provide sign-off on the connectors

Assumptions

• Source systems are identified and access to vendor-specific APIs are already set up and available for use with Fivetran Cloud functions 

• Target Environment(s) are set up within Snowflake (landing database/schema setup and roles configured)

• Transformations (if required) will be performed by the client team post loading

Considerations

• API sources tend to have their own limitations (filter/watermark column not available to perform incremental syncs, limits on number of API calls, etc.) that might require more detailed development to overcome. Any limitations will be handled on a per-source basis. 

• Continued communication with the client team to align on project goals and any source-specific limitations will be important as not all APIs will support the export of all required data and/or in incremental batches. It’s expected that building workarounds could be necessary.

Best Practices

Hashmap & Fivetran

Partnership

• Fivetran Advanced Partner and partnered for since 2018

• 35+ Fivetran Technical Certifications

• Partner Advisory Board

• MDSCON Sponsorship and Participation since 2020

Collaboration

• Tech client

  • Custom connectors for apps such as Gainsight, BambooHR, Peakon, and Bizible​

• Energy client

  • 2B rows from Fivetran into Snowflake including SQL Server, Salesforce, Pardot, Files, and Logs​

• Healthcare client

  • End to end Fivetran replication from on premise RDBMS to Snowflake with dbt transformations and job orchestration​

Content

• Learn from Case Studies

  • Richie Bachala: The Making of a Modern Data Leader at Sherwin-Williams​

• Listen to Hashmap on Tap Podcasts

  • #90 Lessons Learned in Building a Modern Data Stack with Erik Jones from Hyperscience

• Watch Videos

  • Extracting Facebook Insights API via Fivetran & Snowflake​

• Read Blog Posts

  • ​Hashmap on Fivetran on Medium

GET STARTED WITH HASHMAP

Let us know the Fivetran custom connectors you need to accelerate your data initiatives!

Connect Now >