
Custom Connector Build Accelerator for Fivetran
Fivetran Cloud Function Connector +
Hashmap SAM Application Accelerator =
Accelerated Custom Connector Build & Save $


Fivetran Function Connector
A Fivetran function connector enables building a custom data connector as an extension of Fivetran.
fivetran.com/docs/functions

Hashmap SAM Application Accelerator
Hashmap uses AWS SAM template definitions (an IaC approach) to streamline the provisioning of resources in AWS for a Fivetran custom connector.
Lambda functions | Roles | Policies

Accelerate Your Build and Save $
With our pre-built AWS SAM template definitions, time to value for Fivetran Custom Connectors using Cloud Functions is accelerated by ~ 50%

Here's our Flow for a Custom Fivetran Connector
Validate & Plan
-
Assessing, analyzing, and validating (access) to the data sources
-
Design project execution plan, identify user stories, create project backlog, and prioritize user stories into sprint logs
-
User and service account provisioning
-
Sprint iterations to build connector/s
Connector Build
-
Leveraging Python app template, develop AWS Lambda functions and configure as Fivetran ‘Function’ connectors
-
Leveraging Hashmap SAM templates, provision services and features as required such as Lambda Functions, IAM
-
Promote and validate the functions and associated Fivetran connectors in QA environment
-
Deploy (productionalize) to PROD environment and perform data quality tests to confirm
Deliverables
-
AWS Lambda function app (preferably in a Git controlled repository)
-
Additional AWS resources (IaC code/templates) (in a Git controlled repository)
-
Test scripts used to validate data quality post extract and load into Snowflake
-
Project documentation
Dependencies
-
Client Business SME provides input for post-load DQ validation tests
-
Client Tech Admin SME configures user/service accounts on AWS,
Fivetran and Snowflake (as required) -
Project stakeholder to assess and provide sign-off on the connectors
Assumptions
-
Source systems are identified and access to vendor-specific APIs are already set up and available for use with Fivetran Cloud functions
-
Target Environment(s) are set up within Snowflake (landing database/schema setup and roles configured)
-
Transformations (if required) will be performed by the client team post loading
Considerations
-
API sources tend to have their own limitations (filter/watermark column not available to perform incremental syncs, limits on number of API calls, etc.) that might require more detailed development to overcome. Any limitations will be handled on a per-source basis.
-
Continued communication with the client team to align on project goals and any source-specific limitations will be important as not all APIs will support the export of all required data and/or in incremental batches. It’s expected that building workarounds could be necessary.
Best Practices

Hashmap & Fivetran

Partnership
-
Fivetran Advanced Partner and partnered for since 2018
-
35+ Fivetran Technical Certifications
-
Partner Advisory Board
-
MDSCON Sponsorship and Participation since 2020

Collaboration
-
Tech client
-
Custom connectors for apps such as Gainsight, BambooHR, Peakon, and Bizible
-
-
Energy client
-
2B rows from Fivetran into Snowflake including SQL Server, Salesforce, Pardot, Files, and Logs
-
-
Healthcare client
-
End to end Fivetran replication from on premise RDBMS to Snowflake with dbt transformations and job orchestration
-

Content
-
Learn from Case Studies
-
Listen to Hashmap on Tap Podcasts
-
#90 Lessons Learned in Building a Modern Data Stack with Erik Jones from Hyperscience
-
-
Watch Videos
-
Extracting Facebook Insights API via Fivetran & Snowflake
-
-
Read Blog Posts
-
Hashmap on Fivetran on Medium
-