shutterstock_389742484.png

Custom Connector Build Accelerator for Fivetran

Fivetran Cloud Function Connector +
Hashmap SAM Application Accelerator =
Accelerated Custom Connector Build & Save $

HashmapLogoBlack_edited_edited_edited_ed
fivetran_white_logo2.png

Fivetran Function Connector

A Fivetran function connector enables building a custom data connector as an extension of Fivetran.

fivetran.com/docs/functions

Hashmap_Logo_Hash_Only_White.png

Hashmap SAM Application Accelerator

Hashmap uses AWS SAM template definitions (an IaC approach) to streamline the provisioning of resources in AWS for a Fivetran custom connector.

Lambda functions | Roles | Policies

4422182_management_productivity_time_time is money_icon - white.png

Accelerate Your Build and Save $

With our pre-built AWS SAM template definitions, time to value for Fivetran Custom Connectors using Cloud Functions is accelerated by ~ 50%

Hashmap_Fivetran_Custom_Connectors 2.png

Here's our Flow for a Custom Fivetran Connector 

Validate & Plan

  • Assessing, analyzing, and validating (access) to the data sources

  • Design project execution plan, identify user stories, create project backlog, and prioritize user stories into sprint logs

  • User and service account provisioning

  • Sprint iterations to build connector/s

Connector Build

  • Leveraging Python app template, develop AWS Lambda functions and configure as Fivetran ‘Function’ connectors

  • Leveraging Hashmap SAM templates, provision services and features as required such as Lambda Functions, IAM

  • Promote and validate the functions and associated Fivetran connectors in QA environment

  • Deploy (productionalize) to PROD environment and perform data quality tests to confirm

Deliverables

  • AWS Lambda function app (preferably in a Git controlled repository)

  • Additional AWS resources (IaC code/templates) (in a Git controlled repository)

  • Test scripts used to validate data quality post extract and load into Snowflake

  • Project documentation

Dependencies

  • Client Business SME provides input for post-load DQ validation tests

  • Client Tech Admin SME configures user/service accounts on AWS,
    Fivetran and Snowflake (as required)

  • Project stakeholder to assess and provide sign-off on the connectors

Assumptions

  • Source systems are identified and access to vendor-specific APIs are already set up and available for use with Fivetran Cloud functions 

  • Target Environment(s) are set up within Snowflake (landing database/schema setup and roles configured)

  • Transformations (if required) will be performed by the client team post loading

Considerations

  • API sources tend to have their own limitations (filter/watermark column not available to perform incremental syncs, limits on number of API calls, etc.) that might require more detailed development to overcome. Any limitations will be handled on a per-source basis. 

  • Continued communication with the client team to align on project goals and any source-specific limitations will be important as not all APIs will support the export of all required data and/or in incremental batches. It’s expected that building workarounds could be necessary.

Best Practices

Fivetran Connector Best Practices.png

Hashmap & Fivetran

fivetran_white_logo2.png

Partnership

  • Fivetran Advanced Partner and partnered for since 2018

  • 35+ Fivetran Technical Certifications

  • Partner Advisory Board

  • MDSCON Sponsorship and Participation since 2020

Hashmap_Logo_Hash_Only_White.png

Collaboration

  • Tech client

    • Custom connectors for apps such as Gainsight, BambooHR, Peakon, and Bizible​

  • Energy client

    • 2B rows from Fivetran into Snowflake including SQL Server, Salesforce, Pardot, Files, and Logs​

  • Healthcare client

    • End to end Fivetran replication from on premise RDBMS to Snowflake with dbt transformations and job orchestration​

content_white.png

Content

GET STARTED WITH HASHMAP

Let us know the Fivetran custom connectors you need to accelerate your data initiatives!