HASHMAP SOLUTION PROFILE

GLOBAL CONSUMER SERVICES & HEALTH MANAGEMENT COMPANY

Improving Automation & Security with Snowflake

CHALLENGE
  • Build an open-source framework for acquiring and transforming data from a variety of relational sources to persist into Snowflake.

  • Develop a configurable open source library for processing DDL (Data Definition Language) and DML (Data Manipulation Language) events published on Kafka topics for automatic propagation of the DDL/DML changes to mapped target tables in Snowflake.

OUTCOME
  • Created an end-to-end data acquisition, transformation, and persistence layer to enable change data capture and automation across a variety of database operations

  • Designed pre-hooks to detect schema changes from Confluent Schema registry 

  • Created processing flows for data replication using Snowflake Streams, Tasks, & Stored Procedures to publish AVRO events configured to Snowflake target tables

  • Configurated a module for source & target table column mappings along with auditing and traceability 

  • Set up Snowflake configuration to ensure backup(s) when DDL changes on completed tables to provide a trail of the changes performed

APPROACH
  • Discover - Collaborated with key stakeholders to develop an overall architecture design

  • Design - Supplied documentation for overall architecture, incremental flows, data models, audit/traceability, etc.

  • Develop - Built new coding artifacts for github repository, validated solution, and implemented monitoring and tracking metrics

  • Deliver - Provided performance and functional testing of the solution and open source repository with necessary documentation on Github

SOLUTION
  • Snowflake 

  • Confluent Kafka

  • Debezium

  • Avro Messaging

  • Github

  • Hashmap Consulting Services 

    • Snowflake Cloud Data Strategy & Advisory

    • Snowflake Cloud Data Design, Engineering, & Architecture

    • Snowflake Optimization & Management