Transforming Data Engineering with Databricks for a Leading Media and Entertainment Company

Overview

This case study highlights the migration of data engineering pipelines from a combination of AWS EMR and Snowflake architecture to a unified Databricks platform. The migration aimed to unify the data engineering approach across the company, reduce costs, and provide a single interface for data processing.

Objectives

Unify the implementation of data engineering pipelines across the organization.
Achieve significant cost savings by optimizing storage and compute resources.
Consolidate data engineering workloads onto a single platform for ease of use.

About Client

The client is an American multinational media and entertainment company, is renowned for its extensive content library and pioneering data strategies. As a leader in cable television and streaming services, They offer a diverse array of original programming, feature films, documentaries, and more. For over 30 years, it has maintained a strong presence in the industry with a workforce of 35,000, operating across multiple countries and establishing itself as a true global MNC.

Industry

Media and Entertainment

Years in Business

30+ years

Company Size

35,000

Geographical Presence

MNC

The Challenge

High Storage and Compute Costs: Storing unstructured data in Snowflake proved 10x costlier, with Bronze and Silver layers increasing costs
Performance Issues : Complicated queries, especially self-joins, caused performance bottlenecks and the lack of transparency hindered optimization.
Limited Flexibility for ML Workloads : Snowflake's architecture posed challenges for running machine learning tasks.
Inefficient Resource Management : The inability to finely control compute resources led to operational inefficiencies.
Data Consistency : Ensuring that data across Snowflake and Databricks remained identical during the migration was crucial to avoid disruptions.

Solutions

Phase 1: Bronze and Silver Layers : Data pipelines that previously wrote to Snowflake were redirected to write to Databricks, using External Delta Tables on Amazon S3 to optimize storage costs.
Phase 2: Gold Layer : To maintain client application stability, data was written to both Snowflake and Databricks during the transition period. This dual-write strategy ensured no immediate impact on analytics and BI applications.
Client Migration Strategy : They enabled clients to validate Gold layer data in Databricks before fully migrating applications. Upon successful migration, Snowflake was decommissioned.

Technology Stack

Old Architecture

Migration Architecture and Design

Impact

Cost Reduction

This migration cut costs up to 40% and lowered infrastructure and orchestration overhead and it also simplified data pipelines and reduced friction.

Unified Data Engineering Workflows

Replacing AWS EMR and Airflow with Databricks workflow cut pipeline execution times and simplified management within a unified ecosystem.

Enhanced Performance

Fine-tuned resource control and optimized query performance addressed prior performance challenges.

Improved Client Experience

Consistent and reliable data availability through phased migration and dual-write strategy.

Summary

The migration from Snowflake to Databricks was indeed a major shift, as it improved data engineering by centralizing workloads, cutting storage costs, and boosting performance. Databrick’s platform also enabled scalable, efficient data processing, by laying the groundwork for innovation across the company's data pipelines.

Download the case study here!

You will soon receive a download link via email.