Blog / AI-SDLC
Snowflake to Databricks Migration: Architecture, Code, and Data
Nasssir Khan
Head of Product Marketing & GTM
TL;DR
A successful Snowflake to Databricks migration depends more on dependency discovery and validation than table movement.
SQL translation covers only part of the work. Stored procedures, orchestration, governance, and downstream consumers account for most of the migration effort.
Inventory should include Tasks, Streams, UDFs, dashboards, external stages, and permissions before code conversion begins.
Functional validation should compare business outcomes, not only row counts or schema definitions.
Migration projects benefit from structured engineering work that tracks architectural decisions, dependencies, and validation throughout execution.
Why Teams Consider Snowflake to Databricks Migration
A Snowflake to Databricks migration is the process of moving an organization's tables, transformation logic, and access controls off Snowflake's SQL warehouse and onto Databricks' lakehouse, where data lives in Delta Lake and permissions run through Unity Catalog. The harder question, once you have that definition, is why a team that already pays for a working warehouse decides to take this on, and the answer usually starts with a scheduling conflict nobody planned for: the same tables that feed a Tuesday morning revenue dashboard are now also feeding a model-training job that needs GPU compute Snowflake was never built to hand out.
Gartner's read on the broader shift backs this up: in its first Market Guide for Data Lakehouse Platforms, the firm concluded the lakehouse is now the architecture most enterprises are standardizing on for both reporting and AI workloads. That's an architecture decision, not a vendor swap, and it's why migration projects that start as "move the warehouse" usually end up touching orchestration, access control, and the BI layer as well.
The part that catches teams off guard isn't the decision. It's the discovery, as one Databricks Community thread on a 500-table Snowflake migration shows, that stored procedure conversion has no clean one-to-one path and needs test coverage before anyone trusts the output. Years of Tasks, Streams, dbt projects, ingestion jobs, and dashboard dependencies accumulate quietly, and none of that shows up when you count tables.
This piece walks through one migration as a working example: dependency discovery, code conversion, governance mapping, and production cutover, in that order, with the failure modes that show up at each stage.
Table counts tell you almost nothing about what a migration really costs. The objects that break a timeline live outside the data model entirely: in scheduling logic, access grants, and the notebooks nobody remembers writing.
Inventory Misses Hidden Dependencies
Most inventories start with SHOW TABLES and stop there. That misses external stages, Tasks, Streams, UDFs, ad hoc notebooks, scheduled pipelines, and the role grants layered on top of all of it over several years. A table with three consumers and a table with thirty look identical in a row count, but they carry very different migration risk.
The gap shows up worst with orchestration. A Snowflake Task chain that fires a Stream-triggered load, which in turn kicks off a stored procedure, which then refreshes a materialized view a dashboard depends on, is a single dependency chain that a table-only inventory will never surface. Query Snowflake's ACCOUNT_USAGE schema for object access history and task execution logs before you touch a migration plan; it's slower than eyeballing a schema diagram, but it's the only way to catch consumers that exist in someone's forgotten Airflow DAG rather than in the warehouse itself.
A migration inventory that lists only tables gives a false sense of completeness. The objects that extend beyond the warehouse, including schedulers, stored procedures, dashboards, and service accounts, determine how much engineering work the migration really requires.
SoftwareForge's Living Specifications interface showing versioned specifications connected to architecture and Work Orders. Readers should pay attention to how architectural intent, implementation work, and governance remain connected instead of existing as separate project documents. During a migration, this reduces the number of undocumented assumptions engineers have to rediscover midway through execution.
SQL Conversion Stops Being a SQL Problem
Most analytical SQL translates cleanly between Snowflake and Databricks SQL. The 10 to 15% that doesn't, semi-structured queries, time-zone arithmetic, recursive CTEs written against Snowflake-specific syntax, absorbs a disproportionate share of the schedule because each case needs individual review rather than a find-and-replace pass.
Stored procedures are worse. Databricks has no direct equivalent for a Snowflake JavaScript stored procedure, so the logic gets rewritten as PySpark, SQL scripting, or a Delta Live Tables pipeline, and rewriting means re-testing every branch and error path, not just the happy path. Exception handling is where this bites hardest: Snowflake's JavaScript procedures lean on try/catch blocks that reference an error object with specific fields, and that pattern needs a deliberate equivalent on the other side, not a syntax swap.
Snowflake's documented approach to procedure-level error handling shows why this logic can't be mechanically translated: it depends on catching a specific error object and branching on its fields, a control-flow pattern with no line-for-line Spark equivalent.
CREATE OR REPLACE PROCEDURE broken() |
That error object (err.code, err.state, err.message, err.stackTraceTxt) has no direct match in a Databricks notebook or SQL script. Whoever ports this procedure has to decide, table by table, whether to catch exceptions at the Spark job level, the workflow level, or inside a Python wrapper, and each choice changes how failures get logged and retried downstream.
Converting complex procedural logic requires a shift in engineering philosophy rather than a translation tool. The diagram above illustrates how an integrated JavaScript exception block maps to a distributed cloud environment.
Validation Fails Long Before Production Cutover
Migration success isn't established by matching row counts between Snowflake and Databricks. Business users consume reports, machine learning features, operational dashboards, and application APIs. Every downstream consumer expects identical business meaning even when execution engines differ underneath.
Validation therefore expands into several layers. Engineers compare transformed datasets, verify aggregate calculations, inspect execution plans, validate dashboard outputs, review incremental loading behavior, and confirm that scheduled workloads execute in the expected order. Regression testing also extends beyond analytics because application services sometimes depend on warehouse queries directly.
Large migrations frequently introduce subtle differences instead of catastrophic failures. Duplicate key handling, timestamp precision, floating-point calculations, NULL ordering, and merge semantics produce discrepancies that remain invisible until production workloads execute against live data. Engineers who completed an enterprise migration involving hundreds of dbt models reported spending considerable effort validating functional equivalence rather than translating SQL syntax alone.
Validation also benefits from documenting every architectural decision alongside implementation work. SoftwareForge positions modernization around persistent project context, versioned specifications, and governed execution so engineering teams retain architectural intent while migration progresses through multiple implementation phases.
Converting Transformation Logic Without Rewriting Every Pipeline
Most transformation code doesn't need a full rebuild. It needs a translation layer that respects the orchestration assumptions baked into the original pipeline, and those assumptions are usually invisible until something breaks on the new platform.
Stored Procedures Rarely Translate One-to-One
Snowflake stored procedures frequently coordinate more than SQL execution. JavaScript procedures encapsulate branching logic, dynamic SQL generation, exception handling, temporary objects, audit logging, and orchestration decisions that have evolved over multiple releases. Translating the SQL statements addresses only part of the workload. One recurring challenge appears when procedures dynamically construct queries based on runtime metadata. Syntax conversion tools translate the resulting SQL, but they don't interpret why the procedure assembled those statements in the first place. Conditional execution paths, retry behavior, transaction boundaries, and custom error handling still require engineering review before the migrated workflow behaves consistently.
Differences in execution models also introduce subtle migration issues. Temporary object lifecycles, transaction scopes, session variables, and metadata access differ across platforms. A procedure that succeeds during isolated testing might produce inconsistent outputs when multiple production workloads execute simultaneously because concurrency assumptions changed. Migration teams frequently reduce risk by classifying procedures before conversion rather than treating them as one homogeneous workload. Utility procedures with limited branching are suitable for automated translation followed by validation. Procedures containing orchestration logic, dynamic execution, or extensive exception handling benefit from targeted engineering review before migration proceeds to production.
SoftwareForge's software modernization workflow illustrates how legacy application context is analyzed before implementation work begins. Focus on how architectural understanding precedes code transformation rather than following it. During warehouse modernization, preserving procedural intent becomes as important as translating syntax.
ETL Frameworks Carry Platform Assumptions
dbt models port to Databricks through the dbt-databricks adapter with real configuration changes, not just a target swap. Warehouse sizing assumptions baked into Snowflake-specific dbt macros, incremental strategies tuned for Snowflake's MERGE behavior, and connection details all need adapter-specific updates before a project runs cleanly on the new platform.
The dbt-databricks adapter repository documents the profile structure a project needs once it moves off Snowflake, and getting each field right (particularly catalog, which has no Snowflake equivalent since Databricks introduced the three-level Unity Catalog namespace) determines whether models resolve against the right governed objects on the first run.
your_profile_name: |
Orchestration outside dbt needs its own pass. Airflow DAGs built around Snowflake's warehouse auto-suspend behavior, external connectors wired to Snowflake-specific APIs, and scheduling logic that assumed near-instant warehouse resume times all carry platform assumptions that don't hold on Databricks clusters. This is where teams that treated the migration as a syntax exercise get surprised: the orchestration layer usually needs architectural redesign, not a find-and-replace on connection strings.
Generated Work Packages Reduce Manual Coordination
A migration epic that says "convert the ETL layer" doesn't tell an engineer what to do on a Monday morning. Breaking that epic into independently verifiable units, this dbt model, this Airflow task group, this notebook, with a defined acceptance check, is what keeps a multi-team migration from stalling in status meetings.
This is where a platform built for decomposing code and pipeline work into reviewable units earns its place, and it's worth being specific about scope: SoftwareForge's User Stories apply to the application and pipeline code around the migration, orchestration scripts, dbt project structure, and custom connectors, not to Snowflake-specific object scanning. Used that way, they turn a vague "convert the ETL layer" ticket into a set of scoped, auditable engineering tasks that reviewers can validate individually instead of trusting one large pull request.
Governance Changes During Snowflake to Databricks Migration
Snowflake's role-based access control and Unity Catalog's privilege model share vocabulary but not structure, and treating them as interchangeable is how teams end up granting broader access than they intended on day one.
Permission Models Require Explicit Mapping
Snowflake roles are a flat-ish hierarchy that can get tangled through role inheritance chains built up over years. Unity Catalog enforces a strict three-level namespace, catalog, schema, table, and privileges inherit downward through that structure by design, which means a grant at the catalog level reaches every object under it whether that was the intent or not.
Databricks' own setup guide shows the pattern for scoping this correctly, granting usage and select privileges at the schema level rather than the catalog level to avoid over-provisioning access to groups that only need read access to one part of a catalog.
GRANT USE CATALOG ON CATALOG <catalog-name> TO `<group-name>`; |
Service principals need the same scrutiny. A Snowflake service account with a broad role assigned three migrations ago often has no documented owner by the time anyone tries to map it to a Databricks service principal, and security validation belongs in the same test cycle as functional migration testing, not a separate audit that happens after the workload is already live.
Lineage Becomes Incomplete During Parallel Operation
Most migrations run Snowflake and Databricks in parallel for a stretch, sometimes weeks, sometimes months, and that coexistence period creates its own governance problem. The same transformation logic runs in two places, producing two datasets that are supposed to match but drift the moment someone patches a bug in only one location.
Lineage tools that tracked a table's origin cleanly in Snowflake often can't see across the boundary into Databricks, and the reverse is just as true early in a migration before Unity Catalog lineage tracking has full coverage. That gap matters most during an audit, when someone needs to show which system was the source of truth for a specific number on a specific date, and "both, depending on when you asked" isn't an answer that holds up.
SoftwareForge's architecture view illustrates persistent project context across modernization activities. Pay attention to how engineering artifacts remain connected rather than existing as isolated documentation. During phased migration, preserving lineage between implementation work and architectural decisions helps teams manage parallel execution with fewer undocumented dependencies.
Documentation Drifts Faster Than Migration Progress
Architecture decisions made in week two of a migration rarely survive unchanged to week twenty. A team picks a phased rollout order, then reprioritizes around a business deadline, then discovers a dependency that forces a different sequence, and a static design document written at kickoff is stale before the second phase starts.
Teams spanning multiple engineering groups get more value from a specification that evolves as decisions change than from a document someone has to remember to revise. SoftwareForge's Living Specifications apply this idea to architectural intent, preserving the reasoning behind architecture, dependency, and modernization decisions as implementation progresses. As orchestration scripts, dbt models, and connector code change throughout the migration, engineers can trace those implementation changes back to the architectural decisions that drove them instead of relying on a kickoff document written months earlier.
Production Cutover Without Freezing Analytics Delivery
Nobody gets to pause the business while a migration finishes, which means cutover has to happen underneath a workload that's still serving live dashboards and scheduled reports.
Dual Execution Increases Operational Complexity
Running Snowflake and Databricks simultaneously is the most common deployment strategy for enterprise migrations. The approach reduces deployment risk, yet it introduces another operational layer that requires continuous coordination until the legacy platform is retired. Both environments process overlapping datasets during this period. Scheduled transformations execute independently, dashboards point to different sources depending on migration status, and application teams validate outputs against production expectations. Any synchronization gap creates conflicting versions of business data, even when every pipeline executes successfully.
Rollback planning deserves equal attention. Switching reporting traffic back to Snowflake should require configuration changes rather than emergency engineering work. Teams that delay rollback planning until production deployment frequently discover hidden dependencies that were never exercised during testing. Business stakeholders also need clearly defined validation windows. Financial reports, executive dashboards, forecasting systems, and downstream applications shouldn't all move on the same day unless validation demonstrates functional equivalence across every dependency.
SoftwareForge's Work Orders interface shows modernization work divided into independently tracked engineering activities. Pay attention to how implementation tasks, review status, and architectural context remain connected throughout execution. During production cutover, this visibility helps teams coordinate migration activities without relying on disconnected spreadsheets or manually maintained project trackers.
Production deployment isn't a single switch from Snowflake to Databricks. Most enterprise teams keep both environments running while they validate workloads, verify reports, and preserve rollback options.
Cost Models Change After Compute Separation
Snowflake's virtual warehouses hide most of the sizing decision behind a T-shirt-size picker. Databricks gives more direct control over cluster configuration, which means more decisions land on the platform team: instance types, autoscaling bounds, Photon acceleration settings, and cache behavior all now sit inside the migration scope instead of behind a vendor abstraction.
Delta Lake's storage layer behaves differently from Snowflake's micro-partitioning, particularly around compaction and file sizing, and skipping that tuning step is a common reason early post-migration cost comparisons look worse than expected. Migration success measured only against infrastructure spend misses the larger shift: workload isolation lets teams size compute independently per job instead of sharing one warehouse's queue, which changes the unit economics in ways a simple credit-to-DBU comparison won't show.
Modernization Pipelines Need Repeatable Validation
A migration that succeeds once on a test dataset and fails on the next batch isn't done. Regression testing needs to run against every converted pipeline on a schedule, not just at the moment of cutover, because a schema change three weeks later can silently break a conversion that passed its original test.
The same discipline that applies to code review generally, a repeatable checklist instead of an ad hoc pass, holds up better for pipeline validation than a one-time audit. Where a platform like SoftwareForge fits into that discipline is narrow but real: applying structured, auditable review to the orchestration and transformation code itself, distinct from the data reconciliation work that still has to happen against Snowflake and Databricks directly.
Choosing the Right Snowflake to Databricks Migration Strategy
Snowflake to Databricks migration succeeds when a team treats it as an architecture change, not a database swap. Dependency discovery, code conversion, governance mapping, and validation each carry their own failure modes, and skipping depth in any one of them doesn't cause an early failure. It shows up months later, usually during a cutover window nobody planned to reopen.
This article walked through that sequence on one working example: inventories that miss Tasks, Streams, and orchestration chains hiding behind a clean table count; stored procedures and dbt models that need engineering review rather than a syntax pass; Unity Catalog's schema-level permission model replacing Snowflake's role inheritance; and a production cutover built around parallel execution and a rollback plan tested against a real failure, not just written down. None of these stages substitutes for the others. Skipping validation because conversion looks clean is how a migration passes every test except the one that matters, production, under real concurrent load. Before setting a cutover date, confirm the dependency graph, the permission mapping, and the validation coverage are complete, not just the row counts.
FAQs
How Long Does a Snowflake to Databricks Migration Usually Take?
Migration duration depends on workload complexity rather than database size. A platform with extensive stored procedures, orchestration workflows, governance policies, and downstream integrations requires substantially more validation than one consisting primarily of analytical tables.
Can dbt Projects Be Reused During the Snowflake to Databricks Migration?
Many dbt models transfer with limited changes, but adapter-specific macros, incremental models, warehouse configuration, and custom packages should be reviewed before production deployment. Functional validation remains necessary even when SQL compiles successfully.
What Is the Hardest Part of Migrating Snowflake Stored Procedures?
The largest challenge is preserving procedural behavior rather than translating SQL syntax. JavaScript procedures frequently include dynamic SQL, branching logic, exception handling, and transaction management that require engineering review beyond automated conversion.
Should Snowflake and Databricks Run in Parallel During Migration?
Parallel execution reduces deployment risk by allowing workload validation under production conditions before retiring Snowflake. The tradeoff is temporary operational complexity because governance, lineage, synchronization, and monitoring must span both environments until migration completes.
