Earlier than making architectural choices, it’s value revisiting the broader migration technique. In our earlier submit, we launched Databricks Skilled Companies’ strategy to complicated information warehouse migrations, highlighting the significance of early choices round technique and design. These foundational selections straight affect the goal platform’s implementation and structure.
We additionally launched two sequencing methods: ETL-first and BI-first. The BI-first strategy delivers fast worth by modernizing the consumption layer, whereas the ETL-first strategy focuses on upstream pipelines. Every has its place, relying on priorities.
On this submit, we discover one of the vital design selections: selecting between a Elevate-and-Shift or Modernization strategy. We clarify what every strategy entails, when to make use of it, and how you can merge them right into a hybrid strategy for long-term success on Databricks.
From technique to migration strategy: choosing the right path
After you’ve aligned on the broader migration technique—ETL-first or BI-first—the following main choice is how you can construction the migration. Do you replicate what exists, or reimagine it for the long run?
This architectural choice sometimes comes down to 2 core approaches:
Elevate-and-Shift: Transfer workloads as-is to speed up the migration
Modernization: Redesign the platform to unlock long-term effectivity and scale
The correct strategy will depend on your targets, constraints, and timeline. Beneath, we break down the tradeoffs of every and embody a hybrid mannequin that many organizations use to mix the perfect of each.
Elevate and shift migration
Elevate-and-Shift entails shifting your current information fashions and codebase to the brand new platform with minimal modifications. You don’t introduce new use circumstances, and the structure stays unchanged.
This strategy is interesting as a result of it’s simpler to scope, plan, and automate. Instruments like profilers and code analyzers assist measure workload patterns, complexity, and value, making it simpler to guage and execute.
Key advantages embody:
Predictable timelines
Automated tooling (e.g., code converters, reconciliation validators)
Sooner migration when going through deadlines or expiring licenses
For instance, code converters can mechanically deal with as much as 80% of scripts. Since performance stays the identical, validation and working queries on each techniques and evaluating outputs are simpler.
On Databricks, Elevate-and-Shift will get you off legacy platforms shortly whereas unlocking fast efficiency positive factors utilizing options like z-ordering and liquid clustering. After you could have migrated, your group can start incrementally modernizing the platform.
Modernize the migration sample
Modernizing, in distinction to Elevate-and-Shift, means constructing a brand new information platform in your goal system with out being constrained by your legacy structure. The main focus shifts from merely migrating current belongings to reimagining use circumstances and designing for future wants. As a substitute of mapping outdated optimizations, you implement greatest practices and the well-architected pillars of the lakehouse.
On an open lakehouse, this entails refactoring code and re-architecting information buildings to satisfy your group’s present and future scalability, efficiency, value, and functionality necessities, free from legacy limitations.
Tooling stays helpful, however extra for discovery and planning:
Profilers and code analyzers assist stock what that you must migrate
Code converters and reconciliation instruments play a minimal position, since this isn’t a direct code translation
This strategy is good when you could have versatile timelines and an outdated or overly complicated legacy system, usually with hundreds of tables and scripts. Whereas beginning recent can really feel sluggish and overwhelming, the long-term advantages are substantial: simplified structure, higher efficiency, and diminished upkeep overhead.
That mentioned, migrating hundreds of scripts usually means sustaining their upkeep complexity. If that appears daunting, take into account partnering with Databricks Skilled Companies or licensed migration consultants to assist information the planning and design section and guarantee a smoother path.
A hybrid strategy: carry and shift, after which modernize
One other strategy is a hybrid migration technique that balances velocity with long-term worth. You’d start with the Elevate-and-Shift strategy to eliminating their legacy platform as shortly as potential, particularly when going through pressing constraints like expiring licenses. Automation and repeatable tooling assist speed up this preliminary section and cut back threat throughout execution.
You possibly can transfer into the modernization section after you migrate your workloads to Databricks.
Within the hybrid strategy, you:
Combine new and trendy information sources
Implement a knowledge product technique
Allow superior analytics, AI, and new use circumstances that drive enterprise choices
This section usually requires architectural updates however permits you to evolve step by step. With a hybrid technique, you don’t should modernize all the things on day one—you construct on a steady basis whereas aligning with future necessities.
If you happen to’re pursuing this strategy, Databricks Skilled Companies and authorized companions will help information your roadmap, guaranteeing a clean transition and a future-ready structure.
Our viewpoint
Choosing a migration strategy is just not a one-size-fits-all. The commonest strategy is a hybrid migration:
Create a migration manufacturing unit that leverages automation instruments.
Elevate-and-shift the vast majority of the codebase.
Allow out-of-the-box optimizations, resembling z-ordering and liquid clusteringto start out your modernization effort.
Databricks can act as your main information warehouse. For instance, you possibly can migrate saved procedures to notebooks and use SQL Scripting for scalability and AI integration with out leaving the consolation of SQL. Migrating Transact-SQL to another cloud information warehouse requires an identical effort to migrating that Transact-SQL to a pocket book with Python code wrapped round your SQL performance. The good thing about utilizing a pocket book is that you simply additionally get flexibility and an important improvement expertise.
What to do subsequent
Able to modernize your information warehouse? Obtain our eBook, “Remodeling Legacy Information Warehouses: A Strategic Migration Blueprint,” for detailed methods and greatest practices that guarantee a low-risk transition to the Databricks Information Intelligence Platform.
