Willis Towers Watson (WTW) is a multinational firm that gives a variety of companies in business insurance coverage brokerage, threat administration, worker advantages, and actuarial evaluation—serving 91% of Fortune World 500 firms. WTW’s Work and Rewards division delivers data-driven insights, expertise options, and companies to assist purchasers’ hiring and retention choices by way of useful market-based knowledge.
Our core enterprise is dependent upon strong knowledge transformation and governance capabilities. At its coronary heart is our proprietary calculation engine, which powers our wage survey experiences. Not too long ago, we started a strategic migration from SQL Server on Azure VMs to Azure Databricks, unlocking important enterprise benefits. This shift has accelerated report technology by 10x, decreased our knowledge environments by 50%, and lower storage prices to one-third of our earlier SQL Server bills.
Databricks Unity Catalog has been a key enabler of this transformation, redefining our method to knowledge governance and administration. On this weblog, we’ll share the challenges we encountered, how Databricks and Unity Catalog helped us overcome them, and the impression this transformation has had on our enterprise.
Challenges: Scaling, governance, and efficiency Bottlenecks
Earlier than implementing Databricks and Unity Catalog, we confronted a number of technical and organizational challenges stemming from limitations in our current tech stack.
Restricted scalability driving prices, gradual ETL, and delayed time to market: Our report calculation enterprise is seasonal, with distinct peak intervals for knowledge acquisition and report technology. Nevertheless, our legacy database server lacked scalability, forcing us to provision giant database cases year-round—leading to important idle time and pointless prices. Scaling internet and app servers throughout peak intervals required customized logic in our utility code. WTW’s largest experiences typically took between 10 to 36 hours to generate, particularly for our most important surveys. These lengthy runtimes created friction in buyer relationships and added operational prices by slowing down different processes.
Rigid schema design in legacy SQL Server implementation limiting agility: Our use of inflexible relational knowledge fashions made it tough to adapt to evolving knowledge necessities. This inflexibility led to excessive improvement and upkeep prices.
Knowledge and atmosphere duplication complicating governance: We maintained knowledge throughout greater than 5 environments to satisfy processing and regional compliance wants, incurring over $300K yearly. To make sure quick knowledge retrieval, we relied on roughly 70 TB of P40 storage disks in SQL Server on Azure VMs—including to our infrastructure bills.
Compliance challenges in managing regional knowledge: As a world group, WTW should adhere to regional knowledge privateness and compliance laws. As a result of limitations in our prior system, we selected to not accumulate personally identifiable info (PII) to keep away from elevated complexity. Moreover, all knowledge was saved in a single area, limiting our skill to satisfy localized compliance necessities.
Inadequate knowledge lineage and auditing: We lacked automated instruments to trace knowledge lineage and audit modifications—each vital for troubleshooting and understanding the downstream impression of knowledge modifications.
Enhancements with Databricks Knowledge Intelligence Platform and Unity Catalog
In our next-generation system constructed on Databricks and Unity Catalog, we’re already realizing a number of advantages that allow our enterprise to scale extra successfully.
10x sooner report technology unlocks enterprise progress: Reviews that beforehand took 10 hours to generate now full in below 50 minutes—a 10x enchancment in calculation time. Most experiences are seeing 5x to 20x efficiency positive aspects. This acceleration is particularly useful for our largest experiences, which used to take over a day to course of. Quicker report technology decreases time-to-market, unlocks upsell alternatives by delivering insights to purchasers extra rapidly and ceaselessly, and frees up our group to deal with new consumer engagements.
30% value financial savings by way of Unity Catalog’s environment friendly knowledge administration: By centralizing knowledge entry with Unity Catalog, we have decreased storage and infrastructure prices by 30%. Beforehand, our system relied on six separate environments to satisfy compliance and operational wants. Right this moment, a single Databricks workspace can seamlessly entry a number of catalogs throughout areas, eliminating duplication and simplifying governance. As well as, storing knowledge in Unity Catalog managed tables—with built-in compression and efficiency optimizations—has lower storage prices to one-third of our earlier SQL Server bills. These managed tables additionally streamline operations by robotically optimizing desk layouts primarily based on question patterns.
Improved compliance and knowledge residency capabilities: We are able to now meet privateness necessities from the European Union, Germany, China, California, and different jurisdictions utilizing Unity Catalog’s versatile structure, paired with Databricks’ international availability throughout Azure areas. Storing knowledge nearer to the place it’s accessed has the potential to cut back cloud egress prices and enhance report efficiency. Unity Catalog’s fine-grained entry controls and integration with Entra ID teams additionally give us higher management over PII—benefiting each our compliance posture and our prospects.
Higher auditability and lineage elevated developer productiveness by 33%: Unity Catalog’s lineage options, together with Databricks’ orchestration instruments and AI capabilities, allow higher exploratory knowledge evaluation and deeper understanding of our datasets—particularly as new experiences are developed. Primarily based on our estimates, experiences that beforehand took as much as three weeks to develop now take two weeks or much less.
Databricks coaching accelerated group ramp-up: The provision of each free and paid Databricks coaching packages enabled groups with no prior expertise in Databricks or Python to develop into productive inside 2–3 months. By the top of eight months, most builders reached roughly 75% proficiency.
Wanting ahead
The migration to Databricks and Unity Catalog has been a game-changer for the Work & Rewards division at Willis Towers Watson. We’ve streamlined knowledge governance, improved compliance, decreased prices, and dramatically accelerated report technology.
Wanting forward, we plan to harness Databricks’ AI-powered capabilities—comparable to AI/BI Genie and automatic ML options—to construct new data-driven merchandise. With a modernized knowledge infrastructure in place, we’re well-positioned to drive innovation, improve buyer experiences, and unlock new income alternatives.
