We’re excited to announce the Public Preview for Apache IcebergTM help in Databricks, unlocking the complete Apache Iceberg and Delta Lake ecosystems with Unity Catalog. This Preview introduces two new options to Unity Catalog. First, now you can learn and write Managed Iceberg tables utilizing Databricks or exterior Iceberg engines by way of Unity Catalog’s Iceberg REST Catalog API. Powered by Predictive Optimizationthese tables robotically run superior desk operations, together with Liquid Clustering, to ship out-of-box, quick question efficiency and storage effectivity. Managed Iceberg tables are additionally built-in with superior options throughout the Databricks platform, together with DBSQL, Mosaic AI, Delta Sharing, and MVs. Second, as a part of Lakehouse Federation, Unity Catalog now allows you to seamlessly entry and govern Iceberg tables managed by international catalogs akin to AWS Glue, Hive Metastores, and Snowflake Horizon Catalog.
With these new options, you’ll be able to hook up with Unity Catalog from any engine and entry all of your knowledge, throughout catalogs and no matter format, breaking knowledge silos and resolving ecosystem incompatibilities. On this weblog, we are going to cowl:
Figuring out new knowledge silos
Utilizing Unity Catalog as a completely open Iceberg catalog
Extending UC governance to all the Lakehouse
Our imaginative and prescient for the way forward for open desk codecs
The brand new knowledge silos
New knowledge silos have emerged alongside two foundational parts of the Lakehouse: open desk codecs and knowledge catalogs. Open desk codecs allow ACID transactions on knowledge saved in object storage. Delta Lake and Apache Iceberg, the 2 main open desk codecs, developed connector ecosystems throughout a variety of open supply frameworks and industrial platforms. Nevertheless, hottest platforms solely adopted one of many two requirements, forcing clients to decide on engines when selecting a format.
Catalogs introduce extra challenges. A core duty of a catalog is managing a desk’s present metadata information throughout writers and readers. Nevertheless, some catalogs limit what engines are allowed to put in writing them. Even should you handle to retailer all of your knowledge in a format supported by all of your engines, you should still not be capable to use your chosen engine as a result of it can not hook up with your catalog. This vendor lock-in forces clients to fragment knowledge discovery and governance throughout disparate catalogs.
Over the following two sections, we are going to cowl how Unity Catalog makes use of open requirements and catalog federation to resolve format and catalog incompatibilities.
A Totally Open Iceberg Catalog
Unity Catalog breaks format silos by open requirements. Now in Public Preview, you should use Databricks and exterior engines to put in writing Iceberg tables managed by Unity Catalog. Managed Iceberg tables are totally open to all the Iceberg ecosystem by way of Unity Catalog’s implementation of the Iceberg REST Catalog APIs. The REST Catalog is an open API specification that gives a regular interface for interacting with Iceberg tables. Unity Catalog was an early adopter of the REST Catalog, first launching help in 2023. This Preview builds on that basis. Now, nearly any Iceberg shopper suitable with the REST spec, akin to Apache Spark™, Apache Flink, or Trino can learn and write to Unity Catalog.
We plan to retailer all our knowledge in an open format and desire a single catalog that may hook up with all of the instruments we use. Unity Catalog permits us to put in writing Iceberg tables which are totally open to any Iceberg shopper, unlocking all the Lakehouse ecosystem and future proofing our structure.
-Hen Ben-Hemo, Information Platform Architect
With Managed Iceberg, you’ll be able to convey Unity Catalog governance to the Iceberg ecosystem even amongst OSS instruments like PyIceberg that don’t natively help authorization. Unity Catalog permits you to create knowledge pipelines that span the complete Lakehouse ecosystem. For instance, Apache Iceberg affords a fashionable sink connector for writing from Kafka to Iceberg tables. You should use Kafka Join to put in writing Iceberg tables to Unity Catalog and downstream use Databricks’s best-in-class price-performance for ETL, knowledge warehousing, and machine studying capabilities.
All Managed Tables robotically ship best-in-class learn efficiency and storage optimization utilizing Predictive Optimization. Predictive Optimization robotically expires previous snapshots, deletes unreferenced information, and incrementally clusters your knowledge utilizing Liquid Clustering. In our instance utilizing Kafka, this prevents efficiency degradation generally brought on by the proliferation of small information. You possibly can maintain your Iceberg tables wholesome and performant with out the effort of manually managing your personal desk upkeep.
Managed Iceberg tables are built-in with the Databricks platform, permitting you to leverage these tables with superior platform options akin to DBSQL, Mosaic AI, Delta Sharing, and MVs. Past Databricks, Unity Catalog helps a associate ecosystem to securely land knowledge in Iceberg utilizing exterior instruments. For instance, Redpanda ingests streaming knowledge produced to Kafka subjects by Unity catalog’s Iceberg REST Catalog API:
With Unity Catalog Managed Iceberg Tables and the Iceberg REST Catalog, Redpanda can now stream the most important, most demanding Kafka workloads straight into Iceberg tables which are optimized by Unity Catalog, unlocking out-of-box discoverability and quick question efficiency on arbitrary streams. With push-button configuration, all real-time streaming knowledge is now totally obtainable to the Iceberg ecosystem, so clients can believe that their structure is constructed to final, irrespective of how their stack evolves.
— Matthew Schumpert, Head of Product, Platform
We’re excited to have the next launch companions on board: Atlan, Buf, CelerData, Clickhouse, dbt Labs, dltHub, Fivetran, Informatica, PuppyGraph, Redpanda, RisingWave, StreamNative, and extra.
The Lakehouse Catalog
With Unity Catalog, you’ll be able to interoperate not solely throughout desk codecs, but in addition throughout catalogs. Now additionally in Public Preview, you’ll be able to seamlessly question and govern Iceberg tables managed by exterior catalogs akin to AWS Glue, Hive Metastores, and Snowflake Horizon Catalog. Extending Hive Metastore and AWS Glue Federation, these connectors will let you mount whole catalogs inside Unity Catalog, making a unified interface for knowledge discovery and governance.
Federation offers a seamless integration to leverage Unity Catalog’s superior options on Iceberg tables managed by international catalogs. You should use Databricks’ fine-grained entry controls, lineage, and auditing on all of your knowledge, throughout catalogs and no matter format.
Unity Catalog permits Rippling ML engineers and Information Scientists to seamlessly entry Iceberg tables in current OLAP warehouses with zero copy. This helps us decrease prices, create constant sources of reality, and cut back latency of information refresh — all whereas sustaining excessive requirements on knowledge entry and privateness throughout all the knowledge lifecycle.
— Albert Strasheim, Chief Know-how Officer
With federation, Unity Catalog can govern the whole thing of your Lakehouse – throughout all of your tables, AI fashions, information, notebooks, and dashboards.
The Way forward for Desk Codecs
Unity Catalog is pushing the business nearer to realizing the simplicity, flexibility, and decrease prices of the open knowledge lakehouse. At Databricks, we imagine that we are able to advance the business even additional – with a single, unified open desk format. Delta Lake and Apache Iceberg share a lot of the identical design, however refined variations trigger massive incompatibilities for purchasers. To resolve these shared issues, the Delta and Apache Iceberg communities are aligning ideas and contributions, unifying the Lakehouse ecosystem.
Iceberg v3 is a significant step towards this imaginative and prescient. Iceberg v3 consists of key options like Deletion Vectors, Variant knowledge sort, Row IDsand geospatial knowledge varieties that share equivalent implementations in Delta Lake. These enhancements allow you to maneuver knowledge and delete information between codecs simply, with out rewriting petabytes of information.
In future Delta Lake and Apache Iceberg releases, we wish to construct on this basis in order that Delta and Iceberg shoppers can use the identical metadata and thus, can share tables straight. With these investments, clients can understand the unique objective of an open knowledge lakehouse – a completely built-in platform for knowledge and AI on a single copy of information.
Managed and Overseas Iceberg tables at the moment are obtainable in Public Preview. Try our documentation to get began! Replay our bulletins at Information and AI Summit on June 9-12, 2025 to study extra about our latest Iceberg options and the way forward for open desk codecs.
