Be part of the occasion trusted by enterprise leaders for practically 20 years. VB Remodel brings collectively the folks constructing actual enterprise AI technique. Study extra
Gaining visibility — and, finally, insights — into enterprise cloud property is rising ever tougher.
Cloud estates are sprawling and fragmented, and stock capabilities in current instruments might be slender and unintuitive, separating parts like value and safety information into disconnected platforms with restricted flexibility.
Cloud governance firm CloudQuery is positioning itself to deal with this drawback by centralizing cloud property, safety metadata and value in a single place, and making it accessible by straightforward, built-in SQL queries and studies. The corporate is taking a developer-first method to cloud governance, pulling information from 60-plus sources — together with AWS, GCP, Azure, Okta and Wiz — right into a single, queryable information warehouse.
The corporate is now saying a $16 million funding spherical led by Partech to additional scale its method to cloud visibility.
“The largest problem with current instruments is that they’re siloed — one for safety, one for value, one for asset stock — making it laborious to get a unified view throughout domains,” CQ founder Yevgeny Pats advised VentureBeat. “Even easy questions like ‘What EBS quantity is hooked up to an EC2 that’s turned off? are laborious to reply with out stitching collectively a number of instruments.”
CloudQuery below the hood
CloudQuery makes use of two key applied sciences below the hood: Knowledge warehouse and open-source database ClickHouse and the Apache Arrow framework for creating information analytics functions.
This high-performance plugin structure in-built Go connects on to APIs like AWS, Azure, Google Cloud Platform (GCP) and plenty of different platforms pulling in configuration, safety, and value metadata. The platform constantly syncs information from dozens of cloud suppliers and companies right into a normalized, centralized asset stock.
“We place a robust emphasis on information accuracy and freshness, syncing at excessive frequency to make sure groups are working with probably the most dependable, up-to-date data,” mentioned Pats.
That information, he defined, is structured relationally to energy CloudQuery’s SQL engine and built-in studies, in order that groups can have full flexibility with out counting on black-box instruments.
The corporate additionally “selectively” makes use of giant language fashions (LLMs) for pure language querying, SQL era and suggestions, “however at all times on high of a basis of correct, clear information,” mentioned Pats. He identified that as a result of AI understands SQL properly, instruments like Claude and OpenAI can create custom-made studies and evaluation in plain English.
Taking a developer-first method is important, mentioned Pats, as a result of builders are finally those constructing, working and securing right now’s cloud infrastructure. Nonetheless, many cloud visibility instruments had been constructed for top-down governance, not for the folks truly within the trenches.
“Whenever you put builders first, with accessible information, versatile APIs and native language like SQL, you empower them to maneuver sooner, catch points earlier and construct extra securely,” he mentioned.
Prospects are discovering methods to make use of CloudQuery past asset stock. “Many begin with visibility, then shortly develop into use instances like compliance monitoring, safety posture administration, value optimization, all from the identical core platform,” mentioned Pats.
How Hexagon constructed a serverless information lake for all its cloud shops
One enterprise already seeing outcomes is Hexagon. The software program firm’s cloud middle of excellence (CCoE) crew had a aim to construct a totally serverless information lake that might acquire information from all of its cloud accounts and retailer it in a single information lake.
In addition they needed the power to question this information utilizing SQL and visualize it with instruments they had been accustomed to (reminiscent of AWS QuickSight), and discover the historical past of their cloud configuration over time.
The crew constructed a serverless information pipeline utilizing CloudQuery to gather information from all accounts and retailer it in S3. AWS Glue then ingests information into Glue DB in a format that Amazon Athena can question, which Athena then does, then visualises in QuickSight.
“Having a totally serverless resolution was an necessary requirement,” Hexagon cloud governance and FinOps professional Peter Figueiredo and CloudQuery director of engineering Herman Schaaf wrote in a weblog submit. “This resolution introduced a lot of advantages since there isn’t any want for time-consuming updates and nearly zero upkeep.”
They did have to beat some challenges, notably with Amazon S3 assist plugins. The CCoE crew was one of many first to check out CloudQuery options within the S3 vacation spot and provided insights resulting in new options. These embrace:
Parquet assist: The CloudQuery file vacation spot initially solely supported CSV and JSON information codecs. Errors in JSON interpretations led CloudQuery so as to add Parquet assist.
Knowledge partitioning: A CloudQuery file vacation spot plugin now permits partitioning on preliminary write (which beforehand wasn’t accessible, leading to additional pointless steps).
Useful resource view for Athena: CloudQuery initially solely provided a sources view for AWS suitable with Postgres. However Athena didn’t assist this, so CloudQuery added a operate that may retrieve an inventory of all tables to construct or replace a sources view.
Figueiredo’s crew used CloudQuery to interchange AWS’s VPC IP tackle supervisor (IPAM) — which he known as costly and restricted in that it doesn’t cowl different cloud suppliers.
In the end, his crew runs CloudQuery in ‘information lake’ mode utilizing “extremely low-cost infrastructure” together with AWS S3, ECS, Glue, Athena and Lambda,” Figueiredo advised VentureBeat. This retains prices low and permits Hexagon to merge all its IP addresses throughout completely different cloud suppliers.
“We are able to shortly question any IP throughout the board and discover who the homeowners are,” mentioned Figueiredo. “We at the moment are in a position to acquire all we’d like at a really low value with close to zero upkeep. That is the holy grail for our crew.”
Every day insights on enterprise use instances with VB Every day
If you wish to impress your boss, VB Every day has you lined. We provide the inside scoop on what firms are doing with generative AI, from regulatory shifts to sensible deployments, so you may share insights for max ROI.
Thanks for subscribing. Try extra VB newsletters right here.
An error occured.
