How has the recent turmoil within the OpenAI offices changed your plans to use GPT in a business process or product in 2024?
Increased uncertainty means we are more likely to evaluate alternative AI chatbots and LLMs.
No change in plans, though we will keep an eye on the situation.
With Sam Altman back in charge, we are more likely to go all-in with GPT and LLMs.
What recent turmoil?

Integrating a Data Warehouse and a Data Lake

This conversation from the Open Source Summit in Spain provides valuable insights into the significance of Delta Lake, the role of Rust in data engineering, and the collaborative nature of open source communities.
Nov 16th, 2023 8:30am by
Featued image for: Integrating a Data Warehouse and a Data Lake

The Delta Lake open source project integrates data lakes and data warehouses, a needed combination in this new age of scale-out data requiring reliability and controls.

A data lakehouse integrates the advanced data analytics and low-cost storage of a data lake with the performance and reliability of a data warehouse, said Florian Valeye, a data engineer with Back Market, in this episode of The New Stack Makers, recorded at the Open Source Summit in Bilbao, Spain, earlier this fall.

The data warehouse is an approach that emerged over the past two decades. It consists of structured data models that allow for better performance. The datasets are small and constrained, Valeye said. In contrast, the data lake consists of unstructured data from multiple sources. The volume size of data lakes reaches petabytes or even exabytes.

Delta Lake, created by Databricks, breaks down the barriers between data warehouses and data lakes by providing more performance and features in the data lake, Valeye said.

ACID transactions are a staple of a data warehouse, Valeye said. When thinking about the relational database or a data warehouse, a focus is put on model representations and the data structure. A data lake is an open way to push data and add a schema. Data lakes magnify data. Through the “wall breaking,” the lake house provides ACID transactions, Read, Process, Interpret (RPI) ingestions, and metadata scalability. The strength comes with a way to attain knowledge for any usage without a barrier between the data analyst side of the house and the data engineering and data scientist teams.

Databricks is now working on providing ways for anyone to contribute their connectors through Delta Lake, allowing gateways that can be used with, for example, different databases.

“And that’s why it’s really nice when you are working on a specific cloud provider; you don’t want to be bundled, locked inside it,” Valerie said. “So that’s why having this kind of standard format, you can switch and move from one to another, and don’t feel stuck with one provider and one format.”

Valeye said Back Market sells refurbished devices. To determine device quality, the company developed an algorithm to determine whether a device is viable for sale. Back Market uses Delta Lake to ingest data from APIs and other data sources.

Delta Lake is a platform to connect data scientists and data engineers, said Valeye, who previously worked as a data engineer. Before using Delta Lake, deploying models could be complicated due to the complexities of the different tools and programming languages used. The Delta Lake infrastructure closes the gaps. It allows everyone to work on the same infrastructure.

More Episodes from Open Source Summit EU 2023

From Debian to Open Source AI

WebAssembly’s Status in Computing

PostgreSQL Takes a New Turn

Powertools for AWS Lambda Grows with Help of Volunteers

How to Be a Better Ally in Open Source Communities

Open Source Development Threatened in Europe

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.