All the latest tools and trends in data engineering. — Note: This article was originally published by Einat Orr on June 20th, 2022. Introduction A year has passed since we shared the State of Data Engineering 2021. And since we released that article last May, not much has changed in the data landscape. …

Gain the confidence to mess with your data without making a mess of your data. — “If it hurts, do it more often.” is a wise piece of advice that DevOps engineers often repeat. Unless you are a masochist, following this advice will naturally lead you to finding ways to make the process being repeated less painful. In the world of DevOps, these processes are typically…

Take your data lake game to new heights with these two architecture improvements. — What is the Basic Data Lake? A data lake is primarily two things: an object store and the objects being stored. It might look something like this:

In October, it was announced that the Pandas API was being integrated with Spark. This is particularly exciting news for a Pandas-baby like myself, whose first exposure to data exploration involved following tutorials using Jupyter notebooks and Pandas DataFrames. — Spark 3.2 has been out for several months now and a curiosity has been building inside me — how easy it is to take existing pandas code and copy it as is into Spark? …

Let’s put the bagel to use by querying branched lakeFS data from Trino’s distributed engine. — Introduction Dockerize Your Data Pipeline I can remember times when my company started using a new technology — be it Redis, Kafka, or Spark — and in order to try it out I found myself staring at a screen like this: At the time I thought nothing of doing this. And even wore it as…

Already familiar with versioning code with git? A look at how it works to version data using the same abstractions. — “I have never lied to you, I have always told you some version of the truth.” “The truth doesn’t have versions, okay?” — Something’s Gotta Give (2003)

Few people start using lakeFS without first having some data collected. Consequently, it is common that after getting it up and running, one of the first things people do is import their existing data to lakeFS. — There isn’t a one-size-fits-all approach for doing importing data. Instead, there are ways that work great for a single file, and some that are designed to handle millions of them. Let’s walk through, in detail, how it’s done for each situation.