Solving Spark’s Small File Problem for 100x Faster Reads

Understand the Spark common small file problem, learn how to solve in the modern open table formats through offline and online optimizations.

Dec 6 • Junaid Effendi

13

2

2

November 2025

Shopify Data Tech Stack

Explore what tech stack is used at Shopify to process 284 million peak requests per minute generating $11+ billions in sales.

Nov 8 • Junaid Effendi

9

4

2

October 2025

Inside Data Engineering with Erfan Hesami

Join Erfan Hesami as he shares his experience in the world of data engineering, offering insights, exploring challenges, and highlighting emerging…

Oct 11 • Junaid Effendi and Erfan Hesami

13

3

September 2025

How Delta Lake Works

Understand how Delta Lake handles reads and writes using the transaction log, ensuring ACID guarantees through snapshot isolation and optimistic…

Sep 6 • Junaid Effendi

18

4

1

August 2025

Spotify Data Tech Stack

Learn how Spotify ingests 1.4T+ events daily on GCP via 38K+ data pipelines, leveraging BigQuery, Dataflow, and Flyte to power ~5K dashboards and scale…

Aug 16 • Junaid Effendi

22

2

July 2025

Inside Data Engineering with Julien Hurault

Consultant Julien Hurault takes you inside the world of data engineering, sharing practical insights, real-world challenges, and his perspective on…

Jul 26 • Junaid Effendi and Julien Hurault

20

2

1

Benchmarking Spark - Open Source vs EMRs

Diving into four approaches from Spark Operator to EMR (EKS, EC2, and Serverless), sharing benchmarking results and key insights to help you choose the…

Jul 5 • Junaid Effendi

5

2

2

June 2025

Snapchat Data Tech Stack

Learn how Snapchat ingests ~2 trillions of events per day using Google Cloud Platform.

Jun 7 • Junaid Effendi

26

2

May 2025

Inside Data Engineering with Daniel Beach

Veteran data engineer Daniel Beach takes you inside the world of data engineering, sharing hard-earned insights, day-to-day challenges, and what’s on…

May 24 • Junaid Effendi and Daniel Beach

16

3

Data Governance in Lakehouse Using Open Source Tools

Discover how to build a complete data governance ecosystem in a Lakehouse architecture using leading open-source tools. Explore access control, metadata…

May 10 • Junaid Effendi

23

2

6

April 2025

DoorDash Data Tech Stack

Learn about the Data Tech Stack used by DoorDash to process hundreds of Terabytes of data every day.

Apr 26 • Junaid Effendi

23

3

2

Inside Data Engineering with Vu Trinh

Join Vu Trinh as he navigates the world of data engineering, sharing insights, challenges, and emerging industry trends.

Apr 5 • Junaid Effendi and Vu Trinh

34

3

4