Archive - Junaid Effendi | Sharing knowledge for Engineers

Transition: Data Scientist to Data Engineer

Interested in working in the data engineering landscape, check out this natural transition for Data Scientist.

Oct 5 •

September 2024

Challenges: From Databricks to Open Source Spark & Delta

Sharing the challenges to save hours when doing migration from Databricks to Open Source.

Sep 25 •

How to avoid pipeline backfill nightmares

Tired of the backfill process? Checkout these best practices to help you ease the pain of backfilling the data.

Published on DataEngineer.io Newsletter • Sep 17

August 2024

Transition: Software Engineer to Data Engineer

Interested in working in the data engineering landscape, check out this natural transition for SWE.

Aug 31 •

Airbnb Data Tech Stack

Learn about the Data Tech Stack used by Airbnb to process billions of data points every day.

Aug 14 •

Deploying Spark Streaming with Delta on Kubernetes using Terraform

Spark. Kubernetes. Terraform.

Published on Data Engineering Central • Aug 3

July 2024

Types of Data Engineers

Learn different types of focus areas for a Data Engineer in the modern industry with future articles covering the specific roadmaps.

Jul 27 •

Practical Data Engineering using AWS Cloud Technologies

Learn how to build end-to-end data engineering projects using pure AWS cloud technologies like S3, SNS, Lambda, Step Function, and more.

Published on VuTrinh. • Jul 24

Data Modelling Using Complex Data Types

Complex data types like struct, array, map in modern warehouses are game changer, learn the useful aspects from a Data Engineer.

Jul 6 •

June 2024

Uber Data Tech Stack

Learn about the Data Tech Stack used by Uber to process trillions of events every day.

Jun 26 •

Messaging Systems: Queue Based vs Log Based

Learn the key differences and important properties of queue and log based messaging systems.

Jun 15 •

Incremental Deprecation

Deprecate incrementally along with incremental releases to save cost, avoid tech debt and more, all in today's article.

Jun 1 •

#nojs-banner { position: fixed; bottom: 0; left: 0; padding: 16px 16px 16px 32px; width: 100%; box-sizing: border-box; background: red; color: white; font-family: -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 13px; line-height: 13px; } #nojs-banner a { color: inherit; text-decoration: underline; } This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts