Junaid Effendi | Sharing knowledge for Engineers

Junaid Effendi | Sharing knowledge for Engineers

Home
🧑‍💻 Become a DE
🔍 Deep Dive
⭐ Tech Stack
🚀 Growth
🗃️ Archive
❓ About

Deep Dive

How Delta UniForm works
Learn how Delta UniForm enables read and write interoperability from Delta tables to Iceberg and Hudi formats.
 • Junaid Effendi
Solving Spark’s Small File Problem for 100x Faster Reads
Understand the Spark common small file problem, learn how to solve in the modern open table formats through offline and online optimizations.
 • Junaid Effendi
How Delta Lake Works
Understand how Delta Lake handles reads and writes using the transaction log, ensuring ACID guarantees through snapshot isolation and optimistic…
 • Junaid Effendi
Benchmarking Spark - Open Source vs EMRs
Diving into four approaches from Spark Operator to EMR (EKS, EC2, and Serverless), sharing benchmarking results and key insights to help you choose the…
 • Junaid Effendi
Data Governance in Lakehouse Using Open Source Tools
Discover how to build a complete data governance ecosystem in a Lakehouse architecture using leading open-source tools. Explore access control, metadata…
 • Junaid Effendi
Securely Share and Automate File Transfers with AWS Transfer Family & Terraform
Learn to deploy a secure, automated SFTP server with AWS Transfer Family & Terraform. Set up restricted users, enforce SSH & MFA, and leverage workflows…
 • Junaid Effendi
Six Effective Ways to Reduce Compute Costs
Lets look into the top six ways that can help reducing your compute costs on AWS.
 • Junaid Effendi
Terraform vs Asset Bundles for Databricks Workflows
Sharing the experience of moving from Terraform to Asset Bundle for Databricks workflow deployment, covering challenges and benefits.
 • Junaid Effendi
Challenges: From Databricks to Open Source Spark & Delta
Sharing the challenges to save hours when doing migration from Databricks to Open Source.
 • Junaid Effendi
Data Modelling Using Complex Data Types
Complex data types like struct, array, map in modern warehouses are game changer, learn the useful aspects from a Data Engineer.
 • Junaid Effendi
Messaging Systems: Queue Based vs Log Based
Learn the key differences and important properties of queue and log based messaging systems.
 • Junaid Effendi
Handling Duplicates In Streaming Pipeline
Three ways to handle duplicate data in streaming pipelines. Learn the benefits, use cases and more in this article.
 • Junaid Effendi
© 2026 Junaid Effendi · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture