Junaid Effendi | Sharing knowledge for Engineers

Junaid Effendi | Sharing knowledge for Engineers

Home
🧑‍💻 Become a DE
🔍 Deep Dive
⭐ Tech Stack
🚀 Growth
🗃️ Archive
❓ About

Deep Dive

How Delta Lake Works
Understand how Delta Lake handles reads and writes using the transaction log, ensuring ACID guarantees through snapshot isolation and optimistic…
 • 
Junaid Effendi
Benchmarking Spark - Open Source vs EMRs
Diving into four approaches from Spark Operator to EMR (EKS, EC2, and Serverless), sharing benchmarking results and key insights to help you choose the…
 • 
Junaid Effendi
Data Governance in Lakehouse Using Open Source Tools
Discover how to build a complete data governance ecosystem in a Lakehouse architecture using leading open-source tools. Explore access control, metadata…
 • 
Junaid Effendi
Securely Share and Automate File Transfers with AWS Transfer Family & Terraform
Learn to deploy a secure, automated SFTP server with AWS Transfer Family & Terraform. Set up restricted users, enforce SSH & MFA, and leverage workflows…
 • 
Junaid Effendi
Six Effective Ways to Reduce Compute Costs
Lets look into the top six ways that can help reducing your compute costs on AWS.
 • 
Junaid Effendi
Terraform vs Asset Bundles for Databricks Workflows
Sharing the experience of moving from Terraform to Asset Bundle for Databricks workflow deployment, covering challenges and benefits.
 • 
Junaid Effendi
Challenges: From Databricks to Open Source Spark & Delta
Sharing the challenges to save hours when doing migration from Databricks to Open Source.
 • 
Junaid Effendi
Data Modelling Using Complex Data Types
Complex data types like struct, array, map in modern warehouses are game changer, learn the useful aspects from a Data Engineer.
 • 
Junaid Effendi
Messaging Systems: Queue Based vs Log Based
Learn the key differences and important properties of queue and log based messaging systems.
 • 
Junaid Effendi
Handling Duplicates In Streaming Pipeline
Three ways to handle duplicate data in streaming pipelines. Learn the benefits, use cases and more in this article.
 • 
Junaid Effendi
Data Pipeline - Incremental vs Full Load
Learn the pros, cons and use cases about the data pipeline design patterns, full load and incremental commonly used across the industry.
 • 
Junaid Effendi
Data Processing in 21st Century
Timeline of Data Processing technologies covering from MapReduce to Polars.
 • 
Junaid Effendi
© 2025 Junaid Effendi · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture