Junaid Effendi | Sharing knowledge for Engineers

Junaid Effendi | Sharing knowledge for Engineers

Home
🧑‍💻 Become a DE
🔍 Deep Dive
⭐ Tech Stack
🚀 Growth
🗃️ Archive
❓ About

Deep Dive

Iceberg on AWS: Choosing Between S3 Tables and Glue-backed Iceberg
Two ways to run Iceberg on AWS: one managed, one self-managed. Here's how to choose between S3 Tables and Glue-backed Iceberg based on your stack, team…
 • Junaid Effendi
How Feldera Works: A True Incremental View Maintenance Engine
Learn how Feldera treats streams as relational views and uses DBSP to propagate deltas efficiently, enabling scalable, low-cost continuous analytics.
 • Junaid Effendi
How Delta UniForm works
Learn how Delta UniForm enables read and write interoperability from Delta tables to Iceberg and Hudi formats.
 • Junaid Effendi
Solving Spark’s Small File Problem for 100x Faster Reads
Understand the Spark common small file problem, learn how to solve in the modern open table formats through offline and online optimizations.
 • Junaid Effendi
How Delta Lake Works
Understand how Delta Lake handles reads and writes using the transaction log, ensuring ACID guarantees through snapshot isolation and optimistic…
 • Junaid Effendi
Benchmarking Spark - Open Source vs EMRs
Diving into four approaches from Spark Operator to EMR (EKS, EC2, and Serverless), sharing benchmarking results and key insights to help you choose the…
 • Junaid Effendi
Data Governance in Lakehouse Using Open Source Tools
Discover how to build a complete data governance ecosystem in a Lakehouse architecture using leading open-source tools. Explore access control, metadata…
 • Junaid Effendi
Securely Share and Automate File Transfers with AWS Transfer Family & Terraform
Learn to deploy a secure, automated SFTP server with AWS Transfer Family & Terraform. Set up restricted users, enforce SSH & MFA, and leverage workflows…
 • Junaid Effendi
Six Effective Ways to Reduce Compute Costs
Lets look into the top six ways that can help reducing your compute costs on AWS.
 • Junaid Effendi
Terraform vs Asset Bundles for Databricks Workflows
Sharing the experience of moving from Terraform to Asset Bundle for Databricks workflow deployment, covering challenges and benefits.
 • Junaid Effendi
Challenges: From Databricks to Open Source Spark & Delta
Sharing the challenges to save hours when doing migration from Databricks to Open Source.
 • Junaid Effendi
Data Modelling Using Complex Data Types
Complex data types like struct, array, map in modern warehouses are game changer, learn the useful aspects from a Data Engineer.
 • Junaid Effendi
© 2026 Junaid Effendi · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture