Archive - Junaid Effendi | Sharing knowledge for Engineers

Practical Data Engineering using AWS Cloud Technologies

Learn how to build end-to-end data engineering projects using pure AWS cloud technologies like S3, SNS, Lambda, Step Function, and more.

Published on VuTrinh. • Jul 24

Data Modelling Using Complex Data Types

Complex data types like struct, array, map in modern warehouses are game changer, learn the useful aspects from a Data Engineer.

Jul 6 •

June 2024

Uber Data Tech Stack

Learn about the Data Tech Stack used by Uber to process trillions of events every day.

Jun 26 •

Messaging Systems: Queue Based vs Log Based

Learn the key differences and important properties of queue and log based messaging systems.

Jun 15 •

Incremental Deprecation

Deprecate incrementally along with incremental releases to save cost, avoid tech debt and more, all in today's article.

Jun 1 •

May 2024

Everything you need to know about MapReduce

All the key insights from the paper MapReduce: Simplified Data Processing on Large Clusters from Google.

May 29 •

and

Handling Duplicates In Streaming Pipeline

Three ways to handle duplicate data in streaming pipelines. Learn the benefits, use cases and more in this article.

May 22 •

Netflix Data Tech Stack

Learn about the Data Tech Stack used by Netflix to process trillions of events every day.

May 8 •

April 2024

Dev Day vs Dev Hours

Learn about dev day and dev hours, which one to use, when to use and how to make the most of a developer time in an effective way.

Apr 27 •

Data Pipeline - Incremental vs Full Load

Learn the pros, cons and use cases about the data pipeline design patterns, full load and incremental commonly used across the industry.

Apr 13 •

March 2024

Why Job Titles Matter

Sharing the importance of job titles and levels, why they matter in the tech world and how we all benefit from it.

Mar 30 •

Data Processing in 21st Century

Timeline of Data Processing technologies covering from MapReduce to Polars.

Mar 20 •

#nojs-banner { position: fixed; bottom: 0; left: 0; padding: 16px 16px 16px 32px; width: 100%; box-sizing: border-box; background: red; color: white; font-family: -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 13px; line-height: 13px; } #nojs-banner a { color: inherit; text-decoration: underline; } This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts