Subscribe
Sign in
Home
🧑💻 Become a DE
🔍 Deep Dive
⭐ Tech Stack
🚀 Growth
🗃️ Archive
❓ About
Introduction to Spark Optimization
Starting the series of Spark optimization, covering different level of optimization.
Jul 8, 2019
•
Junaid Effendi
5
Testing Data in Apache Spark
How to efficiently test large scale Spark jobs, using open source libraries.
Sep 9, 2023
•
Junaid Effendi
4
Optimizing Spark Job (spark-submit/shell)
This article is second from our series, optimizing the spark command, we usually use two types of spark commands, spark-submit and spark-shell, both of…
Jul 20, 2019
•
Junaid Effendi
2
1
Benchmarking Spark - Open Source vs EMRs
Diving into four approaches from Spark Operator to EMR (EKS, EC2, and Serverless), sharing benchmarking results and key insights to help you choose the…
Jul 5
•
Junaid Effendi
5
2
2
Spark Performance Tuning with Ganglia and Sparklens
Leveraging Sparklens and Ganglia for optimizing Spark Jobs.
May 4, 2020
•
Junaid Effendi
1
Optimizing Spark Query
This is the last article from our Spark Optimization Series. Optimizing a spark query is challenging as well as interesting, as a Data Engineer, I love…
Aug 5, 2019
•
Junaid Effendi
1
Whats new in Spark 3.0?
Asa Data Engineer I wait for improved Spark version every year, and this yearlast month they introduced a major long awaited upgrade known as Spark 3.0.
Nov 24, 2019
•
Junaid Effendi
1
Common Issues faced in Spark
There are several issues everyone faces when they start using spark either at their jobs or for fun.
Apr 9, 2018
•
Junaid Effendi
2
1
Optimizing Spark I/O
Reading and writing files in Spark is part of most of the jobs, and to make it work the best way there are some approaches and techniques that should be…
Jul 27, 2019
•
Junaid Effendi
1
Spark UDFs Are Cruel!
Spark UDFs (User Defined Functions) are not the best thing a developer will use, they look so cool especially the syntax to write them is really cool…
Aug 25, 2019
•
Junaid Effendi
1
Apache Spark - Frequently Asked Questions
Developers and Engineers are now pretty much aware of Apache Spark and its purpose in the technological stack but somehow there are some basic questions…
Jan 27, 2019
•
Junaid Effendi
Optimizing Spark Cluster
Learn the techniques to save cost and improve performance through cluster optimization.
Jul 15, 2019
•
Junaid Effendi
2
This site requires JavaScript to run correctly. Please
turn on JavaScript
or unblock scripts
