Inside Data Engineering with Daniel Beach
Veteran data engineer Daniel Beach takes you inside the world of data engineering, sharing hard-earned insights, day-to-day challenges, and what’s on the horizon for the field.
Today, we're joined by
from , who’s been working in Data Engineering since before it was cool, he will share his journey and insights.To recap: the series follows a Q&A format, featuring professionals who share their journeys, insights, and challenges.
What to Expect:
Inside the Day-to-Day – See what life as a data engineer really looks like on the ground.
Breaking In – Explore the skills, tools, and career paths that can get you started.
Tech Pulse – Keep up with the latest trends, tools, and industry shifts shaping the field.
Real Challenges – Uncover the obstacles engineers tackle beyond the textbook.
Myth-Busting – Set the record straight on common data engineering misunderstandings.
Voices from the Field – Get inspired by stories and insights from experienced pros.
⭐ If you're curious about data engineering or considering it as a career, this series is for you!
Let’s dive into Inside Data Engineering:
How would you describe Data Engineering?
Prior to the rise of AI, Data Engineering has become about being the best at Python, SQL, or whatever new hot tool was released. AI coding assistants, like it or not, have lowered the bar and made Data Engineering less about coding and more about providing business value from data. More than ever Data Engineering is about …
High-level architectural and data platform designs and maintenance.
Reducing data processing costs and complexity
Communication with non-engineering groups
Leading and upskilling others
Project planning and implementation
How did you end up being a Data Engineer?
I came from a non-traditional background, I never took a computer science class in my life. I taught myself during college to write PHP, Perl, MySQL, etc. After a few years of working as an engineer, I decided I wanted to move into tech. I taught myself SQL, got SQL Server certified, and got a job as a Data Analyst on a Business Intelligence team.
This was when Data Engineering was just becoming a thing, so I continued to hone my programming skills and taught myself things like Spark before it was popular. The rest is history.
What's your day to day look like?
I work at a small startup, so it can vary greatly, but it is typically made up of the following different tasks.
Project planning (planning new features, writing docs, implementation plans, etc., creating JIRA tasks etc.)
Answering questions from others and helping unblock others as needed.
Focused coding time.
Could be AWS infra, Spark, Databricks, Postgres, Docker, etc.
What are some stakeholders that you work with?
I work with:
Data Science
Product (the business)
Data Analysts
C-Suite (CTO, etc.)
What real-world business problems do you solve through data?
We work on Machine Learning pipelines at scale that can predict debit card/credit card fraud. This requires building reliable systems that scale and ingest large quantities of data with a small team.
What kind of projects do you work on?
A very large variety of projects, working at startups requires a wide range of skills. AWS infrastructure, lots of Databricks/Spark pipelines, even LLM fine-tuning and RAGs. I enjoy working on a wide variety of technologies, it keeps me engaged and always learning something new.
What kind of data do you work with?
Mostly tabular data.
What data size do you work with?
300TBish.
What technologies do you use?
What is your favorite area of Data Engineering?
Rust-based Data Engineering tools like Polars and Daft, I think it’s the future of Data Engineering.
How can Data Engineering benefit from GenAI?
Increased efficiency in producing results, let GenAI assist in writing tests, finding bugs, and bouncing ideas off it. It can shorten the time horizon in many areas of the Software Development Lifecycle.
It’s fair that some people are skeptical and worried about losing fundamental skills, but if you are a continuous learner before AI, the chances that AI will change who you are are low.
What advice would you give your past self as a beginner Data Engineer?
Work on soft skills as hard as you work on programming and technical skills. Writing, speaking, communicating, project planning, etc.
Never stop learning, always push yourself to do things you don’t understand. Find people smarter than yourself and then work closely with them.
What are some challenging aspects of Data Engineering?
Working with the business in a way that will build relationships while at the same time being strict about best practices and approaching problems in a reasonable manner. Also, in the fast-paced, changing landscape of tech and Data Engineering, it is important to stay focused on the basics and provide reliable and scalable solutions that simply work well.
What is the next big thing in Data Engineering?
DuckDB, Polars etc., they will become distributed in nature and start to eat Spark’s market share (over a long period of time).
What are some common misconceptions about data engineering?
You can simply learn SQL and Python and be a good Data Engineer. Now, with GenAI that is even less true than it was before. We need Engineers who can work well on teams, stay focused, make good tradeoffs, communicate well, and can do more than just write code.
Reach out if you like:
To be the guest and share your experiences & journey.
To provide feedback and suggestions on how we can improve the quality of questions.
To suggest guests for the future articles.
 | A guest post by
|