Inside Data Engineering with Dipankar Mazumdar
Follow Dipankar Mazumdar as he explores open data platforms, architecture, interoperability, and where Data Engineering is headed next, from a Developer Relations perspective.
Most Inside Data Engineering conversations feature data engineers and architects, this one is different. I sat down with Dipankar Mazumdar, who leads Developer Relations at Cloudera. Let's learn today from a different perspective; that of a DevRel.
What to Expect:
A DevRel lens on Data Engineering — shaped by community exposure, not just one org’s roadmap
A broader definition of the field — ingestion, transformation, and platform operations, not just pipelines
Real patterns from the field — moving from “which platform should we standardize on” to “how should data be represented”
A reference architecture for open data platforms, built on open standards, format interoperability, and catalog interoperability
Where the lakehouse is headed — optimization, streaming/batch convergence, and agentic AI workloads
Practical career advice — why fundamentals outlast tools
⭐ If you’re curious about data engineering or considering it as a career, this series is for you!
Let’s dive into Inside Data Engineering:
How did you get into Data Engineering Space, and how has your role evolved to where you are today at Cloudera?
I actually started my career as a software engineer. Over time, I found myself increasingly drawn to the infrastructure behind data systems - specifically how data is stored, processed, queried, and managed at scale. In my very first job, I came across the Hadoop ecosystem. With the support of a few colleagues and a lot of self-learning, we solved a major data processing problem using MapReduce. Seeing that kind of impact firsthand was fascinating, and it sparked my interest in distributed data platforms and the open-source technologies surrounding them.
One thing I would emphasize is that Data Engineering is an incredibly broad field. Over the past several years, my focus has been on data architectures and understanding the underlying storage and compute systems that power modern data platforms. Today, at Cloudera, I lead Developer Relations. While I still spend a significant amount of time building technical solutions, my role has expanded to include educating developers, shaping technical strategy, applied research, creating learning resources, and engaging with the broader data engineering community.
In simple terms, how would you explain what a Data Engineer does to someone new?
Like I mentioned earlier, Data Engineering is a much broader field than many people realize. While most people associate it with building ETL pipelines, that’s really just one part of the picture. I tend to think about Data Engineering in three broad areas:
First is ingestion: moving data from operational systems, applications, or streaming platforms into a centralized repository such as a lakehouse or data warehouse.
Second is transformation: turning raw data into something meaningful by defining business semantics, applying quality checks, building data models, and creating trusted datasets that can be consumed consistently across the organization.
Third is building/operating the data platform itself: the storage, compute, catalog, metadata, governance, and operational capabilities that make ingestion and transformation possible at scale.
Ultimately, a Data Engineer is responsible for building systems that make data reliable, scalable, and accessible. Whether that data is consumed by dashboards, applications, machine learning models, or increasingly AI agents, the goal is to ensure everyone is working from trusted, well-managed data.
What does your typical day or week look like and how much of it is still hands-on with data and engineering?
DevRel is a multi-faceted role. There really isn’t a typical week, which is one of the things I enjoy most about it. Most weeks involve some combination of writing code, reading research papers, building demos, or experimenting with new open-source technologies. A lot of that work eventually translates into a white paper, a book chapter, a conference talk, or a technical deep-dive video explaining how these systems work under the hood. I also spend a fair amount of time engaging with the data engineering community to understand the challenges they’re trying to solve, which often shapes the projects and content I work on next.
From the customers and teams you work with, what kinds of data problems do you see mattering most to the business right now?
To be specific, my customers are highly technical practitioners (imagine a Senior-level engineer in an organization). The conversations that I have usually with them are about architectural decisions and long-term trade-offs. Teams are trying to figure out how to build open data platforms by choosing the right combination of storage formats, catalogs, query engines, streaming technologies, and orchestration frameworks. They are thinking about interoperability between systems, avoiding vendor lock-in, and designing architectures that can evolve as requirements change.
A lot of discussions also revolve around operating these platforms efficiently. Things like performance tuning, cost optimization, governance, metadata management, and ensuring data remains discoverable and trustworthy as the platform scales.
What kind of data, scale, and industries do you typically encounter, and who are the primary developers and engineering teams you work with?
One of the things I enjoy most about my role is the diversity of problems I get exposed to. The industries span financial services, telecommunications, manufacturing, healthcare, and retail. But irrespective of the industry, the interesting part to me is how often the underlying engineering and architectural challenges are remarkably similar.
The data itself spans everything from traditional transactional data and application logs to clickstreams, IoT telemetry, streaming events, documents, images, and increasingly multimodal data. Some organizations are managing terabytes of data, while others operate petabyte-scale platforms with thousands of datasets and multiple compute engines accessing the same underlying data.
The engineering teams I work with are just as diverse. That includes data engineers, platform engineers, software engineers, data architects, and AI engineers building intelligent applications on top of enterprise data.
Can you walk me through one project or customer use case you’ve seen done really well problem, solution, challenges, and impact?
I think a more relevant example could be a pattern I have seen numerous times, which is organizations ending up with multiple tools and vendor platforms. Sometimes it’s because of acquisitions, sometimes because different teams adopted different technologies over time, and increasingly it’s because AI workloads introduce another set of tools into the ecosystem.
The challenge with these kinds of environments is that every platform wants to manage its own copy of the data. Before long you have duplicated datasets, different governance models, and engineering teams spending a surprising amount of time and cost moving data between systems.
In one particular project I remember, the discussions initially were “Which platform should we standardize on?” and then we helped change the narrative to “Can we standardize how the data is represented instead?” That changed the direction of the project completely. Rather than forcing every team onto the same compute engine or platform, they standardized on Apache Iceberg as the common table format. Spark, Trino, and other engines were used for their specific tasks, but they were now operating on the same datasets instead of maintaining separate copies. The interesting part was that data interoperability became the default for teams there rather than something each team had to solve individually. New engines could be introduced without first asking how the data would be moved or translated, because they were operating on the same open copy of data.
At a high level, what does a well-designed open source data platform look like from ingestion to consumption?
This is a reference architecture that I usually work with at a very high-level. Obviously, the individual technologies may vary from organization to organization, but the architectural principles remain largely the same.
Starting from the left, you have a variety of data sources - databases, event streams, files, and operational systems. Those feed into an ingestion layer, which could be Spark, Flink, Kafka, or any other processing framework. To me, the specific technology is less important than what it produces. From there, I think there are three key principles that make an open data platform successful.
First is open standards for components. Every layer of the architecture should be built on open standards rather than proprietary interfaces. That doesn’t mean every component has to be self-managed open source, but it does mean organizations should have the flexibility to mix and match best-of-breed technologies and avoid being locked into a single vendor ecosystem.
Second is interoperability at the data layer. By adopting open table formats like Apache Iceberg, multiple compute engines can work with the same data. Whether you’re using Trino for analytics, Spark for large-scale processing, or Ray for AI workloads, they’re all operating on a shared representation of the data.
The third is interoperability between catalogs. Open table formats solved data interoperability, but metadata interoperability is becoming equally important. Organizations increasingly use multiple catalogs across teams and platforms, and open catalog interfaces like the Iceberg REST Catalog specification provide a common standard for accessing metadata. That makes it easier for different engines and platforms to work together without introducing another layer of vendor lock-in.
What are your thoughts on the future of the lakehouse?
I think we are moving beyond the phase where the conversation is simply “Should I build a lakehouse?” That architectural shift has already happened. The more interesting questions now are around operating the platform efficiently and preparing it for entirely new workloads.
One area is optimization. As organizations have more scale of data to deal with in their lakehouse, things like file layout, compaction, clustering, metadata management, and query planning become just as important as the choice of storage format itself. You can have all the right technologies in place, but if the platform isn’t continuously optimized, performance and cost become real challenges.
Another area is streaming. With a Lakehouse, more organizations are treating streaming and batch as part of the same data architecture. That introduces its own engineering challenges around concurrent writes, metadata growth, and maintaining performance while data is continuously changing. We need to think a bit more about these kinds of low-latency workloads.
Finally, I think agentic AI workloads are pushing lakehouses in an entirely new direction. Traditional data platforms were designed around deterministic pipelines and human users. Agents behave differently. They discover datasets, explore metadata, execute tools, and iterate. That means the lakehouse needs to provide much richer context, isolation, governance, and metadata than we’ve traditionally expected from a data platform.
What’s something challenging or unexpected about Data Engineering that people don’t usually realize especially at enterprise scale?
One thing people often underestimate is that building a data pipeline is usually the easy part. Operating it at enterprise scale is where the real engineering challenges begin.
As organizations grow, you are no longer dealing with a handful of datasets or a single processing engine. You have thousands of tables, multiple compute engines, streaming workloads, and different teams, all interacting with the same platform. In the data platform space for example, small decisions around file layout, partitioning, metadata, or compaction that don’t matter much at small scale can have a significant impact on performance and cost as your volume and variety increases. Another challenge is that data platforms are never static. New workloads are constantly being introduced, technologies evolve, and organizations rarely get to start from a clean slate. A lot of engineering ends up being about evolving existing systems without disrupting the people and applications already depending on them.
That’s one of the reasons I have become increasingly interested in areas like platform optimization and observability. The goal is to understand how the platform behaves over time and continuously improve it as the scale and workloads change.
Where do you see GenAI helping (or not helping) Data Engineering today?
I think GenAI is already making data engineers more productive, particularly around code generation, documentation, debugging, and understanding unfamiliar systems. It lowers the barrier to getting started and helps engineers move faster on repetitive tasks.
Where I think things become more interesting is beyond code generation. For e.g. data platforms contain an enormous amount of metadata that engineers may have very less time to explore manually. AI can help answer questions about data quality, table health, lineage, query performance, or why a particular pipeline is behaving differently from last week. Those are areas I have been spending a lot of time exploring because they combine reasoning with the operational context of the platform.
Where I think we are still early is autonomous operation. Enterprise data platforms are governed systems with security policies, compliance requirements, and real business impact. AI can certainly assist engineers, but it also needs context, guardrails, and the ability to explain why it made a particular recommendation or decision. I think that’s where much of the engineering work will be over the next few years.
What advice would you give to someone considering a career in Data Engineering in 2026?
I would probably tell them not to become an expert in a single tool. Technologies change remarkably quickly in our industry. The fundamentals don’t.
Spend time understanding distributed systems, data modeling, storage formats, databases, query optimization, and how modern data platforms are designed. Those concepts will stay relevant whether you’re using Spark today, something completely different five years from now. The engineers who understand how those fundamental pieces fit together will be in a much stronger position than those who specialize in just one technology.
Finally, build things. Read papers and books, but also experiment. Set up a lakehouse, break it, optimize it, measure performance, and understand why things behave the way they do. I still think there’s no substitute for learning by building.
What are the most common misconceptions about Data Engineering, especially ones you keep running into when talking to teams in the field?
One thing I have noticed is that we sometimes equate Data Engineering with whichever technologies are popular at the time. Five years ago the conversations looked different than they do today, and five years from now they’ll probably look different again.
The reality is that tools come and go. What tends to stay relevant are the underlying concepts like I said - how distributed systems work, how data is organized, how to build reliable pipelines, how to optimize performance, and how to design platforms that evolve over time. Those fundamentals transfer regardless of which technology you are using.
I hope this article was helpful for the readers. Thanks to <> for sharing his experience with my audience. Stay tuned for more!
Please reach out if you like:
To be the guest and share your experiences & journey.
To provide feedback and suggestions on how we can improve the quality of questions.
To suggest guests for the future articles.