Recently I had someone ask if I could write something about how I see AI impacting data engineering and I figured I’d take a stab at that. I think that this can be separated into three parts. While this article does cover AI I’m not going to spend much time discussing the way these tools work and will mostly focus on their application and the pros/cons.
- How is AI impacting data engineering now?
- How will it impact data engineering in the future?
- What should you do with this knowledge?
How is AI impacting data engineering now?
In order to answer this I’ll mostly be speaking from my experience and the applications of AI I’ve seen so far. I will note that I feel like AI in this context generally can be substituted with generative text based models like ChatGPT that assist in software development however there are other tools that aren’t exactly like this.
To start, I’ve been a data engineer for nearly three years now, before that I was an analyst for about two years and I’ve used various AI tools in my work for the past few years. In particular I’ve used both ChatGPT and GitHub Copilot to assist more or less daily for the past year. I use these for different things generally.
I use ChatGPT 3.5 through the browser and generally as either an alternative to Google for searching for documentation, checking my understanding of things…
