Chad Sanderson on Substack: "ChatGPT will not replace data engineers"
ChatGPT's ability to write SQL may be impressive, but it falls short in truly understanding how code is implemented in real-world scenarios. Each business has its unique data storage methods, with customerIDs stored in various formats such as MySQL DB, nested JSON from Mixpanel, or collected from a CDP. Integrating these different data formats into a cohesive data model requires a deep understanding of their semantics and relationships.
While ChatGPT is intelligent, it lacks the ability to grasp the underlying meaning of data objects and how they relate to each other. Automating tasks such as data modeling and ETL processes would require a level of cognition about the business operations and how they are reflected in code. This goes beyond machine learning into the realm of general intelligence.
Many people believe that large language models (LLMs) have magical capabilities, but they often overlook the challenges posed by the complex and messy data ecosystems present in most businesses. Without significant infrastructure improvements, models like ChatGPT would struggle to generate meaningful insights.
However, ChatGPT can still offer valuable contributions, particularly in query optimization. By analyzing common queries, it can provide recommendations for optimizing and simplifying data pipelines, leading to cost reductions and improved usability. With access to well-maintained data contracts, ChatGPT could suggest datasets based on intent and reliability, enhancing developer productivity and data modeling efficiency.
Despite these advancements, data engineers need not fear being replaced by AI. Human expertise is essential for interpreting and navigating complex data infrastructures, a task that machines cannot replicate. While ChatGPT and similar models can streamline certain processes, they ultimately serve as tools to augment human capabilities rather than replacing them entirely.
Chad Sanderson on Substack offers valuable insights into the role of AI in data engineering and emphasizes the irreplaceable expertise that human professionals bring to the table.
