Jesse Anderson

The data landscape is fickle, and once-coveted roles like 'DBA' and 'Data Scientist' have faced challenges. Now, the spotlight shines on Data Engineers, but will they suffer the same fate? This talk dives into historical trends. In the early 2010’s, DBA/data warehouse was the sexiest job. Data Warehouse became the “No Team.” In the mid-2010’s, data scientist was the sexiest job. Data Science became the “mistaken for” team. Now, data engineering is the sexiest job. Data Engineering became the “confused team”. The confusion run rampant with questions about the industry: What is a data engineer? What do they do? Should we have all kinds of nuanced titles for variations? Just how technical should they be? Together, let’s go back to history and look for ways on how data engineering can avoid the same fate as data warehousing and data science. This talk provides a thought-provoking discussion on navigating the exciting yet challenging world of data engineering. Let's avoid the pitfalls of the past and shape a future where data engineers thrive as essential drivers of innovation and success.

Between books and the real world lies the actual reality of implementing and creating data teams. Creating data teams represents the difficulty of taking scientists, engineers, and operations people then forcing them to actually get something working that solves a business problem.

The biggest initial hurdle to success with Big Data isn’t technical - it’s management. Your data engineering project’s initial success is predicated on your management team correctly staffing and resourcing it.