- ABOUT
- VIDEOS
- TOPICS
- JOBS & TALENT
- TESTIMONIALS
- COMMUNITY
- EVENTS
- NOMINATIONS & LISTS
- SPONSORSHIP
- ABOUT
- VIDEOS
- TOPICS
- JOBS & TALENT
- TESTIMONIALS
- COMMUNITY
- EVENTS
- NOMINATIONS & LISTS
- SPONSORSHIP
In a recent Sloan Review article, the outlet discusses key trends in AI and data science, highlighting the evolving nature of data science roles in organizations.
Feeling inspired, I opened up the discussion on LinkedIn and received over 50 survey responses from data science leaders and professionals around the globe. I posed the following prompt:
A recent Sloan Review article discusses key trends in AI and data science for 2024, highlighting the evolving nature of data science roles. With the rise of AutoML, citizen data science, and specialized roles in data management, the 'star power' of data scientists appears to be changing. How do you view this shift?"
The consensus? Change is happening and it’s driven by the following factors.
The need for specialization
The rise of data engineering
Democratization and spreading of data skills
The demand for allied professionals, including lawyers, product managers, and designers involved in bringing solutions to life
The pressures to demonstrate ROI and align strategy
Here’s how professionals in the industry view this shift in the data science role.
With Pandata’s clients, I’ve noticed the demands of project work over the last 12 months have skewed towards more strategy, more engineering, or more specialized machine learning (ML) — and others in the industry seem to agree.
“We have seen a marked drop in the demand from clients for the classic data scientist. In turn, there has been an uptick in the demand for specialists in the ML domain, such as computer vision, NLP, Deep Learning, etc. However, these engineers need to be exceptional programmers and have cloud experience (hence the rise in AI/ML/Data Ops roles). We have also seen clients holding out for candidates who have deep domain experience. I feel this is more a period of realignment than the role of a data scientist diminishing, but in my opinion, unless data scientists pivot to more machine learning-centric work then that market for data scientists is saturated with fantastic talent who are struggling to find work!”
Alex Hutchings, Co-Founder of Dataworks
“My assumption is that data science will head the way I think programming will evolve. Less generalist "coders" and more domain expertise solution creators and problem solvers. If you are a heads-down programmer who is good at structuring code, but doesn't know anything about the business or human problems trying to be solved you are in for a hard time. In that respect, I think data scientists have a leg up because they are usually more steeped in the domain and it might be *THE* technical role in the future.”
Ian Maurer, CTO at GenomOncology
As the rise of LLM applications continues, organizations are realizing, and executing on, the importance of data governance.
As data becomes more complex, it becomes harder to query. With text and multi-modal data becoming more commonly used, the data engineering behind querying this data at scale is only going to increase in sophistication.
In the past, data scientists were expected to do it all. Then data engineering became popularized at deep tech firms, the earliest builders of these multi-modal ML and AI solutions at scale.
Now, the separation between data engineer and data scientist is becoming clearer (more focused on strategy, statistics, measurement, algorithm development, and validation), with ML engineers filling in the gaps (one foot in algorithms and stats, the other in development).
“I think there's one more dynamic in play: The expertise requirements are quickly being lowered by new tools and solutions only beginning to appear. Companies today may be shifting hires towards the ML domain but don't be surprised to see traditional engineers increasingly moving into the space. New tools will play a large part, but so will survivalism—classic dev/engineering will evolve.”
Christopher Ernst, Co-founder of Roli.ai
Increasingly, data analysts are empowered with more tools allowing them to work with more complex datasets at scale.
Toolsets like Dataiku, Alteryx, and KNIME have rapidly increased the capacity for analysts who have quantitative and stats skills, but not necessarily advanced software engineering know-how to perform more effective analyses independently.
“Diminishing is a strong word, data scientists will remain masters of the art, wrangling algorithms and optimizing pipelines. However, as consumption increases and technical lift decreases, this will lead to the natural democratization of AI. Especially with generative AI and open models, 'traditional' software engineers will increasingly lead what must become bread-and-butter use cases as the art pervades the enterprise. Long live the Data Scientist, Long live the Enterprise Developer!”
Christopher Ernst, Co-founder of Roli.ai
In the past, data scientists were forced to take on data analyst work because they were the few with both the statistical knowledge and ability to code. Today, this is no longer the case.
“I think all of these categories ‘classic,’ citizen (as ill-defined as it is), and specialist will continue to grow, and grow fast enough that the classic role will not diminish. It will possibly grow more slowly.”
Keith McCormick, Recognized Analytics Leader
A third group involved with the design of responsible AI includes allied professionals — lawyers, risk managers, and cybersecurity experts — who ensure the reputation and safety of their organizations. While these individuals aren’t direct builders or users, they play a critical role in the adoption and implementation of trustworthy AI.
“I think we are seeing a change in the duties and the focus on what new hires duties are needed for building, maintaining, and operating a data science program. I don't think what has changed is as important as the maturity level of the organizations and programs the roles are hired to function in for successful outcomes.”
Aaron Smith, IT and Services Professional
Results from an MIT Sloan survey suggest that although excitement about AI is very high, value has largely not yet been delivered. In many cases, this pressure to get started with AI and demonstrate ROI leads to hasty, incorrect (although good-intentioned) hiring decisions.
“From my perspective, the data scientist role is not really well defined in business terms. We, "real" data scientists, know what it means. When I see how data scientists are categorized from a business perspective, it is not even close. Recently, I saw a job description for a data scientist that required Excel proficiency, with no other Data Science tool proficiency (Python, R, SAS, Cloud technology, SQL, etc) listed. I also have experienced organizations, who say that they are data-driven, who are really numbers-driven.”
Sarah Kalicin, Founder of Achieve More With Data
“There is an erosion of critical thinking and first-order principles in favor of business objectives and only telling the data “story” that supports a particular view or profit infrastructure. In lieu of statistical knowledge and actual science, we are in danger of being left with whatever part of the Gaussian curve we carve into our messaging.”
Bonny McClain, Spatial Data Analyst and Author
“If you're not keyed in on your business and the why behind your work, you're going to get left behind. I think AI will only serve to escalate the timeline.”
Barton Dunn, Director of Client Services
The survey results spoke loud and clear; they were very eye-opening to me. Data science is not going away, but the continued advancement and adoption of AI tools means more roles are in demand.
And it seems like this demand, at least for some time, will grow faster than traditional data science roles.
Now is a good time for organizations that have built a large bench of data science talent to look at more specialized roles and structures. These might include:
Technical specializations: Computer vision, natural language processing (NLP), ML engineering, large language model (LLM) development
Functional specializations: Strategy, value engineering, model development (more backend), application development (more front-end).
Industry specializations: Healthcare, energy, marketing, manufacturing
It's likely that your data scientists already fall into a specialization based on the tasks they're undertaking, but don't yet have a title that reflects it.
Over the years, I've found that there is a unique and growing demand for professionals with both a technical background in data science and a fine-tuned sense of strategy. I never thought I would experience joy in anything other than a technical role, but I've really taken to this trend.
You might too.
About the Author:
Cal Al-Dhubaib is a globally recognized data scientist and AI strategist in trustworthy artificial intelligence, as well as the Head of AI and Data Science at Further, a data, cloud, and AI company focused on helping make sense of raw data.
A special thanks to all of the contributing experts: Alex Hutchings, Ian Maurer, Christopher Ernst, Keith McCormick, Aaron Smith, Sarah Kalicin, Bonny McClain, and Barton Dunn.