Overcoming Barriers to Enterprise AI Adoption — Insights from Cloudera’s 2024 Survey

IT leaders are revisiting their data architectures to adopt and leverage artificial intelligence (AI), but most find they are met with common barriers.   

Cloudera’s report, The State of Enterprise AI and Modern Data Architectures, is a global survey of 600 IT leaders regarding AI. It dives into how organizations view the technology in relation to their environment, the barriers they face preventing AI adoption at scale, and their biggest concerns with AI. 

It is no secret that AI has incredible potential to boost productivity and even change the way people work. For this reason, organizations in all industries find it critical to leverage this technology to improve efficiency and keep up with competitors. But, despite the proliferation of AI apps, critical obstacles must be overcome to unleash AI power. It is an urgent race like the 100-meter hurdles — as organizations that get their generative AI (GenAI) apps over the finish line see top-line and bottom-line growth over their competitors.

With traditional analytics technologies, the mere mitigation of barriers such as data silos, ungoverned and low-quality data, and data privacy risks could improve the accuracy of insight generation, but for the accuracy of GenAI applications, those barriers must be removed.

Here’s how:

Aggregate data on a unified data platform 

Nearly 70% of respondents said their data was at least somewhat siloed with nearly 40% describing their data landscapes as “completely siloed” or “very siloed.” Compounding the challenge of unifying data is the fact that it is spread across multiple form factors. Data is everywhere.  

Open data lakehouses help break down silos, yet most cloud-native data management tools are not integrated, and few are compatible with on-premises tools. Therefore, IT teams should select a true hybrid platform with portable, interoperable services that facilitate write-once-run-anywhere workloads. The technology exists. True hybrid platforms also enable uniform data governance across the data estate and facilitate central command and control of data security.

Reduce data security and compliance risks while cutting costs

Justifiably, 71% of respondents are worried about the security and compliance risks of AI (opening graphic), with an equivalent percentage of respondents already struggling with data security and reliability (graphic below). Also challenging for many are data management costs (52%) and governance issues (36%).

The key to simultaneously reducing both the risks and costs of AI is to embed cloud-native, open-source, foundational AI models into private, secure data environments where models can be trained safely.

With this approach, models are isolated in your environment and cut off from third parties who have no access to your data, the training, the prompts, or the responses. 

Thereby, data security and compliance risks are significantly reduced by directly prompting cloud-native AI models; in fact, the risk of AI is mitigated down to your current risk profile. Embedding models also reduce costs, especially if your only alternative is to build LLMs from scratch, but also for use cases where model training requires less data (terabytes vs. petabytes), as the training can sometimes be accomplished via CPUs vs. GPUs.

Hike the upskill hill at pace 

(for most it need not be a mountain) 

Despite 43% of respondents reporting that they do not have the proper training and talent to manage AI tools (opening graphic), their responses indicate that the majority are already overcoming skills gaps using multiple methods.

Most organizations are utilizing in-house training and development programs (59%), cross-functional AI teams and projects (59%), hiring and talent acquisition strategies (54%), or a combination of these and other methods.

Yet competition for highly-skilled AI resources has skyrocketed, with some companies acquiring talent only through corporate acquisition. For most, that option is neither available nor necessary. Fortunately, there are ways to overcome these hurdles through methods attainable by most.

Bring open-source models to the data

Open-source models reduce the technical skill required to pilot and deploy AI applications. Data analysts can embed and train open-source foundational models in a RAG architecture while data engineers acquire the technical skill to create vector embeddings and fine-tune models. There are many viable use cases that utilize RAG to deliver operational efficiencies: marketing content generation, market research and summarization, sales communication campaigns, and contract summarization and management to name a few.

Finally, all coders should be upskilling by using copilots to generate, document, and optimize code. This is an easy training strategy for developers to not only uplevel their skills but simplify the tasks of working with AI tools.

Adopting AI technology comes with a variety of common challenges, but they are not insurmountable. Overcoming these hurdles simply requires a clear plan and a little legwork. It all starts with working towards a modern data architecture. To overcome AI adoption barriers, focus on bringing your data together and making it trustworthy, and keep it simple by choosing use cases and AI architectures that are easier to execute. Simplifying your plan will lead to a stronger likelihood of execution.

About the Author

Shayde Christian is Chief Data and Analytics Officer at Cloudera. He guides data-driven cultural change for Cloudera to generate maximum value from data. Christian enables customers to get the absolute best from their Cloudera products such that they can generate high-value use cases for competitive advantage.

Previously a principal consultant, Christian formulated data strategy for Fortune 500 clients and designed, constructed, or turned around failing enterprise information management organizations. He enjoys laughter and is often the cause of it.