Decoding ‘Giant, Convoluted’ Data — Strategies for Effective Management and Compliance

Big Data is getting bigger. More data was created between 2018 and 2020 than in all of human history prior, according to the U.S. International Trade Commission. In 2023, Statista found that global data creation hit an impressive 120 zettabytes (120 trillion gigabytes).

Optimists have long predicted that Big Data would be the solution to drive innovation and enable more informed decision-making across these large swathes of data.

However, despite every sector being increasingly connected and reliant on data-driven decisions, the core challenges in accessing and using data are the same today as when the term was first coined 30 years ago, especially for the compliance sector, which encompasses the regulations and practices that organizations follow to ensure they meet legal, ethical, and regulatory standards.

Both direct and indirect regulations impact data use on a global and local level. Jurisdictions like the U.S. and EU have very different legislation related to data retention and utilization.

In the U.S., for example, data governance is largely shaped by sector-specific regulations such as the Health Insurance Portability and Accountability Act (HIPAA) for healthcare and the Gramm-Leach-Bliley Act (GLBA) for finance. Meanwhile, the EU enforces comprehensive frameworks like the General Data Protection Regulation (GDPR) and the Data Governance Act (DGA), which emphasize strict data protection, privacy rights, and the facilitation of data sharing across member states. Data-centric industries with high compliance standards have even more nuanced legislation and regulations to adhere to.

The compliance professionals need to be more purposeful with how they manage Big Data before it becomes a “Giant, Convoluted Data” problem.

The promise vs the reality of Big Data

In the 2000s, Big Data was sold as super-charged analytics with three key characteristics: volume, velocity, and variety. Technology was improving in both processing speed and storage capacity, and analysts predicted that complex algorithms would keep things running flawlessly. By increasing effectiveness in those key areas, enterprises were meant to increase overall efficiency — but the reality tells a different story.

With more data being created at a faster pace than ever, managing it is more complex and daunting than it used to be. Consider how Google’s AI-powered search tool is reportedly generating worse results than it did a year ago, let alone compared to its legacy search product.

The compliance industry serves as a microcosm for these broader data challenges. Compliance professionals monitor and report on a vast array of regulations, from employee conflicts of interest like political contributions and crypto pre-clearance to incident and policy management. This requires the aggregation and analysis of data from multiple sources, including employee records and activities, as well as a variety of other data stored in both internal and external databases.

Since much of this data is stored on different platforms, achieving a comprehensive view of regulatory risk is difficult. There are three other “Giant, Convoluted Data” challenges standing in the way of better data management in the compliance sector:

1. Incomplete aggregation and synthesis of risks

Siloed and fragmented data leads to a partial view of organizational risks. Without effective aggregation and synthesis of diverse risk and compliance data, organizations may overlook critical risk factors or fail to connect the dots between different compliance requirements. This incomplete data integration can result in unexpected regulatory issues, financial penalties, and reputational damage. Integrating data from various sources and establishing a centralized risk management framework is crucial for comprehensive risk assessment.

2. Inefficiency and human error

Another major obstacle is the reliance on manual data processing and the high potential for mistakes. Manual data entry, spreadsheet-based reporting, and siloed data storage can introduce human errors and inconsistencies, undermining the reliability of data. This inefficient and error-prone process can lead to delayed decision-making, missed deadlines, and increased compliance risks.

3. Technology and infrastructure lags

Despite advancements, there are gaps in leveraging current technologies for data management and analysis. Legacy systems, outdated infrastructure, and a lack of data integration capabilities can hinder an organization’s ability to harness the full potential of data.

Adopting modern data management platforms, cloud-based solutions, and advanced analytics tools can enable more efficient data processing, real-time insights, and predictive risk modeling. Investing in a robust data management infrastructure and defining clear data governance policies, along with upskilling employees to leverage these technologies, is crucial for staying ahead of the evolving data landscape.

If you still aren’t sure whether your organization needs to apply mature data practices toward your compliance initiatives, ask these questions:

• Are you able to access and analyze data across your organization seamlessly?

• Do siloed data and manual processes hamper your decision-making?

• How well can you leverage data for predictive analytics and risk management?

This self-assessment is necessary to determine whether current data practices are future-proof and aligned with the challenges involved in developing robust compliance programs. 

Solving Big Data’s “Giant, Convoluted” problem

Despite its evolution into “Giant, Convoluted Data,” Big Data can still be transformative. Organizations that effectively harness the insights hidden within their data are poised to gain a significant competitive edge. The key lies in embracing change and seeking solutions that not only tackle the sheer volume of data but also streamline its complexity into actionable insights.

To achieve this, adopt a data-driven mindset from the top down, fostering a culture that values evidence-based decision-making such as in the compliance arena. This starts with understanding the importance of data and how it can improve operations, from identifying best practices to forecasting future challenges.

The right data management process, tailored to the specific needs of the compliance industry, can make all the difference. By transforming the “Giant, Convoluted Data” challenge into a strategic advantage, you can unlock new opportunities for growth, innovation, and operational excellence, positioning your organization for success in the years to come.

About the Author:

As CTO of StarCompliance, oversees the company’s technology footprint—how Star products and solutions are delivered to market as well as the firm’s internal corporate technology. Rowland joined Star in 2020 after 19 years at Morgan Stanley, where he was Global Head of Technology for some of the bank’s largest revenue-generating businesses—including Sales & Distribution, Investment Banking, and Capital Markets—and where he regularly delivered enterprise-ready solutions that scaled to the needs of thousands of users.

Rowland earned his Bachelor of Science, Computing for Business, at the UK’s University of Northumbria.