Managing Active Metadata with a Data Knowledge Catalog

As modern businesses look to drive greater value from their data, many are exploring new strategies and approaches to deliver clear and trustworthy information in support of business decisions. For many organizations, this means embracing a culture of data governance as an enabler of the data-driven organization and shifting their approach to metadata management through the use of a data knowledge catalog. Together, these approaches bring data to the end users in a user-friendly way while still employing the policies and processes needed to ensure the data is integrated, protected, and secure.

The Evolution of Metadata Management

In its most basic form, metadata is data about data. It provides context that helps users find the data they need easily and understand key attributes such as ownership and lineage so they can use it more effectively. While metadata management or the data management processes that manage the context of the data isn’t new, many organizations are realizing that their traditional approaches are no longer sufficient in today’s world where data volume and complexity are growing at a staggering rate.

For years, passive metadata management successfully delivered basic technical information about the data, including the data’s profile and the operational features such as who accessed which data and how often they did so. While useful, relying on passive metadata alone is no longer sufficient for organizations that wish to be truly data-driven.

Active metadata, on the other hand, introduces richer contextual elements across all levels of the data stack. It is generally more complex than passive metadata, spanning operational, business, and social metadata as well as technical metadata. It adds value to the data and makes it more meaningful, which, in turn, helps end-users make more meaningful decisions with it.

Active metadata enables a dynamic metadata management model that enables organizations to reveal a more holistic data story that goes well beyond the static data profile. It shows how and where data flows in the data infrastructure, including all modifications, data transformations, and calculations made up to that point.

By employing artificial intelligence and machine learning algorithms to analyze their data, organizations can automatically list, tag, classify, and inform the origins of the data to understand data lineage. Using this information, organizations can uncover new patterns and identify potential blind spots in their data stack, allowing them to fix them before they become an organizational-wide problem.

In fact, according to Gartner, “Through 2024, organizations that adopt aggressive metadata analysis across their complete data management environment will decrease time to delivery of new data assets to users by as much as 70%.”

Elevating the Role of Metadata

For years, organizations have been using metadata to manage their data. But as data becomes increasingly more strategic to an organization, the role of metadata, specifically active metadata, becomes increasingly important as well. Gone are the days when metadata simply provided documentation about the data profile or usage of the data. Instead, active metadata now plays a critical role in the management of data access, data classification, and data quality.

Active metadata forms the foundation for modern governance and data management, benefiting organizations in six key ways:

1. Purging outdated or unused data
   Using active metadata, organizations can systematically determine the last time someone within the organization used a data set or batch of data. They can reveal the number of people who have used the data within a given period. And, using systematic rules, the organization can archive or purge the data when it hasn’t been used for a defined duration.
2. Dynamically allocating data processing resources
   Active metadata management enables organizations to scale up or down their IT resources based on peak periods of usage of BI tools. For example, if an organization knows that more users access their BI tools during the end of a fiscal quarter, they can scale resources accordingly to accommodate this peak load.
3. Enriching the user experience in business intelligence tools
   Active metadata management allows organizations to bring richer context to dashboards within their BI tools which drives increased efficiency and a better user experience. Instead of switching between the BI tool and a data catalog, users can view a table within their BI tool and see relevant metadata including business terms, descriptions, data ownership, and history.
4. Automatically classifying sensitive data for compliance and governance
   Using active metadata management, organizations can classify sensitive and protected data and control who can - and can’t - see it, making it easier to comply with regulations as well as their own governance policies.
5. Identifying the most frequently-used assets across the organization
   Active metadata management enables organizations to create custom popularity scores for each data resource based on usage information from query logs, provenance, and BI dashboards, enabling them to identify the data sets accessed most often in reporting and analytics. The organization can then check these resources regularly for data quality issues and ensure they appear more frequently in search results.
6. Alerting downstream users to resolve issues quickly
   Using active metadata management, the data team can receive a notification when a user modifies a database or creates a potential anomaly, allowing them to review the data and determine if the change caused an error. If it did, they can quickly notify the user of the error or correct it themselves to maintain the integrity of the data.

The Role of a Data Knowledge Catalog in Active Metadata Management

According to Gartner, “Existing metadata management tools are increasingly incapable of fulfilling comprehensive metadata needs in the enterprise.”

That’s why organizations that employ a data knowledge catalog accelerate their efforts to manage metadata more strategically and aggressively in both traditional data management as well as in a data fabric environment.

A data knowledge catalog goes beyond the capabilities of a traditional data catalog by including metadata mapping, active metadata management, and metadata knowledge sharing capabilities. Built for business users, a data knowledge catalog provides a seamless experience that delivers data in a user-friendly way, eliminating the need for extensive searching. Through in-app and in-browser features, users can easily share data knowledge and collaborate to deliver better data and analytics across the business. 

In a data fabric environment, a data knowledge catalog can help improve data governance by providing a centralized platform for data management, ensuring that data is consistent, accurate, and secure. A key benefit of a data fabric architecture is its ability to provide a unified view of data across an organization by providing a single, standardized platform for data management. By incorporating a data knowledge catalog as part of the data fabric environment, organizations can break down data silos, share data knowledge, and create collaborative governance that enables teams to work more efficiently. 

As active metadata management technology matures, organizations will recognize the need to move beyond the capabilities offered in traditional data catalogs to embrace the machine learning-driven capabilities of a data knowledge catalog that enable them to orchestrate more advanced active metadata management use cases across the organization.

DataGalaxy: A Representative Vendor in Gartner’s Market Guide for Active Metadata Management

According to Gartner, “Metadata management is becoming a critical functionality in practically all data-enabling technologies and metadata analytics, augmented and automated design, and even deployment of data management platforms.”

They go on to state, “Increasingly, the metadata-focused solutions will harvest, analyze, and evaluate metadata internally, but more importantly, they will begin orchestrating metadata processes across a broad spectrum of other data management tools.”

Named a representative vendor in Gartner’s Market Guide for Active Metadata Management, DataGalaxy provides active metadata management capabilities that extend beyond those delivered in a traditional data catalog. Dedicated to providing metadata mapping, active metadata management, and metadata knowledge sharing for the largest number of people, DataGalaxy’s data knowledge catalog delivers key capabilities including the ability to:

• Enhance user experiences within business intelligence tools by integrating relevant metadata into an organization’s BI tools

• Classify sensitive data to enable automatic compliance with regulations by customizing access policies according to the organization’s defined governance strategy 

• Identify frequently-used assets and create a popularity score based on usage information from sources such as query logs, data provenance, and BI dashboards

• Integrate with collaboration tools such as Slack and Microsoft Teams to provide a seamless, collaborative experience for business users

When organizations integrate a data knowledge catalog into their data management processes, they are better positioned to accelerate their use of active metadata management to increase collaboration and productivity, champion data governance, and enable data literacy. 

To learn more about the key capabilities required for effective active metadata management, and to assess the maturity of your organization’s active metadata management technology, please download Gartner’s Market Guide for Active Metadata Management, complements of DataGalaxy.

Gartner does not endorse any vendor, product, or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

About the Author

Sebastien Thomas is the CEO and co-founder of DataGalaxy, the Data Knowledge Workplace. With over 20 years of experience in the field of Data Governance, Sebastien is deeply passionate about bridging the gap between IT and business teams through comprehensive knowledge of their data.

About DataGalaxy

About DataGalaxy Founded in 2015 in Lyon, DataGalaxy is the pioneer of collaborative data governance in France. DataGalaxy is the industry’s first data knowledge catalog helping organizations understand how their entire business runs on data. An established leader in Europe, growing rapidly and operating worldwide, DataGalaxy offers a user-centric platform dedicated to metadata mapping, active metadata management, and metadata knowledge sharing. With its innovative approach to data governance and cataloging, DataGalaxy helps businesses of all sizes gain control over their data assets and make better, more informed decisions. www.datagalaxy.com