Data Catalog

This 2-minute video describes the key data catalog concepts and benefits.

Why do you need a data catalog?
Your organization is likely flooded by large and complex datasets from many sources—financial systems, web analytics, ad platforms, CRM systems, marketing automation, partner data, and maybe even real time sources and IoT. Finding the right data and knowing you can trust it is a major challenge in the era of data lakes, big data, and self-service analytics.

• View business metadata and data lineage to improve understanding and trust.

• Apply personalized tags, properties, and business metadata for greater utilization.

• Browse dataset samples and profile statistics to ensure that data sets contain the expected information.

Today, many organizations are taking a product management approach to managing data by building data products. Data products are highly trusted, re-usable, and consumable data assets purposefully designed for domain-specific business outcomes. And a data product catalog provides a comprehensive repository that organizes and documents various data products within your organization. It serves as a federated resource, providing detailed information about each data product, including its purpose, data sources, processing methods, and intended audience.

A data catalog puts all your data into one simplified view where all users can more easily find, understand, and use any enterprise data source to gain insights. This brings your organization a competitive advantage, cost savings, operational efficiencies, and better fraud and risk management.

Here are the key specific benefits of an enterprise data catalog:

• Get data insights faster by having on-demand access to analytics-ready data.

• Trust your data by understanding its data lineage — a detailed history showing the original source and journey of this data.

• Make different kinds of data available to different kinds of users quickly, without compromising risk.

• Streamline the transformation of raw data into analytics-ready information assets through automated profiling and metadata tools.

• Make your data more understandable by collaborating with and capturing knowledge from different teams to enhance metadata.

Learn more about self-service data catalogs

A data catalog should be the single place for all users to find, understand, and govern data across your enterprise. Modern data catalog tools should make all your data transparent, trustworthy, and ready for analysis. Below are the key capabilities of the best data catalog tools.

• Data onboarding. Your data catalog should automatically profile and document the exact content, structure, and quality of any enterprise data as it is brought in from a source. It should generate and harvest rich metadata, and let you choose whether the data is added to the catalog data storage layer or kept at the source. Automated discovery of datasets will reduce manual effort during the initial catalog build and as new data sets are added ongoing. Built-in loaders will simplify the onboarding process and should support a wide variety of source types and locations, including RDBMS, mainframe applications, flat files, JSON, XML, Parquet, Avro, Qlik QVD files, AWS S3, Azure ADLS/WASB, and Kafka queues.

• Data cataloging. The core of the enterprise data catalog is its ability to enrich the catalog by identifying and describing every aspect of the data and data management process. AI and machine learning for metadata management, including collection, tagging, and semantic inference will reduce manual work in this process. Technical, business, and operational metadata makes each data element understandable, transparent, trustworthy, and actionable as you and other users explore the catalog. Data validation, profiling, and quality measures document the exact content and quality of each data source.

• Data searching. Your catalog must provide powerful, multifaceted search capabilities including the ability to search by keywords, facets, and business terms. Business users should be able to use natural language search and you should be able to perform advanced searches by specifying various parameters such as time, format, and owner.

• Data lineage. You need a catalog that gives you full visibility into the origin of your data, what’s happened to it and where it has moved over time. This will make it easier for you to trust your data, identify any duplicated datasets and trace errors back to the root cause.

• Data glossary. Your data catalog software should help you develop and share a data glossary (or dictionary) which defines the business terms and concepts you use in your organization. This will give you consistent business context across multiple tools. For example, everyone should be clear on what qualifies as a “Sales Qualified Lead” or an “Active Customer”.

• Data consumption. Easy, secure consumption by all types of users is another key capability. Your catalog should support one-time exports and the recurring, automated publishing of bespoke data sets to downstream data consumers, including data science or analytic platforms, applications, or cloud data stores. Simple integration with workflow schedulers and built-in event logging and notifications allow catalog jobs to be seamlessly integrated into your broader dataflow and application integration schemes. Sensitive fields should be obfuscated automatically, so data security is enforced, and you should be able to specify record layout specs, file format, and more parameters.