Unlocking Data’s Hidden Treasures

If you asked any CEO today what their organization’s most important asset is, a common response would be their people, and rightly so. But if you asked them which is the asset that helped their company create an “unfair advantage”, you may get a different response.

Frans Feldberg, in a fascinating talk at Qlik’s AI Reality Tour event in Amsterdam that I had the pleasure to attend recently, framed it like this. Uber is a car service company that doesn’t own any cars. Airbnb is a real estate rental company that doesn’t own any real estate. What is their most prized asset that has given them a unique advantage in the market? It’s data. Indeed, Uber’s vast trove of real-time and historical transportation data enables dynamic pricing, efficient driver-rider matching, and route optimization. And Airbnb's extensive collection of user preferences, booking patterns, and property information, allows for personalized recommendations and pricing optimization for hosts.

Companies that recognize the value in their data, and manage it effectively to unlock it, end up opening a treasure of new and unique opportunities for their business. I have worked in the data space for over two decades, and I have seen this play out time and time again. So, after exploring how companies can get the unfair advantage through a great tech ecosystem, and by building resiliency, I thought it was time to investigate what I think is another critical path to organizational success: unlocking your data’s hidden treasures.

Hidden Treasure #1: Unstructured Data

Unstructured data – the information that is locked away in your text files and documents, emails, video files, images, etc. - is arguably the biggest hidden data treasure. It likely represents about 80-90% of an organization’s information, growing at an exponential rate… and largely remaining untapped. A study that the firm ETR conducted for Qlik in April 2024 found that while companies understood the value potential of being able to deliver insights from that data, less than one third felt their organization was well equipped to do so. So, for organizations that have been able to harness it, this creates a clear advantage.

This has been the case for Airbnb, which created a market edge with a great platform that connects a vast network of unique accommodations with travelers seeking personalized experiences, and which continues to show strong financial performance despite some signs of slowing growth and market normalization. The company effectively utilizes unstructured data to enhance customer experience. They developed a system called the Listing Attribute Extraction Platform (LAEP) to gather important information about property listings from unstructured text data, like descriptions and reviews. Instead of relying on hosts to provide all details manually, LAEP automatically identifies key features of listings, such as amenities and facilities. The system works by breaking down text to recognize specific phrases, matching them to standard categories in Airbnb’s database, and assessing how confidently it can identify these features. By analyzing various sources of guest interactions, LAEP helps Airbnb better understand what makes each listing unique. This automated approach not only improves the accuracy of listing information but also enhances the overall experience for guests. It allows Airbnb to offer more personalized services and make informed decisions about how to present properties on their platform.

AI Plot Twist: Generative AI is providing organizations with new ways to harness unstructured data and generate insights from it without needing to build it all yourself. At the same time, your AI models need all the right contextual data – structured and unstructured - to generate content and provide answers that are relevant and accurate.

Bottom Line: whether it’s to make the most informed decisions for your business, enhance customer experience, or to ensure you create value with your AI projects, unstructured data is a hidden data treasure that is critical for you to unlock.

Hidden Treasure #2: Real-Time Data

It’s easy to see the market advantage real-time data can provide organizations when it allows them to make swift, informed decisions, respond instantly to market changes, optimize operations, manage risks more effectively, or deliver personalized customer experiences. But as I see it, real-time data remains a hidden treasure for many organizations: it is there but is not yet being fully leveraged. There is a variety of reasons for this being the case, ranging from the technical complexity in implementing real-time systems, substantial costs associated with upgrading infrastructure and acquiring skilled personnel, and data governance issues related to ensuring quality and security in real-time environments.

Uber, the company that, to quote their mission, “reinvented the way the world moves for the better”, has distinguished itself through its ability to harness real-time data. The company's sophisticated Gairos platform processes and analyzes enormous streams of instantaneous information from diverse sources, including rider requests, driver locations, traffic conditions, and weather patterns. This constant influx of data fuels critical functions such as dynamic pricing, which adjusts fares in real-time based on current supply and demand. Uber's advanced matching algorithms utilize up-to-the-second data to efficiently pair riders with nearby drivers, significantly reducing wait times. The company also equips drivers with live heat maps displaying areas of high demand, enabling them to strategically position themselves for maximum efficiency. Finally, Uber's robust stream processing infrastructure handles petabytes of data daily, supporting real-time decision making across all aspects of the business, from customer experience to operational efficiency and market expansion strategies.

AI Plot Twist: Timely data is critical also for your AI projects. You may remember that early versions of ChatGPT could only retrieve information up to September 2021, which, needless to say, was a huge hinderance in helping ensure accuracy of the information it generated. Timely data requires the right technology solutions include change data capture to locate and record high-velocity database changes and instantly send those updates to a system or process downstream, stream data capture to capture data emitted at a high volume in a continuous, incremental manner with the goal of low-latency processing, and continuous data processing to instantaneously update downstream data stores (operational and analytical) for the latest data.

Bottom Line: as timely data is one of the six data principles for AI-ready data, and it informs better decisions, it’s critical for your organization to be able to harness it.

Hidden Treasure #3: External Data

External data, that information generated and collected from sources outside of a company's own operations, is another hidden treasure for many companies. But while organizations have made progress in utilizing their internal data, relatively few have fully leveraged the power of external data sources. Companies that successfully integrate a wide range of external data into their operations can get valuable insights that complement internal data and offer a broader perspective on markets, customers, and trends, and ultimately gain significant advantages in growth, productivity, and risk management. But leveraging external data presents several challenges, including navigating a fragmented and rapidly expanding data landscape, evaluating the quality and economic value of data products, updating existing data infrastructure, and addressing privacy concerns and consumer scrutiny.

Netflix's strategic use of external data has provided the company with a significant market advantage in the streaming industry. By leveraging a wide array of external sources, including box office information, critic reviews, social media trends, and performance data from other platforms and networks, Netflix gains valuable insights into audience preferences and emerging trends. This external data informs critical decisions in content acquisition, production strategies, and talent recruitment. For instance, Netflix uses industry-wide performance metrics and cultural data to evaluate potential new content and understand regional preferences. In negotiations with top talent, the company can present comprehensive data on how a creator's work performs across various audience segments, demonstrating the potential reach and impact available through their platform. Additionally, Netflix utilizes external technical data, such as internet speed information, to optimize streaming quality in different regions. This approach has enabled Netflix to make more informed decisions, attract high-profile creators, and tailor its content and technical performance to diverse global audiences, ultimately strengthening its position as a leader in the competitive streaming market.

AI Plot Twist: While not every AI use case in your organization will benefit from external data, there are many that will need it. Key applications include market intelligence for analyzing trends and competitor activities, supply chain optimization using factors like weather and traffic data, customer insights for personalized experiences, risk management and fraud detection leveraging public records and social media, and predictive maintenance through equipment data and historical records. However you decide to approach this, make sure you use external data in a way that is ethical and responsible.

Bottom Line: your organization is impacted by external factors so you should always consider how the unique combination of your data with external data will give you an unfair advantage.

Bonus Hidden Treasure: Multi-Modal Data

What if you could leverage all of these types of data together to drive an outcome? That is effectively multi-modal data: as we saw, unstructured, real-time and external data each hold great value potential; but using them in combination acts as value multiplier.

Think of the value this can bring to a healthcare organization that can leverage a patient's electronic health record, an X-ray image, a radiologist's description of an X-ray, and blood test results combined to improve patient diagnosis. Mount Sinai Health System employs multimodal data to enhance patient diagnosis by integrating diverse sources such as electronic health records, medical imaging, genomic data, and clinical notes. Through advanced AI and machine learning algorithms, the system analyzes this comprehensive data to improve diagnostic accuracy and predict patient deterioration in real-time.

AI Plot Twist: Gartner predicts that by 2027, 40% of generative AI solutions will be multimodal, up from just 1% in 2023, highlighting the potential for these models to capture relationships across different data streams and support more complex human-AI interactions. This shift is expected to provide competitive advantages and accelerate time-to-market for enterprises.

Bottom Line: Putting all the right data in your corner will help you unlock the most value.

Don’t Forget Your People

With all that said, I strongly believe that data alone is not enough to deliver magic for your organization. To truly unlock its potential, you need the right people with dual data and AI literacy skills, possessing the ability to not only interpret and analyze data but also understand the capabilities and limitations of the AI systems that leverage it. Ultimately, it's the synergy between data and people that creates the magic, enabling companies to pivot quickly, innovate effectively, and create their unfair advantage.