Qlik Data Integration and Microsoft Change Data Capture for Azure SQL Database Brings Value to a Maturing Cloud Marketplace

**Over the past few years, cloud usage has exploded to power virtually every company’s growth initiative and digital innovation.**

Indeed, Gartner estimates that by the end of 2022, Public Cloud spending will reach nearly half a trillion dollars. Unsurprisingly, Enterprises now rely on the cloud for multiple needs: a data repository, a production system, a central hub for analytics, etc., that places a premium on data availability, recency, and quality. Consequently, the value of new and relevant data has never been higher, and that’s why modern change data capture solutions continue to be in demand.

What’s also increased alongside cloud’s growth is the range and type of enterprise data sources. Most modern enterprises now need the ablity to efficiently leverage fresh data from thousands of sources and typically land that data in a warehouse somewhere. Indeed, one of those oft-used data sources is Microsoft SQL Server. In fact, Microsoft’s recent announcement of the general availability of Microsoft CDC for Azure SQL Database, in which we were a proud launch partner, mirrors that trend.

You may be asking, doesn’t Qlik provide Change Data Capture (CDC) through Qlik Data Integration? Why yes, we do! But this news isn’t a competitive issue between Qlik and Microsoft. Before we go further, let me explain.

CDC has been a feature of Microsoft SQL Server for decades, and this new announcement adds CDC support to Azure SQL Database. So it makes it much easier for customers to leverage Qlik Data Integration’s change data capture to acquire new data for more efficient analytics. Incidentally, this data source was officially added to our May release and is fully supported today.

Let’s dive a little deeper into the specifics:

Microsoft’s traditional change data capture (MS-CDC) method uses a SQL Server agent to record all table inserts, updates, and delete activity in database transaction logs. Qlik’s Data Integration then periodically reads those logs and applies the changes to the target data systems. Microsoft CDC for Azure SQL Database differs since it uses change tables instead of the SQL Server transaction logs for CDC. So Qlik Data Integration periodically reads those tables instead of the log too!

Also, the great thing about the new Microsoft endpoint is that it can provide a more reliable solution for replicating data with Qlik from SQL Server cloud sources such as Azure SQL MI, because online transaction logs may only be available for a short period of time, and transaction logs may not be accessible at all.

So why should you care? In the words of my colleague Kathy Hickey, Vice President, Product Management at Qlik.:

“Joint customers are excited about the potential of leveraging Qlik Data Integration alongside CDC in Azure SQL DB and CDC for SQL MI to securely access more of their valuable data for analytics in the cloud. We are happy to announce that in addition to support for Azure SQL MI as a source, the newly available MS-CDC capabilities will also allow us to support Azure SQL DB sources via our Early Access Program. We look forward to partnering with Microsoft on helping customers leverage these capabilities to confidently create new insights from their Azure managed data sources.”

To learn more about Qlik Data Integration and Microsoft, visit Microsoft Azure Migration & Data Streaming | Qlik.