How Change Data Capture Cuts Costs and Modernises Applications in a Competitive Market

I consider myself pretty lucky: I love technology and get paid to pursue my hobby. I also get to say yes to opportunities that come my way that can increase value to my company and our clients. It was one of these opportunities that led us to adopt change data capture (CDC) technology, resulting in an added value proposition for our customers.

Based in the UK, CDL is a market leader in the highly competitive retail insurance sector. We offer software solutions that cover policy administration, real-time pricing, quoting operations, and data analytics to a variety of insurance providers, from small startups to global enterprise clients. When I joined CDL almost a decade ago, I started as an Oracle database administrator (DBA). Over time, CDL made some changes to capitalise on the shift in the world of data analytics. Our DBA team that had been solely responsible for Oracle expanded to what it is today: four product squads within the data services division.

We Were Proud of Our Data Architecture, But There Was Still More to Do

About a year after I joined CDL, we started to face two challenges. The first issue concerned our flagship product, Strata, our core sales and policy administration system. Strata stored all its data in an Oracle database, but certain data analytics functionalities were available only with the enterprise edition of Oracle. This limitation was fine for our larger clients but sidelined our smaller clients, who had been using Oracle Standard Edition. We’re a SaaS company—any product we sell, we want all our customers to benefit, regardless of their size. We’re all about growing our customers’ businesses because the more successful they are, the more successful we are. And to make that happen, we needed to provide all our clients with the same functionality.

In addition, we were operating two data centres in Stockport, outside of Manchester. These cost about £4 million each to build, and we were rightly proud of the high availability they allowed us to deliver to our customers. But it was a lot to maintain, and when the licence deal came up for renewal, the cost gave us pause. We decided to migrate those databases and determined that a CDC solution would enable us to reduce costs and improve functionality for everyone.

Picking a Dedicated Partner for a Big Project

From the start, Qlik was an obvious contender. They have a solid reputation and were on our shortlist of CDC tools to test with a proof of concept. But the selection came down to more than the technology. We selected Qlik for how they became a part of our team.

I wouldn’t call Qlik a supplier—they’re one of our technology partners, just as important as AWS. Part of our company mission is 'creating powerful partnerships', which is exactly what happened. Our customers are our partners, and we treat our technology partners similarly. Our OEM agreement demonstrated a genuine desire to work together and solve problems. Qlik worked with our commercial and accounting teams to develop a cost model that was in line with our budget, and they took feedback on board for product roadmaps. It was apparent that Qlik was there to help us deliver business value, and that sentiment continued as our relationship matured.

Our Qlik use has evolved since we first adopted the platform in 2015. Our first project was creating our business intelligence (BI) product, CDL Kingfisher. Then, we made a wholesale migration of 600 databases from our on-prem environment to AWS Cloud with Amazon RDS. That was a big project—we started by using the AWS Schema Conversion Tool to give us a report of everything that needed to change. We then stood up the database and enabled apps. When deploying this solution, all we needed was a source database, Qlik Replicate, Qlik Enterprise Manager, the target database, and a way for these components to communicate with one another. Qlik Replicate defines the tasks to process the data between the source and target, creating and replicating all our tables, primary keys, and all the data.

Having completed this migration, Qlik Replicate now also supports the varying profiles we need in our software as part of the Amazon RDS Service Ready program. At CDL, we hold two Amazon RDS Service Ready designations, signifying to customers that our products are fully integrated with AWS services. This guarantee allows customers to spend less time evaluating tools and more time scaling valuable products and services like ours.

Automation Helps Scale Our Greatest Resource

We immediately saw positive outcomes of Qlik Replicate and its integrations.

The biggest cost for any business is its workforce. My colleagues are highly skilled, and even if we had the budget to recruit a bigger team, we couldn’t scale our teams at the rate of our business growth. This gap is why businesses can’t underestimate the importance of automation—process automation means that as a product scales, the impact on the team stays the same. The combination of fewer licences plus the automation of processes has resulted in a 60% reduction in costs.

Businesses can’t underestimate the importance of automation—process automation means that as a product scales, the impact on the team stays the same.

Automation also reduces the risks of making manual mistakes. It’s key to remaining consistent when migrating environments so problems don’t arise in production. Automation delivers that consistency. It also helps us deploy faster because we can roll out changes quickly.

Having to support many different technologies wasn’t great for the cognitive load of the team. Managing everything through a single interface in Qlik has brought greater focus to the data teams and freed them to use their skill sets to solve more complex problems. Since we have such a highly skilled workforce, we want to put them to their highest and best use. In place of performing mundane tasks, engineers can now focus on delivering that next level of value for our customers as quickly as possible.

Offering Better Value and Getting to Market Faster

Our database migration journey began with Strata, which supports insurance providers in bringing products to market. Used by some of the UK’s largest personal insurance providers, the system generates millions of transactions every day. To provide our customers with analytics capability, we needed to make the Strata data model less complex and more consistent.

Additionally, we needed to customise Kingfisher, our BI solution that processes customer data into about 100 optimised analytics workloads. Though CDL Kingfisher worked well previously, it was only available to our enterprise licence customers. Moving to Amazon RDS allowed us to replicate these data visualisation tools for all customers. The Kingfisher platform now enables high-volume, accurate, consistent models with powerful and accessible analytics capabilities.

Automated, repeatable, near real-time analytics drive business insight and improve decision-making for customers.

CDL has always needed high availability and low latency to support our customers. With Qlik, we have that. We can update 100 tables with millions of rows without consuming precious resources, which speeds up the process considerably and gives our customers near real-time analytics. As we are running multiple migrations simultaneously, we can track the latency by task and over time.

Qlik Replicate works out of the box with many popular targets and all the major cloud providers, so we save time by simply configuring the target rather than building bespoke operations for our clients. These automated, repeatable, near real-time analytics and visualisations drive business insight and improve decision-making for our customers.

Modernisation Makes Us More Scalable

I’ll close with one more use case we’ve found for CDC technology. Many modern organisations are running legacy applications, and for whatever reason—maybe the person who built the application is no longer in the same position, or the application is using new source code—this application has become difficult to change. When that happens, adding functionality is not only difficult but also high risk. Using CDC, we can now mitigate that risk and allow new functionality using modern, event-driven architecture. This simplifies horizontal scalability and makes processes much more resilient to failure.

From new and exciting problems emerge new and exciting solutions.

It may sound strange, but I relish the chance to get involved in new and exciting problems, because out of these emerge new and exciting solutions. Bringing data together from different sources is critical to understanding what’s happening in the business. Working with Qlik Replicate and its related integrations means CDL offers customers more than just fast transactions; we offer detailed insights that lead to more revenue-generating activities.