Turbocharge Data Integration with Qlik's Unique No-Code/Pro-Code Approach

The world of Data Integration continues to evolve at a rapid pace, and Qlik has evolved along with it at equal pace, starting with acquisitions of Podium, Attunity, and now Talend, Qlik’s Data Integration and Quality solutions are a formidable force to have in your toolkit when you are looking to modernise data engineering in your organisation. Following on from the exciting announcement of Qlik Talend Cloud now Generally Available (GA) and the many tangible benefits it brings, I thought it would be useful to shine the spotlight on our unique no-code/pro-code approach to defining data pipelines and transformation logic and delve a little deeper into what, how, and why this is immensely valuable to our data engineer heroes out there.

Cloud Focused, but not cloud only!

First things, first. We are cloud focused but not cloud only, we realise there will always be a need for client-managed solutions as there is some data that absolutely, positively cannot leave the room! We won’t force your journey to the cloud, and we continue our commitment to the most relevant client-managed solutions you need. Just know that we are ready when you are, and when you get there, a whole new world awaits – anyone else hearing the music from Aladdin? (nope, just me then)

Taking a cloud-first approach enables rapid innovation and adoption, lets dig into Qlik Cloud, it’s not just for powerful Analytics, there is a whole host of Data Integration and Quality capabilities available today, which are only going to continue to increase and get better and better in the future. This is where Qlik Talend Cloud comes into play.

Creating data pipelines and defining transformation logic in Qlik Talend Cloud

We take a unique no-code/pro-code approach to help you create your data pipelines and define transformation logic with an intuitive drag & drop interface and automation running all the way through, it even includes AI functionality to help you deliver data projects faster than ever before, whether you are building data flows for analytics, machine learning and AI initiatives. As Drew Clarke said on the main stage at Qlik Connect, we are using AI to help you deliver AI.

It all starts with creating your data integration flow within a data project for your data platform that you want to deliver data to and any number of data tasks that are required to deliver the data exactly where and how you want it to be. A data project can be created for different use cases by choosing between Replication and Data Transformation pipelines.

Replication

You can effortlessly create straight forward data replication tasks that move data efficiently and securely from source to target or to land data to a data lake. Within this use case light-weight data transforms can also be applied such as column renames and adding new columns like timestamps. Most importantly it helps you move away for just batch processing and instead keep your data continually up-to-date with our best-in-class change data capture technology (CDC) that captures changes as and when they occur at source so you are always working on the freshest data and opens the door to real-time data delivery.

Data Transformation Pipelines

In addition to creating simple linear pipelines, you can also create data pipelines for all your data integration needs such as Data Warehouse modernization, perform transformations and create automated data marts to leverage your generated and transformed data sets. This use case allows for more complex pipelines consuming several data sources and generating many outputs. Within the data transformation pipeline use case there are several useful data tasks to enable your data warehouse modernization.

• Onboarding data – Use the Landing data and Storage tasks to move data into the project from data sources that are on-premises or in the cloud landing the data to a staging area, and then storing the data in ready-to-consume data sets in a cloud data warehouse.

• Transforming data – Use the Transform tasks to create reusable row-level transformations on the onboarded data based on rules that you create at a click of a button; this is our no-code approach as push-down SQL code is automatically generated for you to save manually creating hundreds and thousands of lines of code. Now you can also use our new ‘SQL assistant’ that utilizes AI to generate code for you just by writing a prompt in natural language. However, we know that won’t solve for ALL your requirements, while it gets you most of the way, there may be scenarios when you just ‘gotta code. That’s why we also offer a pro-code approach so you can create your own custom SQL code when you need to suit your own requirements.

• Creating and managing data marts – Use the Create data mart task with the data from Storage or Transform tasks. You can create as many data marts as your business needs. Data marts contain repositories of summarized data that will be useful for analysis on a specific section or unit within an organization, for example, the Sales department.

• Register Existing Data – You can even register data that already exists on the data platform to curate, transform data, and create data marts within Qlik Talend Cloud. This flexibility empowers you to also use the data that has been onboarded to the data platform with other tools outside of Qlik Talend Cloud, for example, Qlik Replicate, Talend Studio, Stitch or any other ELT/ETL tool you may have access to.

New Data Loader option

Qlik Talend Cloud now offers two methods of data loading capability, via a client-manged gateway and now a new gateway-less option.

Data Movement gateway – you can download a lightweight data movement gateway that you manage within Qlik Talend Cloud for point-to-point real-time data movement with change data capture (CDC). This option caters for sources that are behind a firewall and environments within a private cloud.

New Gateway-less Data Loader - We are pleased to introduce a new data loader capability that doesn’t require you to install a gateway. This new gateway-less data loader option in Qlik Talend Cloud is designed for hundreds of SaaS based cloud sources and some database sources and allows for easy connectivity and configuration. It utilises Full Load and change data capture (CDC) capabilities depending on the source, keep in mind not all SaaS based sources allow for CDC.

Getting Started

Qlik Talend Cloud is available in four editions Starter, Standard, Premium and Enterprise which you can find out about here.

Starter is the newest edition and as the name suggests, is aimed at getting you started with your data projects as quickly as possible. With this new edition you get access to the Replication use case so you can effortlessly start building simple, linear data replication tasks and prove value at speed. With this new edition we are starting to bring the best of Stitch technology into Qlik Talend Cloud and building on it to make it better! Just like Stitch it also comes with a nice simplified, intuitive user interface avoiding the need to learn about complex advanced options, and it utilises the gateway-less data loader. This just the beginning, over time we are looking to bring in more Stitch tech into this edition, so stay tuned for more news coming soon.

Starter edition offers a frictionless and lower cost entry into Qlik Talend Cloud, but it doesn’t have to stop there. You have many more exciting functionalities at your fingertips, all within the same environment, and they are only another edition away. Each edition has the same functionality as the previous edition, so should you have a need for more complex data transformation pipelines, SAP and Mainframe sources and delivering data in real-time with change data capture (CDC) now or in the future then you can get access to these capabilities and much more in the other editions such as Standard, Premium and Enterprise.

New SaaS sources

You may have heard of the Qlik Connector Factory this is a combination of technology and methodology to bring you connectivity to the sources you need access to the most!

We just announced over 100 new connectors and many more will continue to be made available (Read more on Qlik Community)

How do you make the best, better?

At Qlik we love to listen to your feedback and many product innovations and improvements are developed on your valuable feedback, to this end we have our Ideation section on Qlik Community where you can submit your ideas on what will make our products even better, you can also search for ideas that have already been submitted and vote on them. Ideation is used by our product management teams to plan for the next releases and this is also the best way for you to request a data source should you have a need that we don’t currently support. Search and submit your ideas here.

Conclusion

There is no doubt that we are living in a modern world that is fast paced and ever changing. And when it comes to your data engineering methods and strategy you need to be able to quickly adapt to these changes. Qlik Talend Cloud is ideally suited to help you not just adapt and pivot but truly thrive with increasing demands of new data, more data and fresh data.

Learn more about Qlik Talend Cloud and why not take a guided tour to see some of the capabilities on offer first hand, if that wets your appetite get in touch and we’d love to discuss your requirements in more details and help you become an even better data engineer hero.