Talend - CDP Integration Reference

[#1]

Introduction

This page provides a High-Level understanding of how an enterprise might leverage Qlik/Talend integrations to run big data Spark workloads on Cloudera Public Cloud CDE.

A standard setup for integrations could be represented as indicated in the diagram that follows (Fig 1):CDP-PC Talend Cloud Integration Reference

Primary Talend components required to showcase integration are as follows:

• A Trial or Enterprise account on Qlik/Talend Cloud

• Installation of the Talend Studio Application on the user workstation with network line of sight to CDP APIs and endpoints

• Installation of Talend Remote Engine in or alongside the CDP Environment with network line of sight to CDP APIs and endpoints.

Download and install instructions for Studio and Remote Engine are available from the Qlik/Talend Cloud UI available after trial activation (Fig 2)

Download resources for Talend Cloud Integration setup

Instructions for Talend - Cloudera Data Engineering Spark  integration

The following setup instructions are made available for an SE that wants to investigate the Talend-CDP integration. In the following example we have focused on an execution and orchestration of a simple Talend Spark application on a CDE dataservice running within the CDP environment.

Spoiler

:warning: Gotcha: Talend Cloud has no requirement to publish to, or consume from the Git project repository. So credentials to access the repo are not required to be persisted in Talend Cloud. Users interacting with the project from the Talend Studio Application are expected to have read and write access on the  chosen Git repo and provide appropriate credentials

:warning: Help: Define your Talend project on a fork of this Github repo to eventually access the Talend CDE job definition used for the remainder of integration reference Demo repo

1. On Talend Cloud UI set up the demo Project:
   • Set up your Talend project referencing the chosen Git URL (Note Git credentials are not required to be persisted on Talend Cloud)
     image-20260109-142507.png
2. On the new Project in Talend UI assign yourself (your Talend Cloud Identity) as a project collaborator:
   • Set up privileges on the project as a project collaboratorimage-20260109-145919.png
3. Install the Talend Remote Engine on a Linux instance in the CDP env
   • There is no real restriction on where we can install the Talend Remote Engine. For the simple use case showcased here, I have chosen a COD edge node to host the Talend Remote Engine. Instructions for installation are available here Remote Engine on Linux
   • image-20260109-164600.png
4. Install the Talend Studio Application on your workstation

   • Instructions for Talend Studio setup are available here Talend Studio Install

5. Login to open the Talend Studio application
   • Once Talend Studio is installed on the user workstation, open the Talend Studio application. Login is to be performed with Talend cloud credentials . Configuration of login must leverage the appropriate Web-app URL for your allocated Talend cloud tenantimage-20260109-145659.png

   • After logging in Talend Studio will present you with a list of project definitions (Defined on Talend Cloud) that you want to interact withimage-20260109-150205.png

   • Choose the appropriate Talend Cloud project.  The Talend Studio application will launch with the Talend Job Designer Canvasimage-20260109-150550.png
6. Design the Talend Jobs to be executed on the CDP ecosystem
   • Developers can now start the process of designing Talend Jobs to be executed on the CDP ecosystem. In this documentation and defined in the referenced Git repo a simple spark job is predefined in our Talend cloud project
   • Open the CDE job cde1 0.1 from the Repository browser on the Talend Studio UIimage-20260109-151429.png
   • Configure the Spark Configuration of cde1 0.1 as appropriate for your target CDE endpoints.  :warning: Gotcha: Note the quotation in the image above .  Required params are as follows :

     • CDE API endpoint

     • CDE Service API endpoint (used for token retrieval)

     • Workload username & password

   • Once Spark configuration params have been populated you can test and run your Talend job directly from Talend Studio. 
   • :warning: Gotcha: It is expected that the workstation hosting Talend Studio application will have network line of sight to these CDP endpoints image-20260109-152423.png
   • Talend  Spark Jobs will populate CDE  resources with Talend Runtime Libs image-20260109-154159.png
   • Talend Studio leverages CDE Java SDK to upload the Job Artifacts as a CDE resource and submit the CDE Job
     image-20260109-154338.png
     
7. Push Spark Job Artifacts to Talend Cloud
   • After successful execution of the CDE job from Talend Studio App on user workstation next steps are to push the job artifacts  to Talend Cloud for production orchestration
     • Commit job definition back to Gitimage-20260109-155106.png
     • Publish Talend job artifact to Talend Cloudimage-20260109-155226.png

        image-20260109-155500.png

8. Talend Cloud  Job Management 
   • Once Job Artifacts are published to Talend Cloud we can start to design Talend Cloud orchestrationimage-20260109-163742 (1).png
   • Begin Task Orchestration definitions on Talend Cloud UI .:warning: Gotcha: It is expected that the Server hosting the Talend Remote Engine will have network line of sight to these CDP endpoints defined in the Talend artifact .image-20260109-163838.png
   • Task must be associated with the Talend Remote Engine defined in Step 3image-20260109-164013.png
   • Schedule the job according to requirementsimage-20260109-164112.png
9. Inspect Job Run History
   • Talend Cloud provides a dashboard of job runsimage-20260109-165532.png