Talend - CDP Integration Reference

Introduction

This page provides a High-Level understanding of how an enterprise might leverage Qlik/Talend integrations to run big data Spark workloads on Cloudera Public Cloud CDE.

A standard setup for integrations could be represented as indicated in the diagram that follows (Fig 1):CDP-PC Talend Cloud Integration Reference

Primary Talend components required to showcase integration are as follows:

• A Trial or Enterprise account on Qlik/Talend Cloud

• Installation of the Talend Studio Application on the user workstation with network line of sight to CDP APIs and endpoints

• Installation of Talend Remote Engine in or alongside the CDP Environment with network line of sight to CDP APIs and endpoints.

Download and install instructions for Studio and Remote Engine are available from the Qlik/Talend Cloud UI available after trial activation (Fig 2)

Download resources for Talend Cloud Integration setup

Instructions for Talend - Cloudera Data Engineering Spark  integration

The following setup instructions are made available for an SE that wants to investigate the Talend-CDP integration. In the following example we have focused on an execution and orchestration of a simple Talend Spark application on a CDE dataservice running within the CDP environment.

1. On Talend Cloud UI set up the demo Project:
   • Set up your Talend project referencing the chosen Git URL (Note Git credentials are not required to be persisted on Talend Cloud)
     
2. On the new Project in Talend UI assign yourself (your Talend Cloud Identity) as a project collaborator:
   • Set up privileges on the project as a project collaborator
3. Install the Talend Remote Engine on a Linux instance in the CDP env
   • There is no real restriction on where we can install the Talend Remote Engine. For the simple use case showcased here, I have chosen a COD edge node to host the Talend Remote Engine. Instructions for installation are available here Remote Engine on Linux
   • 
4. Install the Talend Studio Application on your workstation

   • Instructions for Talend Studio setup are available here Talend Studio Install

5. Login to open the Talend Studio application
   • Once Talend Studio is installed on the user workstation, open the Talend Studio application. Login is to be performed with Talend cloud credentials . Configuration of login must leverage the appropriate Web-app URL for your allocated Talend cloud tenant

   • After logging in Talend Studio will present you with a list of project definitions (Defined on Talend Cloud) that you want to interact with

   • Choose the appropriate Talend Cloud project.  The Talend Studio application will launch with the Talend Job Designer Canvas
6. Design the Talend Jobs to be executed on the CDP ecosystem
   • Developers can now start the process of designing Talend Jobs to be executed on the CDP ecosystem. In this documentation and defined in the referenced Git repo a simple spark job is predefined in our Talend cloud project
   • Open the CDE job cde1 0.1 from the Repository browser on the Talend Studio UI
   • Configure the Spark Configuration of cde1 0.1 as appropriate for your target CDE endpoints.  :warning: Gotcha: Note the quotation in the image above .  Required params are as follows :

     • CDE API endpoint

     • CDE Service API endpoint (used for token retrieval)

     • Workload username & password

   • Once Spark configuration params have been populated you can test and run your Talend job directly from Talend Studio. 
   • :warning: Gotcha: It is expected that the workstation hosting Talend Studio application will have network line of sight to these CDP endpoints
   • Talend  Spark Jobs will populate CDE  resources with Talend Runtime Libs 
   • Talend Studio leverages CDE Java SDK to upload the Job Artifacts as a CDE resource and submit the CDE Job
     
     
7. Push Spark Job Artifacts to Talend Cloud
   • After successful execution of the CDE job from Talend Studio App on user workstation next steps are to push the job artifacts  to Talend Cloud for production orchestration
     • Commit job definition back to Git

        

     • Publish Talend job artifact to Talend Cloud

        

8. Talend Cloud  Job Management 
   • Once Job Artifacts are published to Talend Cloud we can start to design Talend Cloud orchestration
   • Begin Task Orchestration definitions on Talend Cloud UI .:warning: Gotcha: It is expected that the Server hosting the Talend Remote Engine will have network line of sight to these CDP endpoints defined in the Talend artifact .
   • Task must be associated with the Talend Remote Engine defined in Step 3
   • Schedule the job according to requirements
9. Inspect Job Run History
   • Talend Cloud provides a dashboard of job runs