Community Articles
Find and share helpful community-sourced technical articles
Cloudera Employee

Custom parcels and CSDs allow partners and clients to bring their libraries and packages to CDP.  Traditionally, customers would use recipes to install parcels and CSDs or manually copy them to appropriate nodes. But, with CDP Public Cloud, you need a modular approach to do this. To address that need, Cloudera has developed the capability of specifying the parcel and CSD information as part of the cluster template (blueprint). With this approach, the parcel and CSD will get downloaded, distributed and activated as part of the cluster build process.

 

In the example below, FLINK 1.12 should be installed by default, but a different version of the FLINK parcel and CSD are passed for the Cloudera Manager to download, distribute and activate. FLINK service is added to the cluster during the cluster import.

Steps to construct a custom cluster blueprint:

Get the Image Catalog ID

  1. Get the AMI-ID of the DataLake. Log on the CDP > DataLake > Image Details (for a new/recently created environments). Create a temporary Datahub to get the AMI-ID, if the DL was created a while ago.nij23_1-1608071291234.png
  2. Open the Image Catalog json
  3. Search for AMI-ID in the Image Catalog jsonnij23_2-1608071291255.png
  4. Note the UUID. Make sure the image is in the right region and the cloud provider (AWS/Azure).

Get the Cluster definition/Blueprint

  1. Log on to CDP > Environments > Cluster Definitions > Search for the appropriate version (depends on the DL). For this example, 7.2.2 - Streaming Analytics Light Duty for AWS
    nij23_3-1608071291289.png
  2.  Click on the definition.
  3. Copy paste the raw json to an editor.

Modify the Cluster definition

Add environment name: This is where the cluster will be deployed.

 

"name": “DEFINITION_NAME”,
"environmentName": “CDP_ENV_NAME”,
"cluster": {
"databases": [],
"blueprintName": “7.2.2 - Streaming Analytics Light Duty with Apache Flink"
                                   "validateBlueprint": false,

 

Add CM base url, products: name, version and Parcels.

Also, open Image catalog for the appropriate CM, CDH, Products (FLINK) versions and parcel location.
CM:

 

        "cm": {

                 "enableAutoTls": true,
                   "repository": {
                   "baseUrl": "http://cloudera-build-us-west-1.vpc.cloudera.com/s3/build/4763198/cdh/7.x/parcels/",
                    "version": “7.2.2”

                },

 

Products: CDH, FLINK

 

        "products": [
                {
          "name": "CDH",
           "version": "7.2.2-1.cdh7.2.2.p1.6575992",
           "parcel": "https://archive.cloudera.com/p/cdp-public/7.2.2.1/parcels/"
          },
          {
           "name": "FLINK",
           "version": "1.10.0-csa1.2.1.0-cdh7.2.1.0-240-4844562",
           "parcel": "http://54.153.84.214/parcels/",
            "csd": [ "http://54.153.84.214/parcels/FLINK-1.10.0-csa1.2.1.0-cdh7.2.1.0-240-4844562.jar”

             ]}}

 

Image Catalog: Add image catalog at the end.
Image ID:

 

         "image": {
         "catalog": "cdp-default",
         "id": "eb1d458a-bf82-434b-bc2a-ac06b8cd6c0a"

 

Note: http://54.153.84.214/parcels is a temporary parcels location, this should point to the location of the parcel and CSDs. A finished custom blueprint should look like below:

 

{
  "instanceGroups": [
    {
      "nodeCount": 1,
      "name": "manager",
      "type": "GATEWAY",
      "recoveryMode": "MANUAL",
      "template": {
        "aws": {
          "encryption": {
            "type": "NONE",
            "key": null
          }
        },
        "instanceType": "m5.2xlarge",
        "rootVolume": {
          "size": 50
        },
        "attachedVolumes": [
          {
            "size": 100,
            "count": 1,
            "type": "standard"
          }
        ],
        "cloudPlatform": "AWS"
      },
      "recipeNames": []
    },
    {
      "nodeCount": 2,
      "name": "master",
      "type": "CORE",
      "recoveryMode": "MANUAL",
      "template": {
        "aws": {
          "encryption": {
            "type": "NONE",
            "key": null
          }
        },
        "instanceType": "m5.2xlarge",
        "rootVolume": {
          "size": 50
        },
        "attachedVolumes": [
          {
            "size": 100,
            "count": 1,
            "type": "standard"
          }
        ],
        "cloudPlatform": "AWS"
      },
      "recipeNames": []
    },
    {
      "nodeCount": 3,
      "name": "worker",
      "type": "CORE",
      "recoveryMode": "MANUAL",
      "template": {
        "aws": {
          "encryption": {
            "type": "NONE",
            "key": null
          }
        },
        "instanceType": "m5.2xlarge",
        "rootVolume": {
          "size": 50
        },
        "attachedVolumes": [
          {
            "size": 100,
            "count": 1,
            "type": "standard"
          }
        ],
        "cloudPlatform": "AWS"
      },
      "recipeNames": []
    }
  ],
  "name": "pse-flink-cust",
  "environmentName": "pse-722-cdp-env",
  "cluster": {
    "databases": [],
    "exposedServices": [
      "ALL"
    ],
    "blueprintName": "7.2.2 - Streaming Analytics Light Duty with Apache Flink",
    "validateBlueprint": false,
    "cm": {
        "enableAutoTls": true,
        "repository": {
          "baseUrl": "http://cloudera-build-us-west-1.vpc.cloudera.com/s3/build/4763198/cdh/7.x/parcels/",
          "version": "7.2.2"
        },
        "products": [
          {
            "name": "CDH",
            "version": "7.2.2-1.cdh7.2.2.p1.6575992",
            "parcel": "https://archive.cloudera.com/p/cdp-public/7.2.2.1/parcels/"
          },
          {
            "name": "FLINK",
            "version": "1.10.0-csa1.2.1.0-cdh7.2.1.0-240-4844562",
            "parcel": "http://54.153.84.214/parcels/",
            "csd": [
              "http://54.153.84.214/parcels/FLINK-1.10.0-csa1.2.1.0-cdh7.2.1.0-240-4844562.jar"
            ]
          }
        ]
    },
    "Xcm": {
        "enableAutoTls": true,
        "repository": {
          "baseUrl": "https://archive.cloudera.com/p/cm-public/7.2.2-6458542/redhat7/yum/",
          "version": "7.2.2"
        },
        "products": [
          {
            "name": "CDH",
              "version": "7.2.2-1.cdh7.2.2.p1.6575992",
              "parcel": "https://archive.cloudera.com/p/cdp-public/7.2.2.1/parcels/"
          }
        ]
    }
  },
  "image": {
      "catalog": "cdp-default",
      "id": "eb1d458a-bf82-434b-bc2a-ac06b8cd6c0a"
  },
  "inputs": {}
}

 

CDP CLI Provision to Cluster

 

$ cdp datahub --profile pse-demo  create-aws-cluster --request-template file:///Users/njlamsal/flink-pse-demo.json --cluster-name protegrity-flink-1215

 

 

profile: Configure profile using cdpcli.

request-template: Template created above. Note to use file://

cluster-name: Name of the cluster 

 

ERROR:

An error occurred: {"message":"Image or image catalog settings are incorrect: CloudbreakImageCatalogException: Inconsistent request, base images are disabled but custom repo information is submitted!"} (Status Code: 400; Error Code: INVALID_ARGUMENT; Service: datahub; Operation: createAWSCluster; Request ID: a7feb7a9-5547-43b2-8cfd-9543067677cc;)

 

Check DL and DH image IDs. If they are different, take the DH AMI-ID and find the catalog ID (UUID).

683 Views
Comments
New Contributor

Hi. Thanks for this article. Which version of CDP Platform is this feature available in?

Cloudera Employee

@FRG96  CDP runtime 7.2.2

New Contributor

@nij23, Thanks. I created a cluster definition on CDP 7.2.6 (latest available version) following the above steps and was able to create a Data Hub with my custom CSD and parcels distributed, activated. I used the cdp cli to successfully create the datahub cluster. 

I wanted to save this definition to use from the management console too, so I used "cdp datahub create-cluster-definition" command. But when I viewed the cluster definition JSON on Management console (Shared resources section), the "enableAutoTls" property was set to false even though I had set it to true explicitly in my JSON file.
When I tried creating a cluster with this definition, I wasn't able to access the CM-UI or other service UIs after the cluster was created and in running state. Just wanted to point this out, that I am able to create cluster via cdpcli but via management console UI it was failing. Any suggestions on this?

FRG96_0-1610635732865.png


Additionally, Can you point me to any documentation for the model/schema of the CDP Cluster Definition?

Cloudera Employee

@FRG96 You will need to adjust user permission to access the UI. It is blocked by default. 

Review account-level roles and resource roles here: https://docs.cloudera.com/management-console/cloud/user-management/topics/mc-understanding-roles-res...

 

After adding the user to an appropriate admin role, sync it to freeIPA. More on that here: https://docs.cloudera.com/management-console/cloud/user-management/topics/mc-sync.html

 

New Contributor

@nij23  I have the correct account and resource role in my environment. I restarted the cluster created from the custom cluster definition and when I opened the CM-UI I got this message:

Screenshot 2021-01-15 105437.png

Above Text for reference:

HTTP ERROR 500 javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException: Custom client SSL context, if set, must not be null.
URI: /test5/cdp-proxy/cmf/home/
STATUS: 500
MESSAGE: javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException: Custom client SSL context, if set, must not be null.
SERVLET: cdp-proxy-knox-gateway-servlet
CAUSED BY: javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException: Custom client SSL context, if set, must not be null.
CAUSED BY: javax.servlet.ServletException: java.lang.NullPointerException: Custom client SSL context, if set, must not be null.
CAUSED BY: java.lang.NullPointerException: Custom client SSL context, if set, must not be null.

 

 

When I try to open the CM-UI the second time and in any subsequent tries, I get the 403 Forbidden message. I see the 500 Servlet Exception only the first time when I try to access CM-UI.

 

Could it be because the "enableAutoTls" property was automatically set to false when I registered the custom cluster definition even though I had explicitly set it to true in my cluster definition JSON?

Don't have an account?
Version history
Revision #:
42 of 42
Last update:
‎12-20-2020 10:39 PM
Updated by:
 
Contributors
Top Kudoed Authors