Reply
New Contributor
Posts: 4
Registered: ‎08-18-2016
Accepted Solution

Sreamsets setup on CDH 5.9.1

Hi,

 

We have installed and configure CDH 5.9.1 using couldera director. We are trying to deply streamsets data collector 2.2  as cluster service using percel. I could download, distrubute and activate parcel from Manager. However I tried to add service but I could not find any option to choose  streamsets data collector.  I tried streamsets version 2.1 as well but no luck.

 

Your help will be appreciated. 

 

Thanks

SP

Cloudera Employee
Posts: 6
Registered: ‎08-24-2015

Re: Sreamsets setup on CDH 5.9.1

I'm not sure if there's official support with Director, but it can be done using the conf file with the help of a bootstrap script:

 

Specify a bootstrap script (only for the CM instance) to download and place the csd jar in the appropriate csd directory.

 

instances {

  cminstance {
    type: m4.xlarge
    image: ami-ac5f2fcc

    tags {
      owner: ${?USER}
    }

    bootstrapScript: """#!/bin/sh
yum -y install wget
wget https://archives.streamsets.com/datacollector/2.3.0.0/csd/STREAMSETS-2.3.0.0.jar
mkdir -p /opt/cloudera/csd
mv STREAMSETS-2.3.0.0.jar /opt/cloudera/csd/
"""
  }

  ...
}

You also have to specify the Product, Service, Role name along with the Parcel Repository URL in the conf file. The following worked for me (I went through a manual install to get these values):

 

cluster {

  # add the streamset data collector product
  products {
    CDH: 5
    STREAMSETS_DATACOLLECTOR: 2.3
  }


  # add the streamset parcel repository
  parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.9/",
                       "https://archives.streamsets.com/datacollector/latest/parcel/"]

  # add the service
  services: [HDFS, YARN, STREAMSETS,...]

  ...

  workers {
  	...

    # add the data collector role to the streamset service
    roles {
      HDFS: [DATANODE]
      YARN: [NODEMANAGER]
      STREAMSETS: [DATACOLLECTOR]
      ...
    }
  }
}

 

Highlighted
New Contributor
Posts: 4
Registered: ‎08-18-2016

Re: Sreamsets setup on CDH 5.9.1

Many Thanks AARMAN !! It worked.
Announcements