Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cloudera manager 5.4.0 installing kafka parcel fails

cloudera manager 5.4.0 installing kafka parcel fails

New Contributor

I have tried many times to download , distribute and activate Apach Kafka 0.8.2.0-1.kafka1.3.1.p0.9 on CDH 5.4.0 VM but always get an activation error saying " Dependency not satisfied for release KAFKA(0.8.2.0-1.kafka1.3.1.p0.9): CDH (at least 5.2).". I am certain my version is higher than that.Below is the outout of Hadoop -version command. 


Hadoop 2.6.0-cdh5.4.0
Subversion http://github.com/cloudera/hadoop -r c788a14a5de9ecd968d1e2666e8765c5f018c271
Compiled by jenkins on 2015-04-21T19:18Z
Compiled with protoc 2.5.0
From source with checksum cd78f139c66c13ab5cee96e15a629025
This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.4.0.jar

 

I have also looked at the solutions presented in

 

http://community.cloudera.com/t5/Apache-Hadoop-Concepts-and/cloudera-manager-5-4-0-installing-kafka-...

 

This does not work since this error happens after I move cloudera to a parcel based Install . After I move to a  parcel based install I try to download, distribute and active the parcel as has been recommended by Sean. There seems to be a bug with Cloudera Manager . Can you please check and confirm this behavior ? There seem to multiple users reporting the same bug.

4 REPLIES 4
Highlighted

Re: cloudera manager 5.4.0 installing kafka parcel fails

Master Collaborator
The problem is that the VM uses Linux packages (so that using CM is
optional), whereas Kafka is only available as a parcel, and you can't mix
Linux packages and parcels. The solution is to migrate to using parcels for
CDH, and THEN the dependencies for the Kafka parcel will be satisfied.

In a few hours I'll be able to share a script with you that should automate
the process of migrating to a parcel, which might make it a lot easier.
I'll post back here then.


Re: cloudera manager 5.4.0 installing kafka parcel fails

Master Collaborator

So the following is a script that'll go through the process of disabling the current CDH services and removing the Linux packages, and getting them to use a parcel instead so that the KAFKA parcel's dependencies will be satisfied. Understand this hasn't been that rigorously tested yet (which is why it's not in the VM yet) and that not all of the Cloudera Labs parcels have been tested rigorously in the VM either (Kafka worked well for me, fwiw, but I've had trouble with Phoenix, for instance).

 

You should make sure that the parcel version in the script matches a version for which the repository is configured in CM. For instance, the script is set up for 5.4.2, and the repository is here: http://archive.cloudera.com/cdh5/parcels/5.4.2/. If you use a different repository, update the script with the new parcel version. The documentation for this process when done manually is here: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_migrating_package...

 

Put this in a script (e.g. ~/parcels.sh) and run it (e.g. bash ~/parcels.sh) and hopefully it can make this easier for you...

 

#! /usr/bin/env bash

CM_API=/home/cloudera/cm_api.py
CM_CLUSTER_NAME='Cloudera QuickStart'
PARCEL_VERSION='5.4.2-1.cdh5.4.2.p0.2'

function terminate() {
    if [ "${PAUSE}" == 'true' ]; then
        echo ""
        read -p "Press [Enter] to exit..."
    fi
    exit ${1}
}

function ensure_user_is_root() {
    if [[ "$EUID" -ne "0" ]]; then
        echo "You must run this script as root. Try 'sudo ${0} ${@}'."
        terminate 1
    fi
}

function parse_arguments() {
    while [[ $# > 0 ]]; do
        argument="$1"
        case $argument in
            --version)
                export PARCEL_VERSION="$2"
                shift
            ;;
            --pause)
                export PAUSE='true'
            ;;
            *)
                echo "Unknown option: ${argument}"
                terminate 1
            ;;
        esac
        shift
    done
}

function log() {
    echo "[QuickStart] ${1}"
}

function wait_for_file_to_appear() {
    while [ ! -e $1 ]; do
        sleep 1
    done
}

function wait_for_file_to_disappear() {
    while [ -e $1 ]; do
        sleep 1
    done
}

ensure_user_is_root
parse_arguments ${@}

if [ ! -e /var/lib/cloudera-quickstart/.cloudera-manager ]; then
    cat <<EOF

To migrate from Linux packages to parcels, you must be using Cloudera Manager
to manage your cluster. You can migrate to Cloudera Manager using one of the
following options:

To use Cloudera Express (free), run:

    sudo /home/cloudera/cloudera-manager --express

This requires at least 8 GB of RAM and at least 2 virtual CPUs.

To begin a 60-day trial of Cloudera Enterprise with advanced management
features, run:

    sudo /home/cloudera/cloudera-manager --enterprise

This requires at least 10 GB or RAM and at least 2 virtual CPUs.

Be aware that after rebooting, it may take several minutes before Cloudera
Manager has started all of the services it manages and is ready to accept
connections from clients.

EOF
    terminate 1
fi

if ! service cloudera-scm-server status 2>&1 >/dev/null; then
    cat <<EOF

The Cloudera Manager Server is not running on this machine. It must be running
in order to download and activate parcels. You can confirm the status of the
service by running:

    sudo service cloudera-scm-server status

You can attempt to restart it by running:

    sudo service cloudera-scm-server restart

If you continue to have problems you can find the log files in:

    /var/log/cloudera-scm-server

EOF
    terminate 2
fi

PARCEL_API=clusters/${CM_CLUSTER_NAME}/parcels/products/CDH/versions/${PARCEL_VERSION}/commands
DOWNLOAD_CREATES=/opt/cloudera/parcel-repo/CDH-${PARCEL_VERSION}-el6.parcel.sha
DISTRIBUTE_CREATES=/opt/cloudera/parcel-cache/CDH-${PARCEL_VERSION}-el6.parcel
ACTIVATE_CREATES=/opt/cloudera/parcels/CDH

log 'Downloading parcel...'
if [ ! -e "${DOWNLOAD_CREATES}" ]; then
    ${CM_API} --method POST "${PARCEL_API}/startDownload" >/dev/null
fi
wait_for_file_to_appear ${DOWNLOAD_CREATES}
sleep 60

log 'Distributing / decompressing parcel...'
if [ ! -e "${DISTRIBUTE_CREATES}" ]; then
    ${CM_API} --method POST "${PARCEL_API}/startDistribution" >/dev/null
fi
wait_for_file_to_appear ${DISTRIBUTE_CREATES}
sleep 60
wait_for_file_to_disappear /opt/cloudera/parcel-cache/tmp*

log 'Activating parcel...'
if [ ! -e "${ACTIVATES_CREATES}" ]; then
    ${CM_API} --method POST "${PARCEL_API}/activate" >/dev/null
fi
wait_for_file_to_appear ${ACTIVATES_CREATES}

touch /var/lib/cloudera-quickstart/.parcels

CONFIG_JS=/var/lib/cloudera-quickstart/tutorial/js/config.js
sed -i -e 's/^parcels\s\{0,\}=.*false/parcels = true;/' ${CONFIG_JS}

log "Stopping currently running services..."
${CM_API} --method POST "clusters/${CM_CLUSTER_NAME}/commands/stop" >/dev/null

log 'Redeploying client configuration...'
${CM_API} --method POST "clusters/${CM_CLUSTER_NAME}/commands/deployClientConfig" >/dev/null

log 'Removing packages...'
yum remove -y `yumdb search from_repo cloudera-cdh5 | grep -v ^$ | grep -v from_repo | tail -n +2` 2>&1 >/dev/null

log 'Restarting Cloudera Manager agent...'
service cloudera-scm-agent restart

cat <<EOF
________________________________________________________________________________

Success! Your cluster is now set up to use a parcel instead of Linux packages.
You can now restart the services you wish to use.

EOF

terminate 0

Re: cloudera manager 5.4.0 installing kafka parcel fails

New Contributor

Thanks for the shell script .. I am trying it as I type . However I am not sure I understand you 100% . I  followed tha manual steps that have been highted below for migrating from packages to parcels. However I did that ONLY for Kafka and not for the other parcels available. That gave me the same error. 

 

Are you saying that I download ALL the parcels available ( I see , CDH 5(5.3) , ACCUMULO etc) and install everything( or at least CDH parcel) to satisfy the parcel requirements first, before I download the Kafka parcel ?  

 

 

Download, Distribute, and Activate Parcels

  1. In the Cloudera Manager Admin Console, click the Parcels indicator in the top navigation bar ( or ).
  2. Click Download for the version that matches the CDH or service version of the currently installed packages. If the parcel you want is not shown here—for example, if you want to use a version of CDH that is not the most current version—you can add parcel repositories through the Parcel Configuration Settings page:If your Cloudera Manager Server does not have Internet access, you can obtain the required parcel file(s) and put them into a repository. See Creating and Using a Remote Parcel Repository for Cloudera Manager for more details.
  3. When the download has completed, click Distribute for the version you downloaded.
  4. When the parcel has been distributed and unpacked, the button will change to say Activate.
  5. Click Activate.

Re: cloudera manager 5.4.0 installing kafka parcel fails

Master Collaborator
No, you should only need the CDH parcel.