Reply
Cloudera Employee
Posts: 31
Registered: ‎08-08-2013

Re: Director pipeline SUSPENDED UPDATE_FAILED

Hi, can you please provide details on what steps you took and the problem you're seeing? Sometimes errors look similar, but the details matter. We will try to help you from that point onward. Modifying the database directly is not recommended.

New Contributor
Posts: 4
Registered: ‎03-24-2016

Re: Director pipeline SUSPENDED UPDATE_FAILED

Ok, I was actually able to decript and recreat the JSON for the original cluster.

 

Cloudera director seems to be a bit broken now.

 

Still investigating.

New Contributor
Posts: 4
Registered: ‎03-24-2016

Re: Director pipeline SUSPENDED UPDATE_FAILED

Hi Vinithra, I, unfortunally, saw your message just now(when I came back to post my solution).


So, first things first. The problem was happening whenever I tried to modify the cluster(Create or delete gateway, master or worker nodes)

 

The EC2 instance was been created, was been added to the cluster on Cloudera Manager but, without report any Exception on "application.log" the cluster changed it state to UPGRADE_FAILED

 

I was able to take and understand the JSON that represents I will live it here in case anyone need(It is on CLUSTER column of CLUSTERS table):

 

Solution at the end!

 

{  
   "name":"my_cluster_name",
   "instances":[  
      {  
         "ipAddress":"internal-ip",
         "properties":{  
            "securityGroups":"security-gruop-name",
            "sourceDestCheck":"true",
            "instanceId":"intance-id",
            "privateDnsName":"my-internal-dns",
            "architecture":"x86_64",
            "rootDeviceType":"ebs",
            "subnetId":"my_subnet",
            "virtualizationType":"hvm",
            "privateIpAddress":"internal-ip",
            "rootDeviceName":"/dev/sda1",
            "spot":"false",
            "instanceType":"r3.xlarge",
            "imageId":"my_ami",
            "keyName":"keyfilename-without pem",
            "launchTime":"Mon Apr 18 17:17:57 UTC 2016",
            "vpcId":"vpc",
            "hypervisor":"xen",
            "availabilityZone":"us-east-1b",
            "publicIpAddress":"public_ip",
            "publicDnsName":"your_public_dns",
            "placementGroupName":"",
            "ebsOptimized":"false"
         },
         "virtualInstance":{  
            "id":"the big number in instance name",
            "template":{  
               "name":"r3xl",
               "type":"r3.xlarge",
               "image":"your ami",
               "bootstrapScript":"",
               "config":{  
                  "instanceNamePrefix":"director",
                  "rootVolumeSizeGB":"50",
                  "rootVolumeType":"gp2",
                  "subnetId":"my_sub",
                  "securityGroupsIds":"my_sg",
                  "useSpotInstances":"false"
               },
               "tags":{  
                  "Environment":"meh",
                  "Owner":"meh",
                  "Application":"meh",
                  "Cost Center":"meh",
                  "Cluster":"meh"
               },
               "normalizeInstance":true,
               "sshUsername":null
            }
         },
         "capabilities":{  
            "operatingSystemType":"REDHAT_COMPATIBLE",
            "operatingSystemVersion":"REDHAT_COMPATIBLE_7",
            "virtualizationType":"HARDWARE_ASSISTED",
            "javaVersion":null,
            "javaVendor":null,
            "pythonVersion":"2.7.5",
            "packageManager":"YUM",
            "passwordlessSudoEnabled":true,
            "selinuxEnabled":true,
            "iptablesEnabled":false,
            "dnsConfigured":true,
            "clouderaManagerAgentInstalled":false
         },
         "health":{  
            "status":"NOT_AVAILABLE",
            "lastReported":1461002554769
         }
      },
      {  
         ...
      }
   ],
   "services":[  
      {  
         "serviceName":"HDFS",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/x",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/x/instances"
      },
      {  
         "serviceName":"HIVE",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/Y",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/Y/instances"
      },
      {  
         "serviceName":"HUE",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/Z",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/Z/instances"
      },
      {  
         "serviceName":"OOZIE",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/A",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/A/instances"
      },
      {  
         "serviceName":"SPARK_ON_YARN",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/B",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/B/instances"
      },
      {  
         "serviceName":"SQOOP",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/C",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/C/instances"
      },
      {  
         "serviceName":"YARN",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/D",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/D/instances"
      },
      {  
         "serviceName":"ZOOKEEPER",
         "url":"http://clouderamanager.address:7180/cmf/serviceRedirect/E",
         "roleInstancesUrl":"http://clouderamanager.address:7180/cmf/serviceRedirect/E/instances"
      }
   ],
   "createdExternalDatabases":[  
      {  
         "type":"MYSQL",
         "host":"ip",
         "port":3306,
         "name":"my_db_1",
         "user":"my_user",
         "password":"my_pass"
      },
      {  
         "type":"MYSQL",
         "host":"ip",
         "port":3306,
         "name":"my_db_name",
         "user":"my_user",
         "password":"my_pass"
      }
   ],
   "url":"http://clouderamanager.address:7180/cmf/clusterRedirect/my_cluster_name",
   "instancesUrl":"http://clouderamanager.address:7180/cmf/clusterRedirect/my_cluster_name/hosts",
   "health":{  
      "status":"NOT_AVAILABLE",
      "lastReported":1461000922000
   }
}

 

SOLUTION:

 

So, after some dozens of coffee cups(And a bit of decompilation, sorry Cloudera guys) I was able to find the solution:

 

1) You have to enable CRaSH:
     On /etc/cloudera-director-server/application.properties, enable the following lines: 
          shell.ssh.enabled: true
          shell.ssh.port: 2000

 

2) Restart director

 

3) Inside CD server, use ssh to connect to CRaSH, as follow: ssh -p 2000 -l your_user localhost (The user is the same you use in the web interface, unless you change it in application.properties)

 

4) Insert your password

 

5) Execute(without quotes, changing the values to match you cluster) "clusters reconcile <Environment name> <Deployment name> <Cluster name> true"

 

6) Enjoy your day.

 

 

 

Highlighted
New Contributor
Posts: 1
Registered: ‎05-06-2016

Re: Director pipeline SUSPENDED UPDATE_FAILED

Hi,

 

Thank you so much for the hardwork getting this fixed !

 

You made my day :)

 

Cheers,

 

Michael

Announcements