Member since
09-23-2014
15
Posts
14
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 12875 | 05-13-2015 04:12 AM | |
| 17841 | 05-11-2015 06:33 AM | |
| 1337 | 01-06-2015 12:09 PM |
05-11-2017
12:48 PM
What is the bug? I don't see it here when searching under Spark. https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed_in_510.html#fixed_issues_5101
... View more
10-18-2016
12:33 PM
Greetings, This describes the issue you are having. It is a combination of beeline and Sentry security causing this. http://wp.huangshiyang.com/add-jar-command-throws-insufficient-privileges-to-execute-add-exception-even-for-admin-role https://issues.apache.org/jira/browse/SENTRY-147 The work around we have found was to use the deprecated hive command line (not beeline) which still appears to work with Kerberos and CDH 5.8.2 (latest released). The "supported" solution is in the first link I put up, basically all users must give the JAR files for Hive UDFs to an admin that can put them in a location that is accessible by all users and whitelisted from Sentry security.
... View more
01-28-2016
08:00 AM
This worked! Thank you !!
... View more
01-26-2016
07:30 AM
Here is the same information going directly through the Cloudera Manager API (instead of director). [ec2-user@ip-10-170-0-201 ~]$ curl --user admin:<redacted> http://10.170.2.146:7180/api/v7/clusters/Cluster01/services { "items" : [ { "name" : "CD-HIVE-QAFjCOQn", "type" : "HIVE", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HIVE-QAFjCOQn", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "HIVE_HIVEMETASTORES_HEALTHY", "summary" : "GOOD" }, { "name" : "HIVE_HIVESERVER2S_HEALTHY", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "HIVE-1" }, { "name" : "CD-HUE-UBSUTqcp", "type" : "HUE", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HUE-UBSUTqcp", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "HUE_HUE_SERVERS_HEALTHY", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "HUE-1" }, { "name" : "CD-OOZIE-wMmgwoTc", "type" : "OOZIE", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-OOZIE-wMmgwoTc", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "OOZIE_OOZIE_SERVERS_HEALTHY", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "OOZIE-1" }, { "name" : "CD-SPARK_ON_YARN-AJfBtbso", "type" : "SPARK_ON_YARN", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SPARK_ON_YARN-AJfBtbso", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "SPARK_ON_YARN-1" }, { "name" : "CD-SQOOP-UrPslgqE", "type" : "SQOOP", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SQOOP-UrPslgqE", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "SQOOP_SQOOP_SERVER_HEALTH", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "SQOOP-1" }, { "name" : "CD-YARN-Cahmhgyy", "type" : "YARN", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-YARN-Cahmhgyy", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "YARN_JOBHISTORY_HEALTH", "summary" : "GOOD" }, { "name" : "YARN_NODE_MANAGERS_HEALTHY", "summary" : "GOOD" }, { "name" : "YARN_RESOURCEMANAGERS_HEALTH", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "YARN-1" }, { "name" : "CD-HDFS-Fxvxaehf", "type" : "HDFS", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HDFS-Fxvxaehf", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "HDFS_BLOCKS_WITH_CORRUPT_REPLICAS", "summary" : "GOOD" }, { "name" : "HDFS_CANARY_HEALTH", "summary" : "GOOD" }, { "name" : "HDFS_DATA_NODES_HEALTHY", "summary" : "GOOD" }, { "name" : "HDFS_FREE_SPACE_REMAINING", "summary" : "GOOD" }, { "name" : "HDFS_HA_NAMENODE_HEALTH", "summary" : "GOOD" }, { "name" : "HDFS_MISSING_BLOCKS", "summary" : "GOOD" }, { "name" : "HDFS_UNDER_REPLICATED_BLOCKS", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "HDFS-1" }, { "name" : "CD-ZOOKEEPER-OwinYWHV", "type" : "ZOOKEEPER", "clusterRef" : { "clusterName" : "Cluster01" }, "serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-ZOOKEEPER-OwinYWHV", "serviceState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "ZOOKEEPER_CANARY_HEALTH", "summary" : "GOOD" }, { "name" : "ZOOKEEPER_SERVERS_HEALTHY", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "clientConfigStalenessStatus" : "FRESH", "maintenanceMode" : false, "maintenanceOwners" : [ ], "displayName" : "ZOOKEEPER-1" } ] }[ec2-user@ip-10-170-0-201 ~]$
... View more
01-26-2016
07:25 AM
I have run the API command you suggested that dumps the cluster status. This was the result. [ec2-user@ip-10-170-0-201 ~]$ curl --user admin:<redacted> http://10.170.0.201:7189/api/v4/environments/New%20Platform/deployments/CM01/clusters/ [ "Cluster01" ][ec2-user@ip-10-170-0-201 ~]$ curl --user admin:CDH_g0liv3 http://10.170.0.201:7189/api/v4/environments/New%20Platform/deployments/CM01/clusters/Cluster01 { "name" : "Cluster01", "instances" : [ { "ipAddress" : "10.170.12.140", "properties" : { "subnetId" : "subnet-1e1f6f47", "ebsOptimized" : "false", "virtualizationType" : "hvm", "imageId" : "ami-414b7271", "instanceType" : "m4.large", "keyName" : "NewPlatformRoot", "publicDnsName" : "", "sourceDestCheck" : "true", "availabilityZone" : "us-west-2c", "privateIpAddress" : "10.170.12.140", "rootDeviceType" : "ebs", "launchTime" : "Fri Jan 22 09:59:22 EST 2016", "instanceId" : "i-709e08aa", "placementGroupName" : "", "hypervisor" : "xen", "spot" : "false", "vpcId" : "vpc-1b47767e", "rootDeviceName" : "/dev/sda1", "privateDnsName" : "ip-10-170-12-140.us-west-2.compute.internal", "securityGroups" : "launch-wizard-1(sg-fe2aca99)", "architecture" : "x86_64" }, "virtualInstance" : { "id" : "cce0aec6-3eaf-4126-8ccc-63b1e4af4d9b", "template" : { "name" : "ClouderaManager", "type" : "m4.large", "image" : "ami-414b7271", "bootstrapScript" : "", "config" : { "subnetId" : "subnet-1e1f6f47", "rootVolumeType" : "gp2", "instanceNamePrefix" : "director", "rootVolumeSizeGB" : "50", "useSpotInstances" : "false", "securityGroupsIds" : "sg-fe2aca99" }, "tags" : { "Name" : "NewPlatform-ClouderaManager" }, "normalizeInstance" : true } }, "capabilities" : { "operatingSystemType" : "REDHAT_COMPATIBLE", "operatingSystemVersion" : "REDHAT_COMPATIBLE_6", "virtualizationType" : "HARDWARE_ASSISTED", "pythonVersion" : "2.6.6", "packageManager" : "YUM", "passwordlessSudoEnabled" : true, "selinuxEnabled" : true, "iptablesEnabled" : false }, "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 } }, { "ipAddress" : "10.170.1.109", "properties" : { "subnetId" : "subnet-1e1f6f47", "ebsOptimized" : "false", "virtualizationType" : "hvm", "imageId" : "ami-414b7271", "instanceType" : "m4.large", "keyName" : "NewPlatformRoot", "publicDnsName" : "", "sourceDestCheck" : "true", "availabilityZone" : "us-west-2c", "privateIpAddress" : "10.170.1.109", "rootDeviceType" : "ebs", "launchTime" : "Fri Jan 22 09:59:22 EST 2016", "instanceId" : "i-7f9e08a5", "placementGroupName" : "", "hypervisor" : "xen", "spot" : "false", "vpcId" : "vpc-1b47767e", "rootDeviceName" : "/dev/sda1", "privateDnsName" : "ip-10-170-1-109.us-west-2.compute.internal", "securityGroups" : "launch-wizard-1(sg-fe2aca99)", "architecture" : "x86_64" }, "virtualInstance" : { "id" : "ccdc7ab5-f6ef-4f46-be65-1f4664b42433", "template" : { "name" : "ClouderaManager", "type" : "m4.large", "image" : "ami-414b7271", "bootstrapScript" : "", "config" : { "subnetId" : "subnet-1e1f6f47", "rootVolumeType" : "gp2", "instanceNamePrefix" : "director", "rootVolumeSizeGB" : "50", "useSpotInstances" : "false", "securityGroupsIds" : "sg-fe2aca99" }, "tags" : { "Name" : "NewPlatform-ClouderaManager" }, "normalizeInstance" : true } }, "capabilities" : { "operatingSystemType" : "REDHAT_COMPATIBLE", "operatingSystemVersion" : "REDHAT_COMPATIBLE_6", "virtualizationType" : "HARDWARE_ASSISTED", "pythonVersion" : "2.6.6", "packageManager" : "YUM", "passwordlessSudoEnabled" : true, "selinuxEnabled" : true, "iptablesEnabled" : false }, "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 } }, { "ipAddress" : "10.170.1.108", "properties" : { "subnetId" : "subnet-1e1f6f47", "ebsOptimized" : "false", "virtualizationType" : "hvm", "imageId" : "ami-414b7271", "instanceType" : "m4.large", "keyName" : "NewPlatformRoot", "publicDnsName" : "", "sourceDestCheck" : "true", "availabilityZone" : "us-west-2c", "privateIpAddress" : "10.170.1.108", "rootDeviceType" : "ebs", "launchTime" : "Fri Jan 22 09:59:22 EST 2016", "instanceId" : "i-7c9e08a6", "placementGroupName" : "", "hypervisor" : "xen", "spot" : "false", "vpcId" : "vpc-1b47767e", "rootDeviceName" : "/dev/sda1", "privateDnsName" : "ip-10-170-1-108.us-west-2.compute.internal", "securityGroups" : "launch-wizard-1(sg-fe2aca99)", "architecture" : "x86_64" }, "virtualInstance" : { "id" : "2ff16e18-24e7-4352-92f8-b74e292c4d90", "template" : { "name" : "ClouderaManager", "type" : "m4.large", "image" : "ami-414b7271", "bootstrapScript" : "", "config" : { "subnetId" : "subnet-1e1f6f47", "rootVolumeType" : "gp2", "instanceNamePrefix" : "director", "rootVolumeSizeGB" : "50", "useSpotInstances" : "false", "securityGroupsIds" : "sg-fe2aca99" }, "tags" : { "Name" : "NewPlatform-ClouderaManager" }, "normalizeInstance" : true } }, "capabilities" : { "operatingSystemType" : "REDHAT_COMPATIBLE", "operatingSystemVersion" : "REDHAT_COMPATIBLE_6", "virtualizationType" : "HARDWARE_ASSISTED", "pythonVersion" : "2.6.6", "packageManager" : "YUM", "passwordlessSudoEnabled" : true, "selinuxEnabled" : true, "iptablesEnabled" : false }, "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 } }, { "ipAddress" : "10.170.1.110", "properties" : { "subnetId" : "subnet-1e1f6f47", "ebsOptimized" : "false", "virtualizationType" : "hvm", "imageId" : "ami-414b7271", "instanceType" : "m4.large", "keyName" : "NewPlatformRoot", "publicDnsName" : "", "sourceDestCheck" : "true", "availabilityZone" : "us-west-2c", "privateIpAddress" : "10.170.1.110", "rootDeviceType" : "ebs", "launchTime" : "Fri Jan 22 09:59:22 EST 2016", "instanceId" : "i-7d9e08a7", "placementGroupName" : "", "hypervisor" : "xen", "spot" : "false", "vpcId" : "vpc-1b47767e", "rootDeviceName" : "/dev/sda1", "privateDnsName" : "ip-10-170-1-110.us-west-2.compute.internal", "securityGroups" : "launch-wizard-1(sg-fe2aca99)", "architecture" : "x86_64" }, "virtualInstance" : { "id" : "9bdde564-c490-45ad-864b-172f3907ee0a", "template" : { "name" : "ClouderaManager", "type" : "m4.large", "image" : "ami-414b7271", "bootstrapScript" : "", "config" : { "subnetId" : "subnet-1e1f6f47", "rootVolumeType" : "gp2", "instanceNamePrefix" : "director", "rootVolumeSizeGB" : "50", "useSpotInstances" : "false", "securityGroupsIds" : "sg-fe2aca99" }, "tags" : { "Name" : "NewPlatform-ClouderaManager" }, "normalizeInstance" : true } }, "capabilities" : { "operatingSystemType" : "REDHAT_COMPATIBLE", "operatingSystemVersion" : "REDHAT_COMPATIBLE_6", "virtualizationType" : "HARDWARE_ASSISTED", "pythonVersion" : "2.6.6", "packageManager" : "YUM", "passwordlessSudoEnabled" : true, "selinuxEnabled" : true, "iptablesEnabled" : false }, "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 } }, { "ipAddress" : "10.170.1.107", "properties" : { "subnetId" : "subnet-1e1f6f47", "ebsOptimized" : "false", "virtualizationType" : "hvm", "imageId" : "ami-414b7271", "instanceType" : "m4.large", "keyName" : "NewPlatformRoot", "publicDnsName" : "", "sourceDestCheck" : "true", "availabilityZone" : "us-west-2c", "privateIpAddress" : "10.170.1.107", "rootDeviceType" : "ebs", "launchTime" : "Fri Jan 22 09:59:22 EST 2016", "instanceId" : "i-729e08a8", "placementGroupName" : "", "hypervisor" : "xen", "spot" : "false", "vpcId" : "vpc-1b47767e", "rootDeviceName" : "/dev/sda1", "privateDnsName" : "ip-10-170-1-107.us-west-2.compute.internal", "securityGroups" : "launch-wizard-1(sg-fe2aca99)", "architecture" : "x86_64" }, "virtualInstance" : { "id" : "b3ca74d7-4a82-49d2-8dbd-44cac2fad24a", "template" : { "name" : "ClouderaManager", "type" : "m4.large", "image" : "ami-414b7271", "bootstrapScript" : "", "config" : { "subnetId" : "subnet-1e1f6f47", "rootVolumeType" : "gp2", "instanceNamePrefix" : "director", "rootVolumeSizeGB" : "50", "useSpotInstances" : "false", "securityGroupsIds" : "sg-fe2aca99" }, "tags" : { "Name" : "NewPlatform-ClouderaManager" }, "normalizeInstance" : true } }, "capabilities" : { "operatingSystemType" : "REDHAT_COMPATIBLE", "operatingSystemVersion" : "REDHAT_COMPATIBLE_6", "virtualizationType" : "HARDWARE_ASSISTED", "pythonVersion" : "2.6.6", "packageManager" : "YUM", "passwordlessSudoEnabled" : true, "selinuxEnabled" : true, "iptablesEnabled" : false }, "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 } }, { "ipAddress" : "10.170.1.106", "properties" : { "subnetId" : "subnet-1e1f6f47", "ebsOptimized" : "false", "virtualizationType" : "hvm", "imageId" : "ami-414b7271", "instanceType" : "m4.large", "keyName" : "NewPlatformRoot", "publicDnsName" : "", "sourceDestCheck" : "true", "availabilityZone" : "us-west-2c", "privateIpAddress" : "10.170.1.106", "rootDeviceType" : "ebs", "launchTime" : "Fri Jan 22 09:59:22 EST 2016", "instanceId" : "i-739e08a9", "placementGroupName" : "", "hypervisor" : "xen", "spot" : "false", "vpcId" : "vpc-1b47767e", "rootDeviceName" : "/dev/sda1", "privateDnsName" : "ip-10-170-1-106.us-west-2.compute.internal", "securityGroups" : "launch-wizard-1(sg-fe2aca99)", "architecture" : "x86_64" }, "virtualInstance" : { "id" : "d097a064-82a1-4ef8-b108-fd64edb0450d", "template" : { "name" : "ClouderaManager", "type" : "m4.large", "image" : "ami-414b7271", "bootstrapScript" : "", "config" : { "subnetId" : "subnet-1e1f6f47", "rootVolumeType" : "gp2", "instanceNamePrefix" : "director", "rootVolumeSizeGB" : "50", "useSpotInstances" : "false", "securityGroupsIds" : "sg-fe2aca99" }, "tags" : { "Name" : "NewPlatform-ClouderaManager" }, "normalizeInstance" : true } }, "capabilities" : { "operatingSystemType" : "REDHAT_COMPATIBLE", "operatingSystemVersion" : "REDHAT_COMPATIBLE_6", "virtualizationType" : "HARDWARE_ASSISTED", "pythonVersion" : "2.6.6", "packageManager" : "YUM", "passwordlessSudoEnabled" : true, "selinuxEnabled" : true, "iptablesEnabled" : false }, "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 } }, { "ipAddress" : "10.170.0.142", "properties" : { "subnetId" : "subnet-1e1f6f47", "ebsOptimized" : "false", "virtualizationType" : "hvm", "imageId" : "ami-414b7271", "instanceType" : "m4.large", "keyName" : "NewPlatformRoot", "publicDnsName" : "", "sourceDestCheck" : "true", "availabilityZone" : "us-west-2c", "privateIpAddress" : "10.170.0.142", "rootDeviceType" : "ebs", "launchTime" : "Fri Jan 22 09:59:22 EST 2016", "instanceId" : "i-f8910722", "placementGroupName" : "", "hypervisor" : "xen", "spot" : "false", "vpcId" : "vpc-1b47767e", "rootDeviceName" : "/dev/sda1", "privateDnsName" : "ip-10-170-0-142.us-west-2.compute.internal", "securityGroups" : "launch-wizard-1(sg-fe2aca99)", "architecture" : "x86_64" }, "virtualInstance" : { "id" : "acaa5119-b13f-4051-8841-7567e15e30e5", "template" : { "name" : "ClouderaManager", "type" : "m4.large", "image" : "ami-414b7271", "bootstrapScript" : "", "config" : { "subnetId" : "subnet-1e1f6f47", "rootVolumeType" : "gp2", "instanceNamePrefix" : "director", "rootVolumeSizeGB" : "50", "useSpotInstances" : "false", "securityGroupsIds" : "sg-fe2aca99" }, "tags" : { "Name" : "NewPlatform-ClouderaManager" }, "normalizeInstance" : true } }, "capabilities" : { "operatingSystemType" : "REDHAT_COMPATIBLE", "operatingSystemVersion" : "REDHAT_COMPATIBLE_6", "virtualizationType" : "HARDWARE_ASSISTED", "pythonVersion" : "2.6.6", "packageManager" : "YUM", "passwordlessSudoEnabled" : true, "selinuxEnabled" : true, "iptablesEnabled" : false }, "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 } } ], "services" : [ { "serviceName" : "ZOOKEEPER", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-ZOOKEEPER-OwinYWHV", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-ZOOKEEPER-OwinYWHV/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] }, { "serviceName" : "SQOOP", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SQOOP-UrPslgqE", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SQOOP-UrPslgqE/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] }, { "serviceName" : "SPARK_ON_YARN", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SPARK_ON_YARN-AJfBtbso", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SPARK_ON_YARN-AJfBtbso/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] }, { "serviceName" : "HIVE", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HIVE-QAFjCOQn", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HIVE-QAFjCOQn/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] }, { "serviceName" : "OOZIE", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-OOZIE-wMmgwoTc", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-OOZIE-wMmgwoTc/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] }, { "serviceName" : "HUE", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HUE-UBSUTqcp", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HUE-UBSUTqcp/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] }, { "serviceName" : "YARN", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-YARN-Cahmhgyy", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-YARN-Cahmhgyy/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] }, { "serviceName" : "HDFS", "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HDFS-Fxvxaehf", "roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HDFS-Fxvxaehf/instances", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453733448326 }, "healthChecks" : [ ] } ], "url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/clusterRedirect/Cluster01", "instancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/clusterRedirect/Cluster01/hosts", "health" : { "status" : "NOT_AVAILABLE", "lastReported" : 1453821883390 } }
... View more
01-26-2016
07:17 AM
Greetings, The issue is persistant. We did change the default admin password in Cloudera Manager (necessary to keep the environment secure, or anyone in the company could login). Is there a way I can make Director aware of a password change? It was also unable to see the status before the psw change.
... View more
01-25-2016
07:00 AM
Greetings, I recently deployed Cloudera Director 2.0 and successfully pushed out a CDH 5 AWS cluster. I am running into a wierd issue where Cloudera Director does not know the state of the cluster. http://i.imgur.com/4fxmk1o.png While the cluster shows perfectly healthy in Cloudera Manager. http://i.imgur.com/gBSb6Vc.png I verified that Cloudera Director can SSH into the other instances without issue. Any thoughts? I see no errors in the Cloudera Director application logs other than an INFO level log line "RequestMappingInfo did not match any include patterns" The logs: http://pastebin.com/35D7tspS
... View more
Labels:
05-13-2015
04:12 AM
5 Kudos
Greetings GautamG! Thanks for your reply. I have actually resolved this issue which was caused by a combination of two factors. 1st - Way to many small files being created by pig denormalization jobs. Jobs processing this result data were making to many RPC calls at once, basically performing a denial of service attack on the namenode (around 1800 RPC calls a second at failure point). http://snowplowanalytics.com/blog/2013/05/30/dealing-with-hadoops-small-files-problem/ 2nd - Soon after resolving the small files problem, another problem arose. Seemingly random high queue processing load, at the time appeared very odd because the small files problem had been resolved and the amount of RPC calls was well within stable bound. The first issue turned out to be quite easy to solve. The pig jobs were modified to roll up all these small files into much larger files (not large on the scale of hadoop) so that instead of hundreds of thousands of files we were able to get them down to a just a few hundred. The second issue was far more interesting. This was difficult to troubleshoot and actually resolved by accident. There is actually almost no documentation on what would cause "high queue processing load" which led to me eventually pulling the source code down for HDFS. It eventually turned out to be caused by the JobHistory server trying to pull the logs from /tmp/logs. The JobHistory server was running into permissions failures when trying to pull said logs, which would lead to the high processing queue time and eventually a backlog in the RPC queue. ProTip from my pain: High queue processing happens when a job/someone tries to operate against files in HDFS that they do not have permissions for. The big problem with this is that even your least priviledged users can cause cluster instability by just trying to operate against files they do not have permission to. I found a problem with the JobHistory logs being retrieved in the browser. In my cause this turned out to be caused by some cleanup work I had done in the HDFS /tmp directory. I had accidently removed /tmp/logs without realizing this is the storage location of the YARN JobHistory server logs. The folder was automatically recreated by the yarn process, but with the wrong permissions needed for the JobHistory server to operate. It turns out this problem was caused by previous efforts to clean up the /tmp filespace in HDFS. The directory /tmp/logs is used by the JobHistory server to store logs for retrieval. When the directory was removed as being "aged out" it was automatically recreated, but with the wrong permissions (It was recreated with yarn:supergroup instead of mapred:hadoop). The way the directory works is that it leverages the sticky bit to force inheritance of the group permissions on all new files created. Under the correct conditions this would force all new files and folders to be owned by "hadoop". In this case it led to all new folders and files being owned by supergroup, which prevented the JobHistory server from being able to read the logs. You can see where I commented in this thread on how to fix it.. https://community.cloudera.com/t5/Web-UI-Hue-Beeswax/CDH5-0-VM-Error-getting-logs-for-job-1404657916663-0001/m-p/26498/highlight/false#M786 INCORRECT: drwxrwxrwt - yarn superuser 0 2015-05-10 09:28 /tmp/logs [root@namenode hadoop-hdfs]# sudo -u hdfs hdfs dfs -ls /tmp/logs/ Found 13 items drwxrwx--- - user1 supergroup 0 2015-05-10 09:28 /tmp/logs/user1 drwxrwx--- - user2 supergroup 0 2015-05-06 12:46 /tmp/logs/user2 drwxrwx--- - user3 supergroup 0 2015-05-06 12:54 /tmp/logs/user3 FIXED: drwxrwxrwt - mapred hadoop 0 2015-05-10 09:28 /tmp/logs [root@namenode hadoop-hdfs]# sudo -u hdfs hdfs dfs -ls /tmp/logs/ Found 13 items drwxrwx--- - user1 hadoop 0 2015-05-10 09:28 /tmp/logs/user1 drwxrwx--- - user2 hadoop 0 2015-05-06 12:46 /tmp/logs/user2 drwxrwx--- - user3 hadoop 0 2015-05-06 12:54 /tmp/logs/user3
... View more
05-11-2015
06:33 AM
9 Kudos
Greetings, The proper perms are /tmp/logs should be 777 and should have sticky bit set (the 1 in the 1777, this makes all child directories inherit the group of parent dir aka hadoop). sudo -u hdfs hdfs dfs -chmod 1777 /tmp/logs sudo -u hdfs hdfs dfs -chown mapred:hadoop /tmp/logs sudo -u hdfs hdfs dfs -chgrp -R hadoop /tmp/logs After this I would restart your JobHistory server.
... View more
01-06-2015
12:09 PM
Cloudera Engineering got back to me with a fix. If you run into this problem, disable and then re-enable Oozie HA. This will remove the old keytabs and regenerate them with the proper settings. Thanks!
... View more