ambari cluster + API with request_status fail

[#1]

after we installed the Ambari cluster and blueprint ( hadoop version 2.6 ) ,

I run the following API in order to see if blueprint complete successfully the installation

 curl -s --user  "admin"":""admin" -H "X-Requested-By: ambari"  -X GET http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1 

and it is very strange because when I access the ambari GUI , I see all services are up include the compliments , and all host are up also as master01 , master02 , master03 , kafka , worker01-03

but from the API we get failed on request_status

"request_status" : "FAILED",

please advice how to know exactly why requisite status is fail ? , what was wrong or how to know fail content ?

partital of the output from the API

curl -s --user  "admin"":""admin" -H "X-Requested-By: ambari"  -X GET http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1 | more
{
  "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1",
  "Requests" : {
    "aborted_task_count" : 1,
    "cluster_name" : "hdp",
    "completed_task_count" : 122,
    "create_time" : 1513540772178,
    "end_time" : -1,
    "exclusive" : false,
    "failed_task_count" : 1,
    "id" : 1,
    "inputs" : null,
    "operation_level" : null,
    "progress_percent" : 100.0,
    "queued_task_count" : 0,
    "request_context" : "Logical Request: Provision Cluster 'hdp'",
    "request_schedule" : null,
    "request_status" : "FAILED",
    "resource_filters" : null,
    "start_time" : -1,
    "task_count" : 122,
    "timed_out_task_count" : 0,
    "type" : null
  },
  "stages" : [
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/stages/1",
      "Stage" : {
        "cluster_name" : "hdp",
        "request_id" : 1,
        "stage_id" : 1
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/stages/2",
      "Stage" : {
        "cluster_name" : "hdp",
        "request_id" : 1,
        "stage_id" : 2
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/stages/3",
      "Stage" : {
        "cluster_name" : "hdp",
        "request_id" : 1,
        "stage_id" : 3
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/stages/4",
      "Stage" : {
        "cluster_name" : "hdp",
        "request_id" : 1,
        "stage_id" : 4
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/stages/5",
      "Stage" : {
        "cluster_name" : "hdp",
        "request_id" : 1,
        "stage_id" : 5
      }
    }
  ],
  "tasks" : [
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/tasks/10000",
      "Tasks" : {
        "cluster_name" : "hdp",
        "id" : 10000,
        "request_id" : 1,
        "stage_id" : 1
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/tasks/10001",
      "Tasks" : {
        "cluster_name" : "hdp",
        "id" : 10001,
        "request_id" : 1,
        "stage_id" : 1
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/tasks/10002",
      "Tasks" : {
        "cluster_name" : "hdp",
        "id" : 10002,
        "request_id" : 1,
        "stage_id" : 1
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/tasks/10003",
      "Tasks" : {
        "cluster_name" : "hdp",
        "id" : 10003,
        "request_id" : 1,
        "stage_id" : 1
      }
    },
    {
      "href" : "http://master02.sys6363.com:8080/api/v1/clusters/hdp/requests/1/tasks/10004",
      "Tasks" : {
        "cluster_name" : "hdp",
        "id" : 10004,
        "request_id" : 1,
        "stage_id" : 1
      }
    },
    {

[#2]

@Michael Bronson,

Can you please attach your ambari server logs under (/var/log/ambari-server/ambari-server.log ) . Also can you please attach the Ambari GUI screenshot and the performed operations screenshot ( click the button next to the cluster name).

Thanks,

Aditya

[#3]

this is very huge file , whant you want to search in this file?

[#4]

@Michael Bronson,

Search if you have any ERROR logs related to the blueprint installation. Also can you please attach the GUI screenshots

[#5]

Dear Aditya , sorry for the late response

here the ERROR from the ambari-server.log

regarding the ambari GUI , all services/hosts/components are up ( green ) , so I not see any issue from this side

 grep ERROR ambari-server.log
17 Dec 2017 19:37:58,236 ERROR [pool-3-thread-1] ViewRegistry:1056 - Failed to update 'use.hive.interactive.mode' parameter for viewName: HIVE, version: 1.5.0. Exception: org.apache.ambari.server.ServiceComponentNotFoundException: ServiceComponent not found, clusterName=hdp, serviceName=HIVE, serviceComponentName=HIVE_SERVER_INTERACTIVE
17 Dec 2017 19:37:58,695 ERROR [ambari-client-thread-89] ClusterImpl:2716 - No service found for config types '[cluster-env]', service config version not created
17 Dec 2017 19:38:04,119 ERROR [pool-3-thread-1] ViewRegistry:1056 - Failed to update 'use.hive.interactive.mode' parameter for viewName: HIVE, version: 2.0.0. Exception: org.apache.ambari.server.ServiceComponentNotFoundException: ServiceComponent not found, clusterName=hdp, serviceName=HIVE, serviceComponentName=HIVE_SERVER_INTERACTIVE
17 Dec 2017 19:47:16,278 ERROR [ambari-heartbeat-processor-0] HeartbeatProcessor:554 - Operation failed - may be retried. Service component host: SPARK2_JOBHISTORYSERVER, host: master03.sys6363.com Action id 9-3 and taskId 90
17 Dec 2017 19:47:17,025 ERROR [ambari-action-scheduler] ActionScheduler:440 - Operation completely failed, aborting request id: 9
17 Dec 2017 19:59:19,161 ERROR [Stack Version Loading Thread] URLStreamProvider:245 - Received HTTP 404 response from URL: http://master02.sys6363.com/HDP/hdp_urlinfo.json
17 Dec 2017 19:59:22,550 ERROR [main] ViewRegistry:1056 - Failed to update 'use.hive.interactive.mode' parameter for viewName: HIVE, version: 1.5.0. Exception: org.apache.ambari.server.ServiceComponentNotFoundException: ServiceComponent not found, clusterName=hdp, serviceName=HIVE, serviceComponentName=HIVE_SERVER_INTERACTIVE
17 Dec 2017 19:59:25,500 ERROR [main] ViewRegistry:1056 - Failed to update 'use.hive.interactive.mode' parameter for viewName: HIVE, version: 2.0.0. Exception: org.apache.ambari.server.ServiceComponentNotFoundException: ServiceComponent not found, clusterName=hdp, serviceName=HIVE, serviceComponentName=HIVE_SERVER_INTERACTIVE
17 Dec 2017 17:45:59,151 ERROR [qtp-ambari-agent-67] HeartBeatHandler:200 - CurrentResponseId unknown for worker02.sys6363.com - send register command

[#6]

Another quastion , is it possible to find the real problem from the API and not from the log , ?

[#7]

I have the same problem, does it fix?

[#8]

hi , the answer is no

[#9]

I have the same problem too

Ambari 2.6.1

HDP 2.6.4.0-93

04 Apr 2018 09:13:26,884 ERROR [ambari-client-thread-25] ClusterImpl:1953 - No service found for config types '[cluster-env]', service config version not created
...
04 Apr 2018 09:13:31,614 ERROR [pool-3-thread-1] ViewRegistry:1047 - Failed to update 'use.hive.interactive.mode' parameter for viewName: HIVE, version: 2.0.0. Exception: org.apache.ambari.server.ServiceComponentNotFoundException: ServiceComponent not found, clusterName=mytestcluster, serviceName=HIVE, serviceComponentName=HIVE_SERVER_INTERACTIVE
...
04 Apr 2018 09:13:31,694  INFO [pool-20-thread-1] AmbariContext:567 - Config type hdfs-site is resolved in the cluster config.
04 Apr 2018 09:13:31,695  INFO [pool-20-thread-1] AmbariContext:567 - Config type hive-site is resolved in the cluster config.
04 Apr 2018 09:13:31,704  INFO [pool-20-thread-1] AmbariContext:567 - Config type yarn-site is resolved in the cluster config.
04 Apr 2018 09:13:31,705  INFO [pool-20-thread-1] AmbariContext:567 - Config type zookeeper-env is resolved in the cluster config.
04 Apr 2018 09:13:31,706  INFO [pool-20-thread-1] AmbariContext:567 - Config type oozie-site is resolved in the cluster config.
04 Apr 2018 09:13:31,707  INFO [pool-20-thread-1] AmbariContext:567 - Config type hive-env is resolved in the cluster config.
04 Apr 2018 09:13:31,718  INFO [pool-20-thread-1] AmbariContext:567 - Config type webhcat-site is resolved in the cluster config.
04 Apr 2018 09:13:31,719  INFO [pool-20-thread-1] AmbariContext:564 - Config type cluster-env not resolved yet, Blueprint deployment will wait until configuration update is completed
04 Apr 2018 09:13:31,728  INFO [pool-20-thread-1] AmbariContext:567 - Config type hive-interactive-site is resolved in the cluster config.
04 Apr 2018 09:13:31,729  INFO [pool-20-thread-1] AmbariContext:567 - Config type mapred-site is resolved in the cluster config.
04 Apr 2018 09:13:31,740  INFO [pool-20-thread-1] AmbariContext:567 - Config type core-site is resolved in the cluster config.
04 Apr 2018 09:13:31,740  INFO [pool-20-thread-1] AmbariContext:576 - Waiting for 1 configuration types to be resolved before Blueprint deployment can continue

Looking at the blueprint dumped from my previous running cluster

"cluster-env" : {
...
\"HIVE_SERVER_INTERACTIVE\": {
    \"STACK-SELECT-PACKAGE\": \"hive-server2-hive2\",
    \"INSTALL\": [
    \"hive-server2-hive2\"
    ],
    \"PATCH\": [
    \"hive-server2-hive2\"
    ],
    \"STANDARD\": [
    \"hive-server2-hive2\"
    ]
},
...
}

[#10]

@Alexandre GRIFFAUT

If possible can you paste the blueprint in here and only scramble sensitive info this can help community members analyze it