Server-Error while adding a host to Ambari: NFS_GATEWAY component not found

[#1]

I am actually facing a problem while trying to add the node hosting the ambari-server to the list of the cluster hosts. In the step Review, and when I click on Deploy, I get this error:

Looking at the ambari-server.log. I can see the following WARN (which should be the origin of the error):

29 Dec 2015 17:29:49,593  INFO [qtp-client-20] StackAdvisorRunner:70 -     advisor script stdout: StackAdvisor implementation for stack HDP, version 2.0.6 was loaded
StackAdvisor implementation for stack HDP, version 2.1 was loaded
StackAdvisor implementation for stack HDP, version 2.2 was loaded
StackAdvisor implementation for stack HDP, version 2.3 was loaded
Returning HDP23StackAdvisor implementation
29 Dec 2015 17:29:49,593  INFO [qtp-client-20] StackAdvisorRunner:71 -     advisor script stderr:
29 Dec 2015 17:29:50,269  INFO [qtp-client-22] StackAdvisorRunner:47 - Script=/var/lib/ambari-server/resources/scripts/stack_advisor.py, actionDirectory=/var/run/ambari-server/stack-recommendations/4, command=validate-component-layout
29 Dec 2015 17:29:50,274  INFO [qtp-client-22] StackAdvisorRunner:61 - Stack-advisor output=/var/run/ambari-server/stack-recommendations/4/stackadvisor.out, error=/var/run/ambari-server/stack-recommendations/4/stackadvisor.err
29 Dec 2015 17:29:50,404  INFO [qtp-client-22] StackAdvisorRunner:69 - Stack advisor output files
29 Dec 2015 17:29:50,405  INFO [qtp-client-22] StackAdvisorRunner:70 -     advisor script stdout: StackAdvisor implementation for stack HDP, version 2.0.6 was loaded
StackAdvisor implementation for stack HDP, version 2.1 was loaded
StackAdvisor implementation for stack HDP, version 2.2 was loaded
StackAdvisor implementation for stack HDP, version 2.3 was loaded
Returning HDP23StackAdvisor implementation
29 Dec 2015 17:29:50,405  INFO [qtp-client-22] StackAdvisorRunner:71 -     advisor script stderr:
29 Dec 2015 17:29:53,471  WARN [qtp-client-20] ServletHandler:563 - /api/v1/clusters/HdpCluster/services
java.lang.RuntimeException: Trying to create a ServiceComponent not recognized in stack info, clusterName=HdpCluster, serviceName=HDFS, componentName=NFS_GATEWAY, stackInfo=HDP-2.2
        at org.apache.ambari.server.state.ServiceComponentImpl.<init>(ServiceComponentImpl.java:107)
        at org.apache.ambari.server.state.ServiceComponentImpl$$EnhancerByGuice$$149e948d.<init>(<generated>)

3 months ago, I upgraded my cluster from HDP 2.2 to HDP2.3, the error might be related to that action. Besides, I can see the NFS_GATEWAY component being installed in the Ambari->HDFS, however when I click on it, I find out that no host does really implement the NFS_GATEWAY component.

My ultimate goal is to install Kerberos. To do so, I should add the ambari-server to the list of the cluster hosts. Should I get ride of the NFS_GATEWAY component ? If yes, what actions should I preform ?

PS: When I execute the curl -DELETE command in an attempt to remove the NFS_GATEWAY component, I receive the 404 componont not found...

[#2]

NFS_GATEWAY is defined only in HDP-2.3 but from the logs it looks like its being expected in HDP-2.2. Can you check the DB and check that the cluster is indeed at HDP-2.3 - see tables "stack", "clusters" and "clusterstate". Which version of Ambari are you using and how did you perform the upgrade from HDP-2.2 to HDP-2.3 - manual, or rolling?

[#3]

@smohanty , Thank you for your quick reply.

I performed a Manual upgrade from HDP 2.2 to HDP 2.3. I manually upgraded all services. All the services (hive,hdfs...) are upgraded and runs normally.

I am running the Ambari 2.1 version

Running select * on the 3 tables shows:

In Ambari and in "stack and versions" view, I have: HDP-2.3.0.0 being the "current" cluster version. Would this give you a better picture?

[#4]

And in table servicecomponentdesiredstate, several component_name are still using "3" as desired_stack_id (including HDFS components). Does this mean that I am still running HDP 2.2 behind the scenes ?

[#5 · Accepted solution]

The desired_stack_id for hostcomponentdesiredstate and servicecomponentdesiredstate and current_stack_id for hostcomponentstate should point to HDP-2.3. That could be the reason for the error you see. Also check the version field for hostcomponentstate and see if that is reporting the 2.3 version.

[#6]

I'd check the results of the hdp-select tool and whether you finanlized the previous upgrade.

[#7]

@Artem Ervits

, @smohanty

Indeed, I forgot to run this command:

ambari-server upgradestack HDP-2.3

This updated the entire ambari database!

Thank you guys so much !