Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Adding another RS with a custom service

Highlighted

Adding another RS with a custom service

Contributor

I already posted the manual steps for adding an additional RS to an existing HBase cluster (see here )

Due to the limitation that I can't manage it via Ambari I looked at the sources and other custom service definitions for a stack. the metainfo.xml file will follow below. Anyhow, when I add this to my ambari-server the server starts and I see the new service within the components list.

When I only select my new service Ambari tells me that other services are required in order to have this service running, so that's working fine. But then I get the exception:

500 status code received on GET method for API: /api/v1/stacks/HDP/versions/2.4/recommendations
  Error message: Server Error

and the ambari-server.log shows this

26 Jun 2017 14:22:49,119  WARN [qtp-ambari-client-22] ServletHandler:563 - /api/v1/stacks/HDP/versions/2.4/recommendations
java.lang.NullPointerException
    at org.apache.ambari.server.state.PropertyInfo.getAttributesMap(PropertyInfo.java:145)
    at org.apache.ambari.server.state.PropertyInfo.convertToResponse(PropertyInfo.java:128)
    at org.apache.ambari.server.controller.AmbariManagementControllerImpl.getStackConfigurations(AmbariManagementControllerImpl.java:3898)
    at org.apache.ambari.server.controller.AmbariManagementControllerImpl.getStackConfigurations(AmbariManagementControllerImpl.java:3867)
    at org.apache.ambari.server.controller.internal.StackConfigurationResourceProvider$1.invoke(StackConfigurationResourceProvider.java:114)
    at org.apache.ambari.server.controller.internal.StackConfigurationResourceProvider$1.invoke(StackConfigurationResourceProvider.java:111)
    at org.apache.ambari.server.controller.internal.AbstractResourceProvider.getResources(AbstractResourceProvider.java:302)
    at org.apache.ambari.server.controller.internal.StackConfigurationResourceProvider.getResources(StackConfigurationResourceProvider.java:111)
    at org.apache.ambari.server.controller.internal.ClusterControllerImpl$ExtendedResourceProviderWrapper.queryForResources(ClusterControllerImpl.java:945)
    at org.apache.ambari.server.controller.internal.ClusterControllerImpl.getResources(ClusterControllerImpl.java:132)
    at org.apache.ambari.server.api.query.QueryImpl.doQuery(QueryImpl.java:508)
    at org.apache.ambari.server.api.query.QueryImpl.queryForSubResources(QueryImpl.java:463)
    at org.apache.ambari.server.api.query.QueryImpl.queryForSubResources(QueryImpl.java:482)
    at org.apache.ambari.server.api.query.QueryImpl.queryForResources(QueryImpl.java:436)
    at org.apache.ambari.server.api.query.QueryImpl.execute(QueryImpl.java:216)
    at org.apache.ambari.server.api.handlers.ReadHandler.handleRequest(ReadHandler.java:68)
    at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75)
    at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.getServicesInformation(StackAdvisorCommand.java:356)
    at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:247)
    at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.recommend(StackAdvisorHelper.java:109)
    at org.apache.ambari.server.controller.internal.RecommendationResourceProvider.createResources(RecommendationResourceProvider.java:92)
    at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
    at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
    at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
    at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
    at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75)
    at org.apache.ambari.server.api.services.RecommendationService.getRecommendation(RecommendationService.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

the metainfo.xml file:

<metainfo>
  <schemaVersion>2.0</schemaVersion>
  <services>
    <service>
      <name>HBASE-RS</name>
      <displayName>HBase-RS</displayName>
      <comment>Additional Regionservers on one host
      </comment>
      <version>1.1.2.2.4</version>
      <components>
      
        <component>
          <name>HBASE_REGIONSERVER_01</name>
          <displayName>RegionServer01</displayName>
          <category>MASTER</category>
          <cardinality>1+</cardinality>
          <versionAdvertised>true</versionAdvertised>
          <decommissionAllowed>true</decommissionAllowed>
          <timelineAppid>HBASE</timelineAppid>
          <dependencies>
            <dependency>
              <name>HBASE/HBASE_MASTER</name>
              <scope>cluster</scope>
              <auto-deploy>
                <enabled>false</enabled>
                <co-locate>HBASE/HBASE_MASTER</co-locate>
              </auto-deploy>
            </dependency>
            <dependency>
              <name>HBASE/HBASE_REGIONSERVER</name>
              <scope>host</scope>
              <auto-deploy>
                <enabled>false</enabled>
              </auto-deploy>
            </dependency>
          </dependencies>
          <commandScript>
            <script>scripts/hbase_regionserver.py</script>
            <scriptType>PYTHON</scriptType>
            <timeout>1200</timeout>
          </commandScript>
          <bulkCommands>
            <displayName>RegionServers</displayName>
            <!-- Used by decommission and recommission -->
            <masterComponent>HBASE/HBASE_MASTER</masterComponent>
          </bulkCommands>
          <logs>
            <log>
              <logId>hbase_regionserver01</logId>
              <primary>true</primary>
            </log>
          </logs>
        </component>

      </components>

      <osSpecifics>
        <osSpecific>
          <osFamily>redhat6,redhat7</osFamily>
          <packages>
            <package>
              <name>hbase</name>
            </package>
          </packages>
        </osSpecific>
      </osSpecifics>

      <commandScript>
        <script>scripts/service_check.py</script>
        <scriptType>PYTHON</scriptType>
        <timeout>300</timeout>
      </commandScript>
      
      <requiredServices>
        <service>ZOOKEEPER</service>
        <service>HDFS</service>
        <service>HBASE</service>
      </requiredServices>

      <!-- configuration files that are expected by the service (config files owned by other services are specified in this list) -->
      <configuration-dependencies>
        <config-type>core-site</config-type> <!-- hbase puts core-site in it's folder -->
        <config-type>hbase-policy</config-type>
        <!--<config-type>hbase-site</config-type>-->
        <!--<config-type>hbase-env</config-type>-->
        <!--<config-type>hbase-log4j</config-type>-->
      </configuration-dependencies>

    </service>
  </services>

I tried to set the category to SLAVE, as it basically is a SLAVE, and the master component is a HBASE_MASTER, but same error.

Any ideas what I do wrong?

Thanks.

4 REPLIES 4

Re: Adding another RS with a custom service

Expert Contributor

@David Pocivalnik

every time recommendation API call is made, new entry for logs is made at path "/var/run/ambari-server/stack-recommendations/" on ambari-server host. there will be several and if you are not certain about the last entry then please move all historical existing entries to different location and then again go through add service wizard to produce a repro.

Entry will have stackadvisor.err and stackadvisor.out files. Please refer them for further debugging the issue. If it's not clear from logs, please compress the directory and post it on this thread, So that we can get more information for debugging purpose.

Re: Adding another RS with a custom service

Contributor

Thanks for the hint @jaimin . Attached the files from the folder stack-recommendations.zip , unfortunately the stackadvisor.err file is empty, so I'm not sure that helps. Maybe you see a problem?

Maybe a question for understanding: Do I have to define a MASTER component for a service, or is it ok if I only have SLAVEs? As in my case, the MASTER is the one from HBASE.

Re: Adding another RS with a custom service

Expert Contributor

@David Pocivalnik

It is ok to have service that only has slave. Flume service is a working example for that: https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/FLUME/1...

There seems to be something else in the custom service that you defined which might be causing this issue. Does the new custom service has any configurations ? can you also post zip file for the custom hbase service that you are introducing ?

Re: Adding another RS with a custom service

Contributor

Thanks for the info @jaimin.

I just re-created my virtual machines, now with Ambari 2.4.0 instead of 2.2.2.0. My custom service is again added to the stack definition, and now I'm able to assign masters and slaves, including my custom service. So maybe it's a problem with the 'old' version.

The error I now get is

org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Configuration with tag 'version1' exists for 'hbase-env'

ambari-server.log
ClusterImpl:2523 - Looking for service for config types [hbase-env]
27 Jun 2017 07:27:07,652 ERROR [ambari-client-thread-24] AbstractResourceProvider:343 - Caught AmbariException when modifying a resource
org.apache.ambari.server.AmbariException: Configuration with tag 'version1' exists for 'hbase-env'
    at org.apache.ambari.server.controller.AmbariManagementControllerImpl.createConfiguration(AmbariManagementControllerImpl.java:855)
    at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1653)
    at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1473)
    at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:311)
    at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:308)
    at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:455)
    at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:336)
    at org.apache.ambari.server.controller.internal.ClusterResourceProvider.updateResourcesAuthorized(ClusterResourceProvider.java:308)
    at org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider.updateResources(AbstractAuthorizedResourceProvider.java:301)
    at org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:319)
    at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:125)
    at org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:45)
    at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73)
    at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:145)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90)
    at org.apache.ambari.server.api.services.ClusterService.updateCluster(ClusterService.java:142)

attached the configuration and package folders of my service, the metainfo didn't change (except the version, as it's 1.1.2 now) ambari-hbase-rs.zip

basically I only changed some port numbers, location for log and pid file. and I removed any code regarding Windows OS.

Thanks for your help.

Edit

I was having some problem in my params_linux.py script, as some of the imports ain't available.

Right now I can create the intended symlink, but it's pointing to a wrong location, the original RS is like this:

hbase-regionserver -> /usr/hdp/2.5.6.0-40/hbase

and mine is

hbase-regionserver-01 -> /usr/hdp/2.5.0.0/hbase

The path for the target is created within status_params.py (at the moment) looking like this:

regionserver_root_dir_specific = format("{stack_root}/{stack_version_formatted}/hbase")

So my question is: Is the {stack_version_formatted} wrong? (no change of the code, just re-using it)

or do I have to use another variable?

Thanks for your time

Don't have an account?
Coming from Hortonworks? Activate your account here