Support Questions
Find answers, ask questions, and share your expertise

Livy issue while HDP 2.6.2.0 upgrade

Contributor

I'm upgrading from HDP 2.4.2.0 to 2.6.2.0. All tasks from Ambari wizard were performed OK but Spark fails.

I got this error message while restarting Spark Thrift Server

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py", line 87, in <module>
    SparkThriftServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 830, in restart
    self.stop(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py", line 57, in stop
    import params
  File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/params.py", line 262, in <module>
    livy_principal = livy_kerberos_principal.replace('_HOST', config['hostname'].lower())
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
    raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'livy.server.launch.kerberos.principal' was not found in configurations dictionary!

I just added livy.server.launch.kerberos.principal in /etc/livy/conf/livy-defaults.conf but doesn't work.

What should I do? Thanks in advance!

1 REPLY 1

Super Mentor

@Facundo Bianco

I remember a very similar issue i faced some times back.

Can you please check your "clusterstate" and "stack" table in the ambari DB?

# psql -U ambari ambari
Password for user ambari: bigdata
psql (9.2.18)
Type "help" for help.

ambari=> select * from clusterstate where current_stack_id=2;
 cluster_id | current_cluster_state | current_stack_id 
------------+-----------------------+------------------
          2 |                       |                2
(1 row)

ambari=> select * from stack where stack_id=2;
 stack_id | stack_name | stack_version 
----------+------------+---------------
        2 | HDP        | 2.6
(1 row)

.

The Stack ID =2 should be pointing to the HDP 2.6 stack and also the clusterstate table record should have the same current_stack_id =2.

If you find any difference then you should collect a DB dump of ambari and then update the "clusterstate" table to point to the correct stack_id (which is 2 for HDP 2.6 clusters).

.

This is just a guess at this point, else we will have to look at the livy conf and the upgrade logs.