works !   I have a old cdh version (5.2) upgrading to 5.4.4 last week.   In my db, spark use the following name :   scm=# select * from services;  service_id | optimistic_lock_version |    name    | service_type | cluster_id | maintenance_count |        display_name         | generation ------------+-------------------------+------------+--------------+------------+-------------------+-----------------------------+------------          38 |                      30 | spark      | SPARK        |          2 |                 0 | Spark                       |          1   After deleting spark standalone and installing spark on yarn, the table has been updated :   scm=# select * from services;  service_id | optimistic_lock_version |     name      | service_type  | cluster_id | maintenance_count |        display_name         | generation ------------+-------------------------+---------------+---------------+------------+-------------------+-----------------------------+------------          63 |                       3 | spark_on_yarn | SPARK_ON_YARN |          2 |                 0 | Spark                       |          1   regrads,     ... View more

This may be too unspecific to be helpful, but I recall several JIRAs fixed for Spark 1.4 that concern the .inprogress files and history server. I expect that whatever this is could be related. If so, then the fix would be coming in 5.5 at the latest. ... View more

Thanks for the response. I have been meaning to come back and add a comment here. I found the root of the problem as well as the solution here:   http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/kerberos-authentication-failure-GSSAPI-Failure-gss-accept-sec/td-p/23333   ... View more

Hi,   thank you for your feedback.   I have upgraded into CDH 5.3 and spark history server (via yarn) was successfully started.   regards,     ... View more