https://community.cloudera.com/t5/Support-Questions/Livy-Job-failing-after-upgrading-cluster-to-HDP-2-6-3/m-p/199410#M161449 <P> I came across this same issue after a cluster upgrade from HDP 2.6.1 to HDP 2.6.4. </P><P> From HDP 2.6.3, livy was changed from cloudera 0.3.0 to apache 0.4.0-incubating and so I duly updated my sbt library dependencies to fetch the 0.4.0 from maven (e.g. "org.apache.livy" % "livy-api" % "0.4.0-incubating", etc). However, using pkgdiff I found that HDP's 0.4.0 is *not quite* the same as apache's 0.4.0. In particular there is a new "jobType" parameter added to org/apache/livy/client/common/HttpMessages$SerializedJob and the lack of this parameter being supplied by the client code (linked to apache 0.4.0) directly results in the "IllegalArgumentException: Invalid kind: null" exception and the RpcDispatcher's "NoSuchElementException: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx". This mismatch between the 0.4.0 client and "0.4.0" server can be fixed by forcing the livy client libraries to be picked up from the Hortonworks maven repo in build.sbt as follows (something I did not have to do for livy 0.3.0). </P> <PRE> resolvers += "hortonworks repo" at "http://repo.hortonworks.com/content/repositories/releases" libraryDependencies ++= Seq( ... "org.apache.livy" % "livy-api" % "0.4.0.2.6.4.0-91", "org.apache.livy" %% "livy-scala-api" % "0.4.0.2.6.4.0-91", "org.apache.livy" % "livy-client-http" % "0.4.0.2.6.4.0-91" ) </PRE><P> (Note that this new parameter is in 0.5.0-incubating (not yet on maven) so it seems that either Hortonworks took a later version or Apache removed this afterwards - either way it is not in the 0.4.0-incubating release at <A href="https://github.com/apache/incubator-livy" target="_blank">https://github.com/apache/incubator-livy</A> made on August 30th 2017) </P><P>Hope this helps someone !</P> Thu, 08 Feb 2018 22:25:47 GMT simon_george 2018-02-08T22:25:47Z