When you run OfflineMetaRepair, most likely you will run it from your userid or root.  Then we may get some opaque errors like "java.lang.AbstractMethodError: org.apache.hadoop.hbase.ipc.RpcScheduler.getWriteQueueLength()".    If you check in HDFS, you may see that the meta directory is no longer owned by hbase:   $ hdfs dfs -ls /hbase/data/hbase/ Found 2 items drwxr-xr-x - root hbase 0 2017-09-12 13:58 /hbase/data/hbase/meta drwxr-xr-x - hbase hbase 0 2016-06-15 15:02 /hbase/data/hbase/namespace Manually chown -R it and restart HBase fixed it for me.     ... View more

@MilesYao That may be. On that topic, I don't think anytime soon as Cloudera does not support many features that it does for Spark 1.6. I suspect that sometime post CDH 6 we will see Spark 2.x supplant Spark 1.x as the only version of Spark in CDH. Ah, I checked out HDP and see what you are getting at. It is really trivial on the difference. Cloudera asks you to put a file on the CM host and configure a separate parcel while HDP includes both Spark1 and Spark2 packages in the same repo. ... View more

@Miles Yao Good catch!!! Just updated. The phoenix jar is here to work with JDBC interpreter rather than spark. ... View more

How i can know if the parcels files should has sha or sha1 signature?   When Spark2 will be part of CDH4 parcels, which version?   currently i have 1.6, when i add 2.0 will the spark history server be 2.0 also? if i used 2.0 parcels, which one will be the active one on the cluster 1.6 or 2.0?   ... View more

Yes, that works.  "CSD file" sounds like a text config file.  Adding a simple description that it's a JAR in the instruction page would have clarified.   Thanks again. Miles   ... View more

What version are you using ?  Can you send the workflow.xml generated by Hue ?  Most likely it's fixed on the latest version. ... View more

It has always been documented in "Known Issues": https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_spark_ki.html  Generally speaking, there aren't differences. Not supported != different. However there are some pieces that aren't shipped like the thrift server and SparkR.   Usually differences crop up when upstream introduces a breaking change and it can't be followed in a minor release. For example: default in CDH is for the "legacy" memory config parameters to be active so that default memory config doesn't change in 1.6. Sometimes it relates to other stuff in the platfrom that can't change, like I think the Akka version is (was) different because other stuff in Hadoop needed a different version.   The biggest example of this IMHO is Spark Streaming + Kafka. Spark 1.x doesn't support Kafka 0.9+ but CDH 5.7+ had to move to it to get security features. So CDH Spark 1.6 will actually only work with 0.9+ because the Kafka differences are mutually incompatible. Good in that you can use recent Kafka, but, a difference!   Most of it though are warnings about incompatibilities between what Spark happens to support and what CDH ships in other components.  ... View more

@sgiri - This has already been answered above. If you use un-secure YARN, all your containers and the commands the container runs (such as in shell action) will run as the "yarn" user (i.e. the user your NodeManager daemon runs as). If you want to run containers in YARN as the actual submitting user, use the LinuxContainerExecutor instead. The caveat of using it is that you need your user accounts available on all NodeManagers so a setuid can be done against them when spawning the container processes. ... View more

You just found a bug; that page was updated prematurely and will be rolled back. 5.8 will be released very, very soon though. ... View more