Hi @AR Firstly please check if Hbase still has XaSecureAuthorization Coprocessor conf inside hbase-site.xml. Look for below properties. If its there then kindly remove it from through UI and restart the Hbase services and then check if underlining HDFS has correct permsiion and see if that resolve the issue hbase.coprocessor.master.classes hbase.coprocessor.region.classes ... View more

Sorry can't see logs. ... View more

The only recommended way is to upgrade the cluster to HDP2.4.2. You can also install spark 1.6.1 manually on HDP 2.3.2 but we don't recommend that. Thanks. ... View more

once upgrade completed we also need to clicked on finalize button to complete the upgrade process. if you have done this and still see previous version of spark then you should login to node and manually check spark package version through "rpm -qa|grep spark" command. ... View more

@Kumar Veerappan did you finalized the upgrade? ... View more

Hi @Jan Kytara I tried reproducing this issue on my cluster with your dataset and table but looks like it working fine even with skip.header.line.count parameter. hive> select count(*) from corrupt_rows; Query ID = hdfs_20160622192941_a2505b4a-96a7-4148-87ce-a52e92bd75c7 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1466074160497_0010) -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 .......... SUCCEEDED 1 1 0 0 0 0 Reducer 2 ...... SUCCEEDED 1 1 0 0 0 0 -------------------------------------------------------------------------------- VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 4.89 s -------------------------------------------------------------------------------- OK 90 Time taken: 5.467 seconds, Fetched: 1 row(s) -bash-4.1$ wc -l data.txt 91 data.txt -bash-4.1$ Which HDP version you are using? ... View more

Your sorting should happens on the basis of the key, here is an example for scala. val file = sc.textFile("some_local_text_file_pathname") val wordCounts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _, 1) // 2nd arg configures one task (same as number of partitions) .map(item => item.swap) // interchanges position of entries in each tuple .sortByKey(true, 1) // 1st arg configures ascending sort, 2nd arg configures one task .map(item => item.swap) ... View more