Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Rising Star

During the upgrade :

if you are using postgres DB, you will face a script failure on the following statement

Alter table users add constraint "uniq_user_0"

Solution :

psql -U ambari_user -d bigope_ambari

alter table ambari.user drop constraint "uniq_user_0"

After the Upgrade :

Spark

1-you have to recompile and rebuild your jars using the new dependencies of Spark 1.6.2 Your pom.xml needs to have the new versions ( new jars of HDP 2.5.3 )

2-Update any custom jars and check all versions used in your Pom.xml

i.e

<spark.version>1.6.2</spark.version>
<hbase.version>1.2.1</hbase.version>

3- Remove the argument spark.yarn.jar if you are using it when submitting the job.

--conf spark.yarn.jar=hdfs://bigopeclu/user/oozie/share/lib/lib_20160515013357/spark/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar.

Error noticed when leaving the arg above :

java.lang.IllegalArgumentException: Invalid ContainerId: container_e97_1484824056011_0038_02_000001

or

java.lang.IllegalArgumentException

Spark - Oozie

1- if you are launching Spark via Oozie ( or via Hue ) chek you get all the libs here with the right permissions :

-rwxr-xr-x   3 oozie hdfs     339666 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/hbase-site.xml
-rwxr-xr-x   3 oozie hdfs     339666 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/datanucleus-api-jdo-3.2.6.jar
-rwxr-xr-x   3 oozie hdfs    1890075 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/spark/datanucleus-core-3.2.10.jar
-rwxr-xr-x   3 oozie hdfs    1809447 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/spark/datanucleus-rdbms-3.2.9.jar
-rwxr-xr-x   3 oozie hdfs      22440 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/spark/oozie-sharelib-spark-4.2.0.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs      44846 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/py4j-0.9-src.zip
-rwxr-xr-x   3 oozie hdfs     357563 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/pyspark.zip
-rwxr-xr-x   3 oozie hdfs  188897932 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs  110488188 2017-01-19 17:43 /user/oozie/share/lib/lib_20170119102144/spark/spark-examples-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs  188897932 2017-01-19 17:43 /user/oozie/share/lib/lib_20170119102144/spark/spark-hdp-assembly.jar
-rwxr-xr-x   3 oozie hdfs     516062 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/aws-java-sdk-core-1.10.6.jar
-rwxr-xr-x   3 oozie hdfs     258578 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/aws-java-sdk-kms-1.10.6.jar
-rwxr-xr-x   3 oozie hdfs     570101 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/aws-java-sdk-s3-1.10.6.jar
-rwxr-xr-x   3 oozie hdfs      10092 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/azure-keyvault-core-0.8.0.jar
-rwxr-xr-x   3 oozie hdfs     745325 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/azure-storage-4.2.0.jar
-rwxr-xr-x   3 oozie hdfs     434678 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/commons-lang3-3.4.jar
-rwxr-xr-x   3 oozie hdfs    1648200 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/guava-11.0.2.jar
-rwxr-xr-x   3 oozie hdfs     153855 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/hadoop-aws-2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs     163348 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/hadoop-azure-2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs      38605 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/jackson-annotations-2.4.0.jar
-rwxr-xr-x   3 oozie hdfs     225302 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/jackson-core-2.4.4.jar
-rwxr-xr-x   3 oozie hdfs    1076926 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/jackson-databind-2.4.4.jar
-rwxr-xr-x   3 oozie hdfs     570478 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/joda-time-2.1.jar
-rwxr-xr-x   3 oozie hdfs      16046 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/json-simple-1.1.jar
-rwxr-xr-x   3 oozie hdfs      12543 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/oozie-hadoop-utils-hadoop-2-4.2.0.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs      51854 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/oozie-sharelib-oozie-4.2.0.2.5.3.0-37.jar

PS : here we have 755 can be restricted more

hdfs dfs -chmod -R 755 /user/oozie/share/lib/

2-once you have all the libs, please refresh Oozie to load the lib list.

su oozie

oozie admin -oozie http://localhost:11000/oozie -sharelibupdate

oozie admin -oozie http://localhost:11000/oozie -shareliblist spark*

Hive

1-If you don't install Atlas or you are not using it , you can can remore the hooks

Go to Hive -> Advanced -> General - replaced these by space

hive.exec.failure.hooks

hive.exec.post.hooks :

hive.exec.pre.hooks :

2-If you are using INSERT OVERWRITE queries and you are getting java.util.ConcurrentModificationException I have found a workaround :

add "SORT BY 0" to your query.

PS : This is a temporary solution to allow your prod jobs running.

3-allow impersonation for hcat

hadoop.proxyuser.hcat.hosts =*

If you have any question regarding the above, please let me know

more to follow up ...

559 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎01-20-2017 05:44 PM
Updated by:
 
Contributors
Top Kudoed Authors