HDP upgrade 4.2.2 to 2.5.3 -Challenges faced and solutions

[#1]

During the upgrade :

if you are using postgres DB, you will face a script failure on the following statement

Alter table users add constraint "uniq_user_0"

Solution :

psql -U ambari_user -d NN_HA_ambari

alter table ambari.user drop constraint "uniq_user_0"

After the Upgrade :

Spark

1-you have to recompile and rebuild your jars using the new dependencies of Spark 1.6.2 Your pom.xml needs to have the new versions ( new jars of HDP 2.5.3 )

2-Update any custom jars and check all versions used in your Pom.xml

i.e

<spark.version>1.6.2</spark.version>
<hbase.version>1.2.1</hbase.version>

3- Remove the argument spark.yarn.jar if you are using it when submitting the job.

--conf spark.yarn.jar=hdfs://NN_HA/user/oozie/share/lib/lib_20160515013357/spark/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar.

Error noticed when leaving the arg above :

java.lang.IllegalArgumentException: Invalid ContainerId: container_e97_1484824056011_0038_02_000001

or

java.lang.IllegalArgumentException

Spark - Oozie

1- if you are launching Spark via Oozie ( or via Hue ) chek you get all the libs here with the right permissions :

-rwxr-xr-x   3 oozie hdfs     339666 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/hbase-site.xml
-rwxr-xr-x   3 oozie hdfs     339666 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/datanucleus-api-jdo-3.2.6.jar
-rwxr-xr-x   3 oozie hdfs    1890075 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/spark/datanucleus-core-3.2.10.jar
-rwxr-xr-x   3 oozie hdfs    1809447 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/spark/datanucleus-rdbms-3.2.9.jar
-rwxr-xr-x   3 oozie hdfs      22440 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/spark/oozie-sharelib-spark-4.2.0.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs      44846 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/py4j-0.9-src.zip
-rwxr-xr-x   3 oozie hdfs     357563 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/pyspark.zip
-rwxr-xr-x   3 oozie hdfs  188897932 2017-01-19 10:22 /user/oozie/share/lib/lib_20170119102144/spark/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs  110488188 2017-01-19 17:43 /user/oozie/share/lib/lib_20170119102144/spark/spark-examples-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs  188897932 2017-01-19 17:43 /user/oozie/share/lib/lib_20170119102144/spark/spark-hdp-assembly.jar
-rwxr-xr-x   3 oozie hdfs     516062 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/aws-java-sdk-core-1.10.6.jar
-rwxr-xr-x   3 oozie hdfs     258578 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/aws-java-sdk-kms-1.10.6.jar
-rwxr-xr-x   3 oozie hdfs     570101 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/aws-java-sdk-s3-1.10.6.jar
-rwxr-xr-x   3 oozie hdfs      10092 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/azure-keyvault-core-0.8.0.jar
-rwxr-xr-x   3 oozie hdfs     745325 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/azure-storage-4.2.0.jar
-rwxr-xr-x   3 oozie hdfs     434678 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/commons-lang3-3.4.jar
-rwxr-xr-x   3 oozie hdfs    1648200 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/guava-11.0.2.jar
-rwxr-xr-x   3 oozie hdfs     153855 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/hadoop-aws-2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs     163348 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/hadoop-azure-2.7.3.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs      38605 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/jackson-annotations-2.4.0.jar
-rwxr-xr-x   3 oozie hdfs     225302 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/jackson-core-2.4.4.jar
-rwxr-xr-x   3 oozie hdfs    1076926 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/jackson-databind-2.4.4.jar
-rwxr-xr-x   3 oozie hdfs     570478 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/joda-time-2.1.jar
-rwxr-xr-x   3 oozie hdfs      16046 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/json-simple-1.1.jar
-rwxr-xr-x   3 oozie hdfs      12543 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/oozie-hadoop-utils-hadoop-2-4.2.0.2.5.3.0-37.jar
-rwxr-xr-x   3 oozie hdfs      51854 2017-01-19 10:21 /user/oozie/share/lib/lib_20170119102144/oozie/oozie-sharelib-oozie-4.2.0.2.5.3.0-37.jar

PS : here we have 755 can be restricted more

hdfs dfs -chmod -R 755 /user/oozie/share/lib/

2-once you have all the libs, please refresh Oozie to load the lib list.

su oozie

oozie admin -oozie http://localhost:11000/oozie -sharelibupdate

oozie admin -oozie http://localhost:11000/oozie -shareliblist spark*

Hive

1-If you don't install Atlas or you are not using it , you can can remore the hooks

Go to Hive -> Advanced -> General - replaced these by space

hive.exec.failure.hooks

hive.exec.post.hooks :

hive.exec.pre.hooks :

2-If you are using INSERT OVERWRITE queries and you are getting java.util.ConcurrentModificationException I have found a workaround :

add "SORT BY 0" to your query.

PS : This is a temporary solution to allow your prod jobs running.

3-allow impersonation for hcat

hadoop.proxyuser.hcat.hosts =*

If you have any question regarding the above, please let me know

more to follow up ...