Created 10-01-2017 05:58 PM
After my sandbox reset
docker stop sandbox docker rm sandbox
I've got error:
Diagnostics: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1875268269-172.17.0.2-1493988757398:blk_1073742717_1896 file=/hdp/apps/2.6.0.3-8/spark2/spark2-hdp-yarn-archive.tar.gz
I've checked from Ambari UI, Ambari also raise error (500). Where I can download the file ?
Created 10-02-2017 02:23 AM
See if you can find this file in your local file system under /usr/hdp/current/spark2/spark2-client. If it is there then you can copy it from this location to hdfs using following command (remove existing corrupted file first).
hdfs dfs -rm hdp/apps/2.6.0.3-8/spark2/spark2-hdp-yarn-archive.tar.gz
hdfs dfs -put /usr/hdp/current/spark2/spark2-client/<file name> /hdp/apps/2.6.0.3-8/spark2
Created 10-02-2017 07:25 AM
Hi @Triffids G,
As mentioned by @mqureshi, remove the corrupted file first by running
hdfs dfs -rm hdp/apps/2.6.0.3-8/spark2/spark2-hdp-yarn-archive.tar.gz
You can build the proper tar file from the existing jars
cd /usr/hdp/2.6.0.3-8/spark2/jars # create tar file from existing jars tar -czvf spark2-hdp-yarn-archive.tar.gz * # put the new tar file in hdfs hdfs dfs -put spark2-hdp-yarn-archive.tar.gz /hdp/apps/2.6.0.3-8/spark2
Thanks,
Aditya
Created 10-03-2017 07:52 AM
Thanks, adding /usr/hdp/2.6.0.3-8/spark2/jars into archive solved the issue. Spark jobs is running OK now.
Created 10-04-2017 06:14 AM
Can you please accept the answer. This will be helpful for the users to directly check the answer instead of reading the whole thread.
Thanks,
Aditya