Exception while reading parcel: CDH-5.10.2-1

[#1]

I'm working to build a Virtualised 3 node test environment using:

Virtualbox

Centos 7.3 Minimal build.

Anaconda 4.2.0

Spark2. 2.2.0

CDH 5.10.2-1

Java  jdk1.8.0_144

Plenty of Disk space

IP6 disabled

SELinux  Permissive

Using a local repository for parcel files

My build and deploy process fails when distributing/unpacking the CDH parcel. Anaconda and Spark distribute , install and activate with no problems. Checking the Agent log files I see the following.

[13/Apr/2018 12:52:23 +0000] 16047 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/CDH-5.10.2-1.cdh5.10.2.p0.5-el7.parcel/CDH-5.10.2-1.cdh5.10.2.p0.        5-el7.parcel into /opt/cloudera/parcels

[13/Apr/2018 12:52:56 +0000] 16047 MainThread parcel       INFO     Loading parcel manifest for: Anaconda-4.2.0

[13/Apr/2018 12:52:56 +0000] 16047 MainThread parcel       INFO     Loading parcel manifest for: SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354

[13/Apr/2018 12:52:56 +0000] 16047 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.10.2-1.cdh5.10.2.p0.5

[13/Apr/2018 12:52:56 +0000] 16047 MainThread parcel       ERROR    Exception while reading parcel: CDH-5.10.2-1.cdh5.10.2.p0.5

Traceback (most recent call last):

  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.2-py2.7.egg/cmf/parcel.py", line 121, in refresh

    fd = open(manifest)

IOError: [Errno 2] No such file or directory: u'/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/meta/parcel.json'

[13/Apr/2018 12:53:31 +0000] 16047 Thread-13 parcel_cache INFO     Unpack of parcel /opt/cloudera/parcels/.flood/CDH-5.10.2-1.cdh5.10.2.p0.5-el7.parcel/CDH-5.10.2-1.cdh5.10.2.p0.5-el7.parcel successful

Telling me the the parcel.json file is missing/not accessible, but it is avail when I check:

[vagrant@node0 meta]$ ls -l /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/meta

total 7776

-rw-r--r--. 1 root root   11812 Jun 27  2017 alternatives.json

-rw-r--r--. 1 root root    1962 Jun 27  2017 cdh_env.sh

-rw-r--r--. 1 root root 7598084 Jun 27  2017 filelist.json

-rw-r--r--. 1 root root  316053 Jun 27  2017 notices.txt

-rw-r--r--. 1 root root   17730 Jun 27  2017 parcel.json

-rw-r--r--. 1 root root    2815 Jun 27  2017 permissions.json

[vagrant@node0 meta]$ namei -m /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/meta/parcel.json

f: /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/meta/parcel.json

 dr-xr-xr-x /

 drwxr-xr-x opt

 drwxr-xr-x cloudera

 lrwxrwxrwx parcels -> /data/parcels

   dr-xr-xr-x /

   drwxrwxrwx data

   drwxr-xr-x parcels

 drwxr-xr-x CDH-5.10.2-1.cdh5.10.2.p0.5

 drwxr-xr-x meta

 -rw-r--r-- parcel.json

The last line of the agent log appears to say that the unpack of the parcel has completed after the check for the parcel.json file so of course it will fail.

If I then go and restart the cloudera-scm-agent the install process can see the parcel.json file and then completes.

How do I trigger the agent to recheck for the parcel file?

[#2]

I am running CM version 6.1.1

I encountered same issue. this is looks like a bug from Cloudera parcel distribution.

you get error below before the unpack complete. As you can see from the log the error show up 2 minutes before the unpack complete successfully. I check using "watch" command the file doesn't exist when the error occur in the log.  This should be a simple logic to make sure the unpack complete before reading the parcel.

IOError: [Errno 2] No such file or directory: u'/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/meta/parcel.json'

[19/Jun/2020 15:02:34 +0000] 13865 Thread-13 downloader   INFO     Failed adding torrent: file:///opt/cloudera/parcel-cache/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel.torrent Already present torrent: CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel
[19/Jun/2020 15:02:34 +0000] 13865 Thread-13 downloader   INFO     Current state: CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel [totalDownloaded=2043680175 totalSize=2043680175 upload=0 state=seeding seed=[] location=/opt/cloudera/parcels/.flood/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel progress=1000000]
[19/Jun/2020 15:02:34 +0000] 13865 Thread-13 downloader   INFO     Completed download of https://xxxxx.xxxx.com:7183/cmf/parcel/download/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel code=200 state=downloaded
[19/Jun/2020 15:02:34 +0000] 13865 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
[19/Jun/2020 15:02:34 +0000] 13865 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel into /opt/cloudera/parcels
[19/Jun/2020 15:02:49 +0000] 13865 MainThread throttling_logger INFO     There is already an active download for https://xxxxx.xxx.xxxxx.com:7183/cmf/parcel/download/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel
[19/Jun/2020 15:05:04 +0000] 13865 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.8.4-1.cdh5.8.4.p0.5
[19/Jun/2020 15:05:04 +0000] 13865 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.12.1-1.cdh5.12.1.p0.3
[19/Jun/2020 15:05:04 +0000] 13865 MainThread parcel       INFO     Loading parcel manifest for: CDH-6.1.1-1.cdh6.1.1.p0.875250
[19/Jun/2020 15:05:04 +0000] 13865 MainThread parcel       ERROR    Exception while reading parcel: CDH-6.1.1-1.cdh6.1.1.p0.875250
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.6/site-packages/cmf/parcel.py", line 114, in refresh
    fd = open(manifest)
IOError: [Errno 2] No such file or directory: u'/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/meta/parcel.json'
[19/Jun/2020 15:07:17 +0000] 13865 Thread-13 parcel_cache INFO     Unpack of parcel /opt/cloudera/parcels/.flood/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel/CDH-6.1.1-1.cdh6.1.1.p0.875250-el6.parcel successful

[#3]

@Daddy,

This is a known issue in CM when either unpacking or moving of parcel directory is slow.

When CM unpacks the parcel file, it performs below steps:

1. untar the parcel file under /opt/cloudera/parcel-cache
2. then "move" the parcel files to /opt/cloudera/parcels

If the I/O is slow on the FS that holds directory /opt/cloudera/parcel-cache, the untar command will be slow. If the directories /opt/cloudera/parcel-cache and /opt/cloudera/parcels are mounted on different mount points, then the "move" command will become "copy" and then "delete", so operation will be much slower than the actual "move" which only involves rename.

Since CM uses different threads to perform operations, it can happen that the checking happens before the untar operation finishes, hence you hit the issue.

Can you please check if above is the issue in your case?

So you can perform the same steps I mentioned:

1. manually untar parcel under /opt/cloudera/parcel-cache
2. run "mv" command to move directory from /opt/cloudera/parcel-cache to /opt/cloudera/parcels and time it

If either of the above is slow, then you have found your issue.

Cheers
Eric

[#4]

Hi Eric @EricL ,

I manually untar and move the directory from /parcel-cache/ to /parcels and it took 2min 56 seconds. The transfer rate is around 16.8 MB/s. I don't know why this is slow for cloudera manager?

Does CM have any setting to say copy expected to complete in given time frame?
As this known issue at CM, any workaround to this?

[root@rbalvhadoo01 ~]# time mv /opt/cloudera/parcel-cache/CDH-6.1.1-1.cdh6.1.1.p0.875250 /opt/cloudera/parcels/

real 2m56.169s
user 0m0.565s
sys 0m7.708s

[#5]

Hi @ram76,

Any reason why the mv is slow? Are they on different mount points? We did see issues around 2-3 minutes delay, but unfortunately that I am not aware of a workaround. You should focus on finding out why the "move" command took this long.

In my test, it took 2 minutes to copy:

time cp -r /opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p3573.3750 /opt/cloudera/parcel-cache/5.13.3

real 1m59.830s
user 0m0.126s
sys 0m5.621s

But my "mv' was instant:

time mv /opt/cloudera/parcel-cache/5.13.3 /opt/cloudera/parcels/5.13.3

real 0m0.138s
user 0m0.000s
sys 0m0.002s

Can you please share the output of :

df -h

So we can see the disk break down?

Thanks
Eric

[#6]

Hi @EricL ,

My cluster running on VM and it is sandbox and we are not expecting very high I/O throughput.

I created softlink parcel-cache into /opt/cloudera/parcels/parcel-cache as /opt/cloudera/parcels different mountpoint with /opt/cloudera/parcel-cache. I tested the mv statement complete instantaneously and parcel distribution phase completed. 

This is just my workaround. 

Hope cloudera can fix this issue so thread wait until the unpack and copy complete successfully before reading the parcel or have longer timeout or set it as configurable parameter.

Thank you very much for your help

rgds,

Rama.