Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH5.5.3: Checksum mismatch on parcel distribution

CDH5.5.3: Checksum mismatch on parcel distribution

New Contributor

I'm trying to configure a new CDH5.5.2 cluster using CM 5.5.3, and I got continuous "Checksum mismatch" error on parcel distribution process during the installation wizard.
(All hosts are CentOS 7.1)


Agents' log shows that they calculated different checksum on each retry.

 

[22/Feb/2016 18:10:17 +0000] 3298 Thread-13 downloader   INFO     Finished download [ url: http://dev-hdp-cm201.my.local:7180/cmf/parcel/download/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel, state: exception, total_bytes: 1450162955, downloaded_bytes: 1064017920, start_time: 2016-02-22 18:06:53, download_end_time: 2016-02-22 18:10:14, end_time: 2016-02-22 18:10:17, code: 601, exception_msg: Checksum mismatch, path: /opt/cloudera/parcel-cache/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel ]
[22/Feb/2016 18:13:44 +0000] 3298 Thread-13 parcel_cache INFO     Checking checksum of parcel CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel...
[22/Feb/2016 18:13:47 +0000] 3298 Thread-13 parcel_cache WARNING  Parcel CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel did not match checksum from http://dev-hdp-cm201.my.local:7180/cmf/parcel/download/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel: header 19b7fdbb450894f8b2687be74a7d889eb3829105 != calculated 9d3291dcb71146eb7412b55cae1e59eabf328a95.
[22/Feb/2016 18:13:47 +0000] 3298 Thread-13 downloader   INFO     Finished download [ url: http://dev-hdp-cm201.my.local:7180/cmf/parcel/download/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel, state: exception, total_bytes: 1450162955, downloaded_bytes: 1055858688, start_time: 2016-02-22 18:10:24, download_end_time: 2016-02-22 18:13:44, end_time: 2016-02-22 18:13:47, code: 601, exception_msg: Checksum mismatch, path: /opt/cloudera/parcel-cache/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel ]
[22/Feb/2016 18:17:50 +0000] 3298 Thread-13 parcel_cache INFO     Checking checksum of parcel CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel...
[22/Feb/2016 18:17:52 +0000] 3298 Thread-13 parcel_cache WARNING  Parcel CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel did not match checksum from http://dev-hdp-cm201.my.local:7180/cmf/parcel/download/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel: header 19b7fdbb450894f8b2687be74a7d889eb3829105 != calculated a5748ca79317eb0c13ace2591b20ae22c3b91372.
[22/Feb/2016 18:17:52 +0000] 3298 Thread-13 downloader   INFO     Finished download [ url: http://dev-hdp-cm201.my.local:7180/cmf/parcel/download/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel, state: exception, total_bytes: 1450162955, downloaded_bytes: 451436544, start_time: 2016-02-22 18:13:55, download_end_time: 2016-02-22 18:17:50, end_time: 2016-02-22 18:17:52, code: 601, exception_msg: Checksum mismatch, path: /opt/cloudera/parcel-cache/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel ]
[22/Feb/2016 18:21:15 +0000] 3298 Thread-13 parcel_cache INFO     Checking checksum of parcel CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel...
[22/Feb/2016 18:21:16 +0000] 3298 Thread-13 parcel_cache WARNING  Parcel CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel did not match checksum from http://dev-hdp-cm201.my.local:7180/cmf/parcel/download/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel: header 19b7fdbb450894f8b2687be74a7d889eb3829105 != calculated ffb7d8edd29f7ebd9ff3957d574700938bfa4097.

 

When I lowered the maximum parcel upload number to 1, the error disappeared.
Is there issue regarding concurrency on Cloudera Manager Server?

 

5 REPLIES 5

Re: CDH5.5.3: Checksum mismatch on parcel distribution

New Contributor

I have tried again.

 

  • When I set maximum parcel upload >= 2, parcel distribution fails everytime with "Checksum Mismatch".
  • The error doesn't occur when I set the configuration to 1.
  • The same error reproduces even after I downgraded cloudera-manager to 5.5.1-1.cm551.p0.8.el7.
  • The same problem has not occurred on another cluster with CM 5.5.1 and CentOS 6.6.


I would like to know whether this error is caused by my system configuration, or by potential bug of CM 5.5, but I ran out of ideas of what I should check for.

 

Any suggestion about this issue (or a clarification that it would not affect cluster stability) would be appreciated.

Re: CDH5.5.3: Checksum mismatch on parcel distribution

New Contributor

This might be a network stability issues , Can you try to create 1 GB file and copy to all nodes at same time with scp?

 

Also check MTU value of network card.

 

ifconfig

 

 

then you can see MTU value and try to reduce if it is 9000 to 1500 and try.

Re: CDH5.5.3: Checksum mismatch on parcel distribution

New Contributor

ianeeshps,

 

Thank you for the suggestion. I have tested copying the parcel (1.4GB) from CM server host to two agent hosts at the same time.

Tried this twice, and in the both time, copy succeeded.

 

1. Prepare parcel to scp

cm-host$ sudo cp /opt/cloudera/parcel-repo/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel /tmp
cm-host$ sudo chmod +r /tmp/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel
cm-host$ LANG=C ls -lh /tmp/*.parcel
-rw-r--r--. 1 root root 1.4G Mar 11 13:25 /tmp/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel
cm-host$ sha1sum /tmp/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel
19b7fdbb450894f8b2687be74a7d889eb3829105  /tmp/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel

 

2. Scp from two agent hosts

agent-host1$ LANG=C date; scp -l 160000 cm-host:/tmp/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel .; sha1sum CDH-5.5.2-1.cdh5.5.2.p0.4-el7.pa
rcel
Fri Mar 11 13:34:57 JST 2016
CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel                                                                                            100% 1383MB  19.8MB/s   01:10
19b7fdbb450894f8b2687be74a7d889eb3829105  CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel
agent-host2$ LANG=C date; scp -l 160000 cm-host:/tmp/CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel .; sha1sum CDH-5.5.2-1.cdh5.5.2.p0.4-el7.pa
rcel
Fri Mar 11 13:34:57 JST 2016
CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel                                                                                            100% 1383MB  19.8MB/s   01:10
19b7fdbb450894f8b2687be74a7d889eb3829105  CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel

After the first posts, I have been using the cluster, but doesn't have new issue regarding concurrency.
( though sometimes some of the agents hang on distribution of new client settings) .

The useage includes DistCp-ing hundred gigabytes data with another cluster, distributing new configuration from Cloudera Manager, Staring/Stopping the cluster, and so on. 

 

 

Highlighted

Re: CDH5.5.3: Checksum mismatch on parcel distribution

Explorer

You solution to download one parcel at a time fix me issue too,

Thanks

Re: CDH5.5.3: Checksum mismatch on parcel distribution

I have the same issue and my version is CDH-5.5.1-1.cdh5.5.1.p0.11.

 

I'm using python start_distribution() method in ApiParcel

 

If you limit distribution to one node at a time, it will take significant time to complete over all hosts in a cluster.

 

Thanks

Gregory

Don't have an account?
Coming from Hortonworks? Activate your account here