Thank you for the great insights These are the disctp options distcp -skipcrccheck -copybuffersize 32768 -update -pugp -bandwidth 50 -strategy dynamic and yes, total execution time increases almost linearly with nr of files/directories ... View more

Based on the error shared, i see CM is 7.11.3 and CDP runtime is 7.1.7. There is a known bug as is discussed in the following documentation: Link: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/manager-release-notes/topics/cm-known-issues-7113.html#:~:text=OPSAPS%2D69357%3A%20Python%20incompatibility%20issues%20when%20Cloudera%20Manager%20(Python%203.x%20compatible)%20manages%20a%20cluster%20with%20Cloudera%20Runtime%207.1.7%20(Python%202%20compatible)   OPSAPS-69357: Python incompatibility issues when Cloudera Manager (Python 3.x compatible) manages a cluster with Cloudera Runtime 7.1.7 (Python 2 compatible) If Cloudera Manager is compatible with Python 3, then scripts that are packaged with this Cloudera Manager are also ported to Python 3 syntax. So, using Cloudera Manager (7.11.3 or any other Cloudera Manager version ported to Python 3.x version) to manage a cluster with Cloudera Runtime 7.1.7 (Python 2 compatible) would cause Python incompatibility issues because the process assumes Python 2 environment but the scripts that are packaged with this Cloudera Manager are ported to Python 3 syntax.   ... View more