I had same issue, and we are using HDP 3.1.0.0-78 . https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_tez.html TEZ-3894 seems to be already applied to HDP 3.1. (Also, I've checked the source code a little, yes, it looks already applied.) But I still have this issue... I can avoid this issue by changing fs.permissions.umask-mode from "077" to "022" in a HS2 session. 0: jdbc:hive2://XXXX > set fs.permissions.umask-mode=022; So I guess, this issue may not be fixed completely with TEZ-3894 (with HDP 3.1.0.0-78)... ... View more

Looks, it's tez issue comes from "fs.permissions.umask-mode" setting. https://community.hortonworks.com/questions/246302/hive-tez-vertex-failed-error-during-reduce-phase-h.html ... View more

I'm having same issue with HDP3.1 (Tez 0.9.1). I can reproduce it with: 1) create two files - file1.csv and file2.csv 2) add two fields to the csv files as below one,two one,two one,two 3) create external table use testdb; create external table test1(s1 string, s2 string) row format delimited fields terminated by ',' stored as textfile location '/user/usera/test1'; 4) Copy one csv file to hdfs - /user/usera/test1 hdfs dfs -put ./file1.csv /user/usera/test1/ 5) select count(*) from testdb.test1; => works fine. 6) copy the second csv file to HDFS hdfs dfs -put ./file2.csv /user/usera/test1/ 7) select * from testdb.test1; => Can see the data in both hdfs files. 8) select count(*) form testdb.test1; => Get this problem. And we can see following error in the mapper task's log. 2019-05-17 10:08:10,317 [INFO] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to read data to memory for InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1]. len=25, decomp=11. ExceptionMessage=Not a valid ifile header 2019-05-17 10:08:10,317 [WARN] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1] from XXXXX java.io.IOException: Not a valid ifile header at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.verifyHeaderMagic(IFile.java:859) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.isCompressedFlagEnabled(IFile.java:866) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:616) at org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:121) at org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:950) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:599) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:486) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:284) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:76) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I think, it's similar to https://issues.apache.org/jira/browse/TEZ-3699 I've confirmed the patch already applied to tez with HDP 3.1. So I guess, it's new bug with Tez 0.9.x (I confirmed there is no problem with HDP2.6/Tez 0.7.0). Any idea? ... View more

I had same issue and solved by changing "yarn.nodemanager.resource.memory-mb" from 468GB to 200GB with Amabri (There were following 11 changes.) (I shared it because it's really hard to find out the reason by reading error messages...) ... View more

Hi, I'm facing completely same issue with HDP2.6.2. HDFS client have to wait about 20 seconds when the 1st NameNode is powered off. (Actually, we had this issue then the 1st NameNode had kernel hang (kernel panic).) Do you find good solution or workaround??? If so, please share it. Any information will help us ! ... View more

I had same problem "ranger can sync users with ldap but can't login to ranger UI with ldap password." Finally I could solved this problem, so, let me share lessons learned and how I solved to help you guys who has same problem as I faced. lessons learned 1. We have to configure ranger admin to speak ldaps protocol if we want to use ldaps for user authentication. paramaters in ranger-admin-site: ranger.truststore.file, ranger.truststore.password I had to import self-signed CA from LDAP team to "/etc/ranger/admin/conf/ranger-admin-keystore.jks". Set password which I specified for this import to "ranger.truststore.password". Command example: keytool -importcert -alias rangeradmin -noprompt -trustcacerts -file ./ca.crt -keystore /etc/ranger/admin/conf/ranger-admin-keystore.jks -storepass xasecure ref: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/configure_non_ambari_ranger_ssl_self_signed_cert_admin.html 2. setting syncing user info with ldap and setting to use ldap for authentication are technically different. For example, we can use ldap authentication for ranger UI login even when we disable "Enable User Sync". In other words, we can use ldap authentication when "Ranger Usersync" service is not running. 3. debug logs from "org.springframework" and "org.apache.ranger" were very useful for the trouble shooting. We can change the log level with "admin-log4j.xml". log4j.category.org.springframework=debug,xa_log_appender log4j.category.org.apache.ranger=debug,xa_log_appender 4. Here are the key configurations for ldap authentication (not for user sync with ldap.) Authentication method: LDAP LDAP URL: ldaps://xxxxxx User Search Filter: (uid={0}) Group Search Filter: (member=uid={0},ou=xxxxx,o=xxxxx) ranger.ldap.user.dnpattern: uid={0},ou=xxxxx,o=xxxxx ranger.truststore.file: /etc/ranger/admin/conf/ranger-admin-keystore.jks <= in case with ldaps. ranger.truststore.password: xasecure <= in case with ldaps. this is the passwoed you set when you import ca to jks. I hope, this memo help guys who have same problem as I faced 🙂 ... View more

Hi, can anyone reply? ... View more

Sorry, I needed to say, "I also think, if we want to control security between clients and hadoop clusters only with Knox (i.e. use Knox as the security "proxy" between clients and hadoop clusters), we have to eliminate hadoop clients on Edge Node, because Knox is only for O/JDBC hadoop client." ... View more