Looks, it's tez issue comes from "fs.permissions.umask-mode" setting. https://community.hortonworks.com/questions/246302/hive-tez-vertex-failed-error-during-reduce-phase-h.html ... View more

I'm having same issue with HDP3.1 (Tez 0.9.1). I can reproduce it with: 1) create two files - file1.csv and file2.csv 2) add two fields to the csv files as below one,two one,two one,two 3) create external table use testdb; create external table test1(s1 string, s2 string) row format delimited fields terminated by ',' stored as textfile location '/user/usera/test1'; 4) Copy one csv file to hdfs - /user/usera/test1 hdfs dfs -put ./file1.csv /user/usera/test1/ 5) select count(*) from testdb.test1; => works fine. 6) copy the second csv file to HDFS hdfs dfs -put ./file2.csv /user/usera/test1/ 7) select * from testdb.test1; => Can see the data in both hdfs files. 8) select count(*) form testdb.test1; => Get this problem. And we can see following error in the mapper task's log. 2019-05-17 10:08:10,317 [INFO] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to read data to memory for InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1]. len=25, decomp=11. ExceptionMessage=Not a valid ifile header 2019-05-17 10:08:10,317 [WARN] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1] from XXXXX java.io.IOException: Not a valid ifile header at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.verifyHeaderMagic(IFile.java:859) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.isCompressedFlagEnabled(IFile.java:866) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:616) at org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:121) at org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:950) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:599) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:486) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:284) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:76) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I think, it's similar to https://issues.apache.org/jira/browse/TEZ-3699 I've confirmed the patch already applied to tez with HDP 3.1. So I guess, it's new bug with Tez 0.9.x (I confirmed there is no problem with HDP2.6/Tez 0.7.0). Any idea? ... View more

One update. Hive seems to have an issue about handling "viewfs://"? For example, hive db creation got failed with permission error even if the permission of the target hdfs dir is 777. HDFS permission: hdfs dfs -ls -d viewfs://fed/user/hadoop/warehouse/hadoop_viewfs6.db drwxrwxrwx - hadoop hadoop 0 2019-04-24 07:44 viewfs://fed/user/hadoop/warehouse/hadoop_viewfs6.db Hive DB creation got failed with the hdfs dir. 0: jdbc:hive2://XXXX > create database hadoop_viewfs6 location '/user/hadoop/warehouse/hadoop_viewfs6.db'; Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [hadoop] does not have [WRITE] privilege on [viewfs://fed/user/hadoop/warehouse/hadoop_viewfs6.db] (state=42000,code=40000 Succeeded with "hdfs://" 0: jdbc:hive2://XXXX > create database hadoop_viewfs6 location 'hdfs://ns1/user/hadoop/warehouse/hadoop_viewfs6.db'; ... View more

Hi, I'm facing completely same issue with HDP2.6.2. HDFS client have to wait about 20 seconds when the 1st NameNode is powered off. (Actually, we had this issue then the 1st NameNode had kernel hang (kernel panic).) Do you find good solution or workaround??? If so, please share it. Any information will help us ! ... View more

Hello, we are getting totally same problem with HDP2.6.2. Could someone confirm https://issues.apache.org/jira/browse/RANGER-1136 is included??? I've checked source, but I couldn't... ... View more

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_command-line-installation/content/determine-hdp-memory-config.html This page also says "CORES (number of CPU cores)". So, "CORES" means "physical cores"? If it's true, 12 physical CPU cores (24 vcores by Intel HT) may be good enough for a node with 12 HDDs. I'm welcome any opinions from all of you 🙂 ... View more

Seems, we can create new ranger service with any account with admin permission in Ranger Admin with password such as "admin" on Ranger Admin UI. ... View more

Hi, I'm installing ranger with following docs because we are not using Ambari. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/ch_installing_ranger_chapter.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/installing_ranger_kerberized_environment.html So, I've like to know how we set for this "Username" and "Password" on "Create Service" without Ambari with Kerberos. In my test, hdfs (HDFS superuser) worked fine by creating hdfs account with Ranger Admin UI with a password in lab cluster. Can we create a new Service with an account without password? ... View more

I had same problem "ranger can sync users with ldap but can't login to ranger UI with ldap password." Finally I could solved this problem, so, let me share lessons learned and how I solved to help you guys who has same problem as I faced. lessons learned 1. We have to configure ranger admin to speak ldaps protocol if we want to use ldaps for user authentication. paramaters in ranger-admin-site: ranger.truststore.file, ranger.truststore.password I had to import self-signed CA from LDAP team to "/etc/ranger/admin/conf/ranger-admin-keystore.jks". Set password which I specified for this import to "ranger.truststore.password". Command example: keytool -importcert -alias rangeradmin -noprompt -trustcacerts -file ./ca.crt -keystore /etc/ranger/admin/conf/ranger-admin-keystore.jks -storepass xasecure ref: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/configure_non_ambari_ranger_ssl_self_signed_cert_admin.html 2. setting syncing user info with ldap and setting to use ldap for authentication are technically different. For example, we can use ldap authentication for ranger UI login even when we disable "Enable User Sync". In other words, we can use ldap authentication when "Ranger Usersync" service is not running. 3. debug logs from "org.springframework" and "org.apache.ranger" were very useful for the trouble shooting. We can change the log level with "admin-log4j.xml". log4j.category.org.springframework=debug,xa_log_appender log4j.category.org.apache.ranger=debug,xa_log_appender 4. Here are the key configurations for ldap authentication (not for user sync with ldap.) Authentication method: LDAP LDAP URL: ldaps://xxxxxx User Search Filter: (uid={0}) Group Search Filter: (member=uid={0},ou=xxxxx,o=xxxxx) ranger.ldap.user.dnpattern: uid={0},ou=xxxxx,o=xxxxx ranger.truststore.file: /etc/ranger/admin/conf/ranger-admin-keystore.jks <= in case with ldaps. ranger.truststore.password: xasecure <= in case with ldaps. this is the passwoed you set when you import ca to jks. I hope, this memo help guys who have same problem as I faced 🙂 ... View more

Jay SenSharma Thank you for replying. Our NameNode manages over 300 Millions hdfs files and dirs. So, the table above says, we need more than "104473m" heap size for "150-200 Millions". And If number of hdfs files and dirs become 600 Millions, we would need about 300 GB (100GB x 3) memory for NameNode. > The JVM should not complain if the heap size is set to a larger value (until we have enough RAM available) So, you mean, we can set 300GB for java heap size, and the NameNode or the JVM for the NameNode should not complain if the node has enough physical memory, for example, 500 GB, right? ... View more

Joseph Niemiec , Alberto Ramon thank you for your replying. About full GC, we use CMS GC for these NameNodes, it takes about 2 mins in total, but the STW GC ("Initial Mark" and "Final Remark") is less than a few second in total. So, we don't worry about the GC issue for now 🙂 Yes, we are trying to use HDFS federation with nn3 and nn4 for second nameservice now to reduce heap usage of the first set of namenodes (nn1 and nn2) for the first nameservice. (Support HDFS Federation, please...) ... View more

Sorry, There was copy and paste miss in question above. Now we use "-XX:CMSInitiatingOccupancyFraction=90" for current NameNode with CMS. ... View more