Member since
05-11-2016
42
Posts
2
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1916 | 02-07-2018 06:22 AM | |
1549 | 11-13-2017 08:04 AM | |
1725 | 07-20-2017 03:01 AM |
05-22-2019
05:54 PM
I had same issue, and we are using HDP 3.1.0.0-78 . https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_tez.html TEZ-3894 seems to be already applied to HDP 3.1. (Also, I've checked the source code a little, yes, it looks already applied.) But I still have this issue... I can avoid this issue by changing fs.permissions.umask-mode from "077" to "022" in a HS2 session. 0: jdbc:hive2://XXXX > set fs.permissions.umask-mode=022; So I guess, this issue may not be fixed completely with TEZ-3894 (with HDP 3.1.0.0-78)...
... View more
05-22-2019
04:27 AM
Looks, it's tez issue comes from "fs.permissions.umask-mode" setting. https://community.hortonworks.com/questions/246302/hive-tez-vertex-failed-error-during-reduce-phase-h.html
... View more
05-21-2019
03:44 AM
I'm having same issue with HDP3.1 (Tez 0.9.1). I can reproduce it with: 1) create two files - file1.csv and file2.csv
2) add two fields to the csv files as below
one,two
one,two
one,two
3) create external table
use testdb;
create external table test1(s1 string, s2 string) row format delimited fields terminated by ',' stored as textfile location '/user/usera/test1';
4) Copy one csv file to hdfs - /user/usera/test1
hdfs dfs -put ./file1.csv /user/usera/test1/
5) select count(*) from testdb.test1;
=> works fine.
6) copy the second csv file to HDFS
hdfs dfs -put ./file2.csv /user/usera/test1/
7) select * from testdb.test1;
=> Can see the data in both hdfs files.
8) select count(*) form testdb.test1;
=> Get this problem. And we can see following error in the mapper task's log. 2019-05-17 10:08:10,317 [INFO] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to read data to memory for InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1]. len=25, decomp=11. ExceptionMessage=Not a valid ifile header 2019-05-17 10:08:10,317 [WARN] [Fetcher_B {Map_1} #1] |shuffle.Fetcher|: Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=1, attemptNumber=0, pathComponent=attempt_1557383221332_0289_1_00_000001_0_10003, spillType=0, spillId=-1] from XXXXX java.io.IOException: Not a valid ifile header at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.verifyHeaderMagic(IFile.java:859) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.isCompressedFlagEnabled(IFile.java:866) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:616) at org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:121) at org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:950) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:599) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:486) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:284) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:76) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I think, it's similar to https://issues.apache.org/jira/browse/TEZ-3699 I've confirmed the patch already applied to tez with HDP 3.1. So I guess, it's new bug with Tez 0.9.x (I confirmed there is no problem with HDP2.6/Tez 0.7.0). Any idea?
... View more
03-29-2019
09:32 AM
1 Kudo
I had same issue and solved by changing "yarn.nodemanager.resource.memory-mb" from 468GB to 200GB with Amabri (There were following 11 changes.) (I shared it because it's really hard to find out the reason by reading error messages...)
... View more
09-18-2018
08:36 AM
Hi, I'm facing completely same issue with HDP2.6.2. HDFS client have to wait about 20 seconds when the 1st NameNode is powered off. (Actually, we had this issue then the 1st NameNode had kernel hang (kernel panic).) Do you find good solution or workaround??? If so, please share it. Any information will help us !
... View more
02-07-2018
06:22 AM
I don't think so.
... View more
02-07-2018
06:19 AM
I had same problem "ranger can sync users with ldap but can't login to ranger UI with ldap password." Finally I could solved this problem, so, let me share lessons learned and how I solved to help you guys who has same problem as I faced. lessons learned 1. We have to configure ranger admin to speak ldaps protocol if we want to use ldaps for user authentication. paramaters in ranger-admin-site: ranger.truststore.file, ranger.truststore.password I had to import self-signed CA from LDAP team to "/etc/ranger/admin/conf/ranger-admin-keystore.jks". Set password which I specified for this import to "ranger.truststore.password". Command example: keytool -importcert -alias rangeradmin -noprompt -trustcacerts -file ./ca.crt -keystore /etc/ranger/admin/conf/ranger-admin-keystore.jks -storepass xasecure ref: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/configure_non_ambari_ranger_ssl_self_signed_cert_admin.html 2. setting syncing user info with ldap and setting to use ldap for authentication are technically different. For example, we can use ldap authentication for ranger UI login even when we disable "Enable User Sync". In other words, we can use ldap authentication when "Ranger Usersync" service is not running. 3. debug logs from "org.springframework" and "org.apache.ranger" were very useful for the trouble shooting. We can change the log level with "admin-log4j.xml". log4j.category.org.springframework=debug,xa_log_appender log4j.category.org.apache.ranger=debug,xa_log_appender 4. Here are the key configurations for ldap authentication (not for user sync with ldap.) Authentication method: LDAP LDAP URL: ldaps://xxxxxx User Search Filter: (uid={0}) Group Search Filter: (member=uid={0},ou=xxxxx,o=xxxxx) ranger.ldap.user.dnpattern: uid={0},ou=xxxxx,o=xxxxx ranger.truststore.file: /etc/ranger/admin/conf/ranger-admin-keystore.jks <= in case with ldaps. ranger.truststore.password: xasecure <= in case with ldaps. this is the passwoed you set when you import ca to jks. I hope, this memo help guys who have same problem as I faced 🙂
... View more
11-13-2017
08:04 AM
After upgrading our cluster from HDP 2.6.1 to HDP 2.6.2, the following WARN message is not output in NameNode's log. WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_xxxx_xxxx blockMap has 0 but corrupt replicas map has 1
So, the problem seems to be fixed by this upgrading in our clusters.
... View more
11-13-2017
02:05 AM
Hi, can anyone reply?
... View more
11-08-2017
04:54 AM
Sorry, I needed to say, "I also think, if we want to control security between clients and hadoop clusters only with Knox (i.e. use Knox as the security "proxy" between clients and hadoop clusters), we have to eliminate hadoop clients on Edge Node, because Knox is only for O/JDBC hadoop client."
... View more