Hi, I am getting the following message on ambari on every worker nodes: 1/1 local-dirs usable space is below configured utilization percentage/no more usable space [ /data/2/hadoop/yarn/local : used space above threshold of 90.0% ] ; I have also checked on the ResourceManager UI > Nodes. 6 nodes are currently on a unhealthy state. Could you please help me get them back to a healthy state? Thank you ... View more

Hello, I have currently a kerberized hadoop HDP 3.0.1 cluster where HIVESERVER2 INTERACTIVE has been deployed. It was working just fine but today... HIVESERVER2 INTERACTIVE reported the following alert on ambari: Hive LLAP Application The application reported a 'NOT RUNNING' state. Check took 7.116s. Any idea what could be the issue? Thank you for your help! ... View more

HI, I am new with YARN, I am not able to restart the nodemanagers on some of nodes. Could you please help me out? ... View more

Hi I am trying to start live the nodemanager previously decommisionned on a node but it stops soon after it starts... ERROR collector.NodeTimelineCollectorManager (NodeTimelineCollectorManager.java:startWebApp(314)) - The per-node collector webapp failed to start. java.io.IOException: Problem starting http server   any idea what could be the problem? ... View more

Hi , I have added new worker nodes to my Hadoop cluster. i get the following error on the yarn logs less hadoop-yarn-nodemanager-hd-c2-wn09.log: 2019-08-09 08:44:44,898 ERROR ssl.SSLFactory (SSLHostnameVerifier.java:check(357)) - Host check error {} javax.net.ssl.SSLException: hostname in certificate did't match: != OR Any idea how to fix this? ... View more

HI, I am trying to connect to the hive server of my kerberized HDP 3.0.1 using beeline on a external host. I have imported the keytab and also changed the /etc/krb5.conf on the external host, i am able to kinit using the keytab but i stilll get the following error when i am trying to connect to the hive server:   shell> kinit mapr/<FQDN@REALM> -k -t /home/user1/Desktop/hive.service.keytab shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>" SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/user1/Documents/HDP/apache-hive-3.1.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/user1/Documents/HDP/hadoop-3.0.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM> Java config name: null Native config name: /etc/krb5.conf Loaded from native config 19/07/29 08:49:43 [main]: ERROR transport.TSaslTransport: SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_212] at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[hive-exec-3.1.0.jar:3.1.0] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) [hive-exec-3.1.0.jar:3.1.0] at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [hive-exec-3.1.0.jar:3.1.0] at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:51) [hive-exec-3.1.0.jar:3.1.0] at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:48) [hive-exec-3.1.0.jar:3.1.0] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_212] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_212] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965) [hadoop-common-3.0.1.jar:?] at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport.open(TUGIAssumingTransport.java:48) [hive-exec-3.1.0.jar:3.1.0] at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:339) [hive-jdbc-3.1.0.jar:3.1.0] at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:224) [hive-jdbc-3.1.0.jar:3.1.0] at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-3.1.0.jar:3.1.0] at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_212] at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_212] at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.Commands.connect(Commands.java:1641) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.Commands.connect(Commands.java:1536) [hive-beeline-3.1.0.jar:3.1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212] at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1384) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1423) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:900) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:795) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1048) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:538) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.main(BeeLine.java:520) [hive-beeline-3.1.0.jar:3.1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212] at org.apache.hadoop.util.RunJar.run(RunJar.java:239) [hadoop-common-3.0.1.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:153) [hadoop-common-3.0.1.jar:?] Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_212] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_212] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_212] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_212] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_212] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_212] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_212] ... 36 more 19/07/29 08:49:43 [main]: WARN jdbc.HiveConnection: Failed to connect to <hostname>:10000 Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>: GSS initiate failed (state=08S01,code=0) Beeline version 3.1.0 by Apache Hive     NOTE: I was following the following recommendation, section Using Beeline with Kerberos: https://mapr.com/docs/archive/mapr40x/Configuring-Hive-on-a-Secure-Cluster_28869090.html#ConfiguringHiveonaSecureCluster-UsingBeelinewithKerberos   Could you please help?   Thank you ... View more

HI, I am trying to connect to the hive server of my kerberized HDP 3.0.1 using beeline on a external host. I have imported the keytab and also changed the /etc/krb5.conf on the external host, i am able to kinit using the keytab but i stilll get the following error when i am trying to connect to the hive server: shell> kinit mapr/<FQDN@REALM> -k -t /home/user1/Desktop/hive.service.keytab shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>" SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/user1/Documents/HDP/apache-hive-3.1.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/user1/Documents/HDP/hadoop-3.0.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM> Java config name: null Native config name: /etc/krb5.conf Loaded from native config 19/07/29 08:49:43 [main]: ERROR transport.TSaslTransport: SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_212] at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[hive-exec-3.1.0.jar:3.1.0] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) [hive-exec-3.1.0.jar:3.1.0] at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [hive-exec-3.1.0.jar:3.1.0] at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:51) [hive-exec-3.1.0.jar:3.1.0] at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:48) [hive-exec-3.1.0.jar:3.1.0] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_212] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_212] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965) [hadoop-common-3.0.1.jar:?] at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport.open(TUGIAssumingTransport.java:48) [hive-exec-3.1.0.jar:3.1.0] at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:339) [hive-jdbc-3.1.0.jar:3.1.0] at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:224) [hive-jdbc-3.1.0.jar:3.1.0] at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-3.1.0.jar:3.1.0] at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_212] at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_212] at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.Commands.connect(Commands.java:1641) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.Commands.connect(Commands.java:1536) [hive-beeline-3.1.0.jar:3.1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212] at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1384) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1423) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:900) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:795) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1048) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:538) [hive-beeline-3.1.0.jar:3.1.0] at org.apache.hive.beeline.BeeLine.main(BeeLine.java:520) [hive-beeline-3.1.0.jar:3.1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212] at org.apache.hadoop.util.RunJar.run(RunJar.java:239) [hadoop-common-3.0.1.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:153) [hadoop-common-3.0.1.jar:?] Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_212] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_212] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_212] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_212] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_212] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_212] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_212] ... 36 more 19/07/29 08:49:43 [main]: WARN jdbc.HiveConnection: Failed to connect to <hostname>:10000 Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>: GSS initiate failed (state=08S01,code=0) Beeline version 3.1.0 by Apache Hive NOTE: I was following the following recommendation, section Using Beeline with Kerberos: https://mapr.com/docs/archive/mapr40x/Configuring-Hive-on-a-Secure-Cluster_28869090.html#ConfiguringHiveonaSecureCluster-UsingBeelinewithKerberos Could you please help? Thank you ... View more

Hi, I am new in Hadoop(HDP 3.0), i would like to know what to do when a host/slave component is down? I believe that one of the worker node is currently down and i would like to know what would be the right procedure to turn it back up? note: It seems that according to ambari the services on that host are still running Thank you ... View more

Hi, My environment is kerberized with multple nodes. So we would like to acces hive db via knox gateway setup. I am facing some issue. the following steps: in HDFS: hadoop.proxyuser.hive.groups=* hadoop.proxyuser.hive.hosts=* hadoop.proxyuser.knox.groups=* hadoop.proxyuser.knox.hosts=* Configured the server2 transports and SASL: hive.server2.transport.mode=http hive.server2.thrift.sasl.qop=auth-conf in Ranger: Knox Topology = default Knox Service = HIVE i have extracted the knox certificat using the following cmd: shell> cd /usr/hdp/current/knox-server/data/security/keystores/ shell> keytool -exportcert -alias gateway-identity -keystore ./gateway.jks -file knox.crt Created a client keystore and imported the certificate on a client machine shell> keytool -keystore clientkeystore.jks -genkey -alias client shell> keytool -import -keystore clientkeystore.jks -file knox.crt And finally i tried to connect using beeline on my client machine (i am able to connect to a non kerized hadoop environment) shell> beeline -u jdbc:hive2://<knox host>:8443/;ssl=true;sslTrustStore=<keystore client path>=< keystore pwd>;user=<ad username>;password=<adpassword>hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/client77/Documents/HDP/apache-hive-3.1.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/client77/Documents/HDP/hadoop-3.0.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://<knox host>:8443/ 19/07/17 09:10:34 [main]: WARN jdbc.HiveConnection: Failed to connect to <knox host>:8443 Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable. Error: Could not open client transport with JDBC Uri: jdbc:hive2://<knox host>:8443/: null (state=08S01,code=0). Not sure about the missing steps, could you please help? Thank you ... View more

HI I have exported a table from a hadoop envrionment using the following command: export table department to 'hdfs_exports_location/department'; I tried to import the same table into another hadoop environment using the command: import from 'hdfs_exports_location/department'; i get the following error: Error: Error while compiling statement: FAILED: SemanticException [Error 10027]: Invalid path (state=42000,code=10027) i tried using import table imported_dept from 'hdfs_exports_location/department'; i get the following error: Error: Error while compiling statement: FAILED: SemanticException [Error 10324]: Import Semantic Analyzer Error (state=42000,code=10324) Any idea what could be the issue? i am using hive 3.1.0.3.0.1.0-187. Thank you ... View more

HI, I am trying to connect a BO tool (BO 4.2 SP5 Patch1) to a Kerberized Hortonworks Hadoop environment (HDP 3.0.1) HIVE(3.1.0 )via a JDBC connector but i am not able to find any decent online documentation about the subject. Is it something that is supported by HortonWorks Hadoop? I was able to connect BO to a non kerberized Hadoop via the SAP native JDBC connector. However, I was told that from the SAP side connection to a Kerberized Hadoop HIve was not supported? Any idea on that? Does HortonWorks/Cloudera provide any alternative tool? ... View more

HI, I am trying to connect a BO tool (BO 4.2 SP5 Patch1) to a Kerberized Hortonworks Hadoop environment (HDP 3.0.1) HIVE(3.1.0 )via a JDBC connector but i am not able to find any decent online documentation about the subject. Is it something that is supported by HortonWorks Hadoop? I was able to connect BO to a non kerberized Hadoop via the SAP native JDBC connector. However, I was told that from the SAP side connection to a Kerberized Hadoop HIve was not supported? Any idea on that? Does HortonWorks/Cloudera provide any alternative tool? ... View more

Is it possible to encrypt a file in PGP using a Nifi EncryptContent Processor with a DSA Public Key of 1024 bits? I tried it and i always get the following error messages: Error Cannot encrypt .... Can't use DSA for encryption. thank you ... View more