Member since
01-14-2019
28
Posts
14
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 244 | 11-30-2016 04:08 PM | |
| 2062 | 04-22-2016 09:30 PM |
09-15-2017
06:26 PM
Your cluster is in poor shape and the errors could be a result of one or more of this outside of hive. Get your cluster stable, at least HDFS which Hive is depended on. This would entail making sure your zookeeper is up and running if you're using HDFS High availability Restart your hive service once your HDFS is stable. Check to see if port 10000 or 10001 depending on where your hiveserver2 port is listening on is up and running Don't use 'hive' when testing. Use 'beeline'. > beeline
> !connect <jdbc url> Once you have a more stable cluster, come back here and paste the error logs again.
... View more
07-26-2017
09:12 PM
It is not clear if you're using two tables - a source and a destination table, or you simply want to select the time element from timestamp datatype from a single table.
Extracting the time from a timestamp involves extracting the different components in the timestamp, for example:
select concat(hour(current_timestamp()), ':', minute(current_timestamp()), ':', second(current_timestamp()));
** https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions
... View more
07-24-2017
07:49 PM
This is an NLP classification type of problem. You'll run into a few road blocks along the way. Unless you already have a trained model that classifies your information, via supervised or unsupervised methods, there are no simple code examples. At the very basic level, without using Machine learning, you can potentially use regex, albeit a complicated one, to test for specific words. Even then, you'll end up with several rules which would be unmaintainable. You may want to search for pre-existing NLP classification datasets that are already 'labeled' or have been trained to recognize these categories -- I don't know of any off hand. If you do find one, then things become easy and you could simply create a UDF that can run the text against the model and it would simply provide the label for you in return. I know this isn't the answer your looking for but hope it puts you in the right path.
... View more
06-22-2017
06:43 PM
If i'm understanding correctly, you simply want a way to transfer from you shared folder to another linux file system on the edge node and NOT to hdfs. There are a few ways to do it. 1. Use winscp like @Jay SenSharma mentioned 2. Create an nfs share on the edge node so you could simply use your regular winodws folder to drag and drop 3. Create an ftp(s) service on the edge node so you could use ftp to transfer the file 4. Use a program like filezilla to do the transfer for you. And these are just some of the options. You may also be interested in how to upload from your workstation to, eventually hdfs. 1. After uploading to edge node and if you have the hdfs client, simply use hdfs commands as @Jay SenSharma has mentioned 2. Use an NFS gateway. This way, the hadoop file system can be displayed as a regular folder in your windows machine.. pretty cool actually. 3. Use ambari files view to upload files in your shared folder directly to hdfs 4. Use 3rd party tools to move the files for you. BI tools have hooks that can you webhdfs api to upload the files for you directly to hdfs.
... View more
06-20-2017
06:22 PM
Seems like you are adding these properties directly in the pig properties. Have you tried configuring the actual log4j.properties file that pig uses (I believe under /etc/pig/conf)?
... View more
02-21-2017
06:53 AM
1 Kudo
Does performing a:
hdfs haadmin -failover nn1 nn2
Ever a disruptive procedure? Suppose all services are up and running (zookeeper, zkfc, JN), it's my understanding that this should be a safe procedure and wouldn't cause jobs to fail, but wanted to know in what circumstances would this potentially be problematic.
... View more
Labels:
12-07-2016
01:34 AM
Ahh.. actually misread your query... thought you were simply reading off a schema... @Michael Young explains it better 🙂
... View more
12-07-2016
01:31 AM
1 Kudo
Simply put, the first query only hits the metastore database and doesn't launch a map reduce job. On the other hand, the second query runs a map side mapreduce job EDIT: Interestingly enough, for the first query, hive makes some good decisions on how to read the data. A simple select * could essentially simply be fetching a file from hdfs like an hdfs get... simplified, but true.
... View more
12-06-2016
11:56 PM
1 Kudo
as @Ramesh Mani mentioned, this seems to be more authorization related. For a quick fix, try to assigning read permissions the hdfs level (hadoop fs -chmod 755 /apps/hive/warehouse). For a more valid way of doing it, go to ranger and go to your hdfs policies and make sure you have the proper permissions for hive user to access the said directory.
... View more
11-30-2016
04:08 PM
1 Kudo
@Sunile Manjee This would really depend on the cluster size and the number of jobs running. Very hard to gauge. A 10 node cluster with around 15-20 components can easily generate 1GB of audit logs PER DAY. Again depends on the cluster activity. You could use this as a baseline, but again, really hard to gauge. Then again consider this only if being forced to use DB and after strongly advising against using DB as oppose to using Solr for ranger audits 🙂
... View more
10-24-2016
01:18 PM
The falcon service fails after a few hours with a GSSException error saying it can't get a kerberos key. We have to restart the service after every few hours. After restarting, things are OK and we are able to run the jobs, but then after 3-4 hours, everything start going to an "UNKNOWN" state. There's currently a JIRA https://issues.apache.org/jira/browse/FALCON-1595 that is related to what we have but my understanding is that the affected version is 0.8 and we're using 0.6 (HDP 2.4.2). Based on the logs there's background timer that runs to check for the keys. I've lowered the ticket validity setting from 10 hours to 5 hours which makes me believe this will force falcon check the tickets. Our ticket expires after 24 hours so not sure why this will help, but I gave it a shot. Here's the error log: 2016-10-21 14:56:42,992 INFO - [Timer-2:] ~ Logging in user1 (CurrentUser:65)
2016-10-21 14:56:43,019 INFO - [Timer-2:] ~ Creating FS impersonating user user1 (HadoopClientFactory:196)
2016-10-21 14:56:43,023 WARN - [Timer-2:] ~ Exception encountered while connecting to the server : (Client:685)
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563)
at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492)
at org.apache.hadoop.ipc.Client.call(Client.java:1402)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy23.getListing(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:575)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy24.getListing(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2140)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2123)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:849)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:107)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:911)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:907)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:907)
at org.apache.hadoop.fs.Globber.listStatus(Globber.java:69)
at org.apache.hadoop.fs.Globber.glob(Globber.java:217)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)
at org.apache.falcon.cleanup.AbstractCleanupHandler.getAllLogs(AbstractCleanupHandler.java:102)
at org.apache.falcon.cleanup.AbstractCleanupHandler.delete(AbstractCleanupHandler.java:143)
at org.apache.falcon.cleanup.ProcessCleanupHandler.cleanup(ProcessCleanupHandler.java:39)
at org.apache.falcon.service.LogCleanupService$CleanupThread.run(LogCleanupService.java:68)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 40 more
2016-10-21 14:56:43,025 INFO - [Timer-2:] ~ Exception while invoking getListing of class ClientNamenodeProtocolTranslatorPB over <falcon_host>/xxx.xxx.56.168:8020. Trying to fail over immediately. (RetryInvocationHandler:148)
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "<falcon_host>/xxx.xxx.56.168"; destination host is: "<falcon_host>":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
at org.apache.hadoop.ipc.Client.call(Client.java:1430)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy23.getListing(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:575)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy24.getListing(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2140)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2123)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:849)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:107)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:911)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:907)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:907)
at org.apache.hadoop.fs.Globber.listStatus(Globber.java:69)
at org.apache.hadoop.fs.Globber.glob(Globber.java:217)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)
at org.apache.falcon.cleanup.AbstractCleanupHandler.getAllLogs(AbstractCleanupHandler.java:102)
at org.apache.falcon.cleanup.AbstractCleanupHandler.delete(AbstractCleanupHandler.java:143)
at org.apache.falcon.cleanup.ProcessCleanupHandler.cleanup(ProcessCleanupHandler.java:39)
at org.apache.falcon.service.LogCleanupService$CleanupThread.run(LogCleanupService.java:68)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:690)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:653)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:740)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492)
at org.apache.hadoop.ipc.Client.call(Client.java:1402)
... 28 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563)
at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727)
... 31 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 40 more
2016-10-21 14:56:43,028 WARN - [Timer-2:] ~ Exception encountered while connecting to the server : (Client:685)
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563)
at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492)
at org.apache.hadoop.ipc.Client.call(Client.java:1402)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy23.getListing(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:575)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy24.getListing(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2140)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2123)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:849)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:107)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:911)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:907)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:907)
at org.apache.hadoop.fs.Globber.listStatus(Globber.java:69)
at org.apache.hadoop.fs.Globber.glob(Globber.java:217)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)
at org.apache.falcon.cleanup.AbstractCleanupHandler.getAllLogs(AbstractCleanupHandler.java:102)
at org.apache.falcon.cleanup.AbstractCleanupHandler.delete(AbstractCleanupHandler.java:143)
at org.apache.falcon.cleanup.ProcessCleanupHandler.cleanup(ProcessCleanupHandler.java:39)
at org.apache.falcon.service.LogCleanupService$CleanupThread.run(LogCleanupService.java:68)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 40 more
... View more
Labels:
10-05-2016
08:21 PM
I'm working with HDP 2.3.4.7 I'm unable to pull the policies in Ranger for knox with SSL turned on. I've been successful for both hdfs and hive though. My knox is using a certificate that's signed with the organization's CA. I've downloaded both the root and sub ca and placed in under the default JAVA location for Ranger under /usr/jdk/.../securiy/cacerts. I've also added the public certificate of the private certificate that Knox server is using in the /usr/hdp/current/knox-server/data/security/keystore/gateway.jks. In addition, I've also added ranger's public certificate to knox's public keystore as defined under the knox configuration which is currently under /usr/hdp/current/knox-server/conf/ranger-plugin-truststore.jks. However, I'm still unable to pull the policies. Surprisingly there are no ERRORS in the gateway.log file, so I had to turn on DEBUGGING. When I did that, these are the errors that come up which have all the sings of a misconfigured SSL happening somewhere. I just don't know where since I've already added all the certificates I could think of. Here's the error: 2016-10-05 14:46:30,786 DEBUG io.nio (SelectorManager.java:createEndPoint(842)) - created SCEP@49ebe0fd{l(/xxx.xxx.111.243:23766)<->r(/xxx.xxx.111.243:8443),s=0,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0}-{SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=-1/-1/-1 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
2016-10-05 14:46:30,791 DEBUG nio.ChannelEndPoint (ChannelEndPoint.java:shutdownChannelInput(118)) - ishut SCEP@49ebe0fd{l(/xxx.xxx.111.243:23766)<->r(/xxx.xxx.111.243:8443),s=1,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0r}-{SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
2016-10-05 14:46:30,792 DEBUG nio.ssl (SslConnection.java:process(347)) - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} NOT_HANDSHAKING filled=-1/0 flushed=0/0
2016-10-05 14:46:30,794 DEBUG nio.ChannelEndPoint (ChannelEndPoint.java:shutdownChannelOutput(157)) - oshut SCEP@49ebe0fd{l(/xxx.xxx.111.243:23766)<->r(/xxx.xxx.111.243:8443),s=1,open=true,ishut=true,oshut=false,rb=false,wb=false,w=true,i=0r}-{SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
2016-10-05 14:46:30,794 DEBUG nio.ChannelEndPoint (ChannelEndPoint.java:close(209)) - close SCEP@49ebe0fd{l(/xxx.xxx.111.243:23766)<->r(/xxx.xxx.111.243:8443),s=1,open=true,ishut=true,oshut=true,rb=false,wb=false,w=true,i=0r}-{SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
2016-10-05 14:46:30,795 DEBUG http.HttpParser (HttpParser.java:parseNext(281)) - filled -1/0
2016-10-05 14:46:30,795 DEBUG io.nio (SelectorManager.java:destroyEndPoint(851)) - destroyEndPoint SCEP@49ebe0fd{l(null)<->r(0.0.0.0/0.0.0.0:8443),s=1,open=false,ishut=true,oshut=true,rb=false,wb=false,w=true,i=0!}-{SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=0,c=0},r=0}}
2016-10-05 14:46:30,796 DEBUG nio.ssl (SslConnection.java:process(347)) - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=0,c=0},r=0} NOT_HANDSHAKING filled=-1/0 flushed=0/0
2016-10-05 14:46:30,796 DEBUG server.AbstractHttpConnection (AbstractHttpConnection.java:onClose(738)) - closed AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=0,c=0},r=0
2016-10-05 14:46:30,796 DEBUG server.AsyncHttpConnection (AsyncHttpConnection.java:handle(145)) - Disabled read interest while writing response SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=0,c=0},r=0}
2016-10-05 14:46:30,797 DEBUG nio.ssl (SslConnection.java:handle(203)) - [Session-1, SSL_NULL_WITH_NULL_NULL] handle SslConnection@1d0027f5 SSL NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false {AsyncHttpConnection@c3d6fb9,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=0,c=0},r=0} progress=false
Interestingly enough, the ip addresses that are displayed in the DEBUG is the same. I would have thought I would see the other ip address reference both the knox server and the ranger server. I must be missing something here.
... View more
Labels:
08-11-2016
08:58 PM
Whenever I start up the KMS server, I get the following errors:
2016-08-11 16:39:23,561 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=clusterprod_kms).PolicyRefresher()
2016-08-11 16:39:23,561 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=clusterprod_kms).PolicyRefresher()
2016-08-11 16:39:23,561 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=clusterprod_kms).loadPolicy()
2016-08-11 16:39:23,561 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=clusterprod_kms).loadPolicyfromPolicyAdmin()
2016-08-11 16:39:23,561 DEBUG RangerAdminRESTClient - ==> RangerAdminRESTClient.getServicePoliciesIfUpdated(-1)
2016-08-11 16:39:24,240 ERROR RangerAdminRESTClient - Error getting policies. request=https://cluster004.localhost.local:6182/service/plugins/policies/download/clusterprod_kms?lastKnownVersion=-1&pluginId=kms@cluster004-clusterprod_kms, response={"httpStatusCode":400,"statusCode":1,"msgDesc":"Unauthorized access - unable to get client certificate","messageList":[{"name":"OPER_NOT_ALLOWED_FOR_ENTITY","rbKey":"xa.error.oper_not_allowed_for_state","message":"Operation not allowed for entity"}]}, serviceName=clusterprod_kms
2016-08-11 16:39:24,240 ERROR PolicyRefresher - PolicyRefresher(serviceName=clusterprod_kms): failed to refresh policies. Will continue to use last known version of policies (-1)
java.lang.Exception: Unauthorized access - unable to get client certificate
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:83)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:205)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:175)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:132)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:106)
at org.apache.ranger.authorization.kms.authorizer.RangerKMSPlugin.init(RangerKmsAuthorizer.java:358)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.init(RangerKmsAuthorizer.java:280)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:114)
at org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer.<init>(RangerKmsAuthorizer.java:132)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.getAcls(KMSWebApp.java:241)
at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:134)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4992)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5490)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-08-11 16:39:24,241 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=clusterprod_kms).loadPolicyfromPolicyAdmin()
2016-08-11 16:39:24,241 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=clusterprod_kms).loadFromCache()
2016-08-11 16:39:24,243 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=clusterprod_kms).loadFromCache()
2016-08-11 16:39:24,243 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=clusterprod_kms).loadPolicy()
2016-08-11 16:39:24,244 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=clusterprod_kms).run()
2016-08-11 16:39:24,244 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=clusterprod_kms).loadPolicy()
2016-08-11 16:39:24,244 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=clusterprod_kms).loadPolicyfromPolicyAdmin()
2016-08-11 16:39:24,244 DEBUG RangerAdminRESTClient - ==> RangerAdminRESTClient.getServicePoliciesIfUpdated(-1)
2016-08-11 16:39:24,245 DEBUG RangerKmsAuthorizer - <== RangerkmsAuthorizer.init()
2016-08-11 16:39:24,275 INFO log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-08-11 16:39:24,275 INFO log - ------------------ Ranger KMSWEbApp---------------------
2016-08-11 16:39:24,275 INFO log - provider string = dbks://http@localhost:9292/kms
2016-08-11 16:39:24,275 INFO log - URI = dbks://http@localhost:9292/kms scheme = dbks
2016-08-11 16:39:24,275 INFO log - kmsconf size= 427 kms classname=org.apache.hadoop.conf.Configuration
2016-08-11 16:39:24,275 INFO log - ----------------INstantiating key provider ---------------
2016-08-11 16:39:24,292 ERROR RangerAdminRESTClient - Error getting policies. request=https://cluster004.localhost.local:6182/service/plugins/policies/download/clusterprod_kms?lastKnownVersion=-1&pluginId=kms@cluster004-clusterprod_kms, response={"httpStatusCode":400,"statusCode":1,"msgDesc":"Unauthorized access - unable to get client certificate","messageList":[{"name":"OPER_NOT_ALLOWED_FOR_ENTITY","rbKey":"xa.error.oper_not_allowed_for_state","message":"Operation not allowed for entity"}]}, serviceName=clusterprod_kms
2016-08-11 16:39:24,292 ERROR PolicyRefresher - PolicyRefresher(serviceName=clusterprod_kms): failed to refresh policies. Will continue to use last known version of policies (-1)
java.lang.Exception: Unauthorized access - unable to get client certificate
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:83)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:205)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:175)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:154)
2016-08-11 16:39:24,293 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=clusterprod_kms).loadPolicyfromPolicyAdmin()
I've already installed the certificates using keytool, all configs are pointed to the correct keystores. Ranger is SSL enabled. Also seems like the KMS plugin isn't being installed correctly. HDP: 2.3.4.7
Ambari: 2.2.2.0 Any thoughts?
... View more
Labels:
08-03-2016
05:45 PM
Oh, sorry misread the path... it works. much appreciated!
... View more
08-03-2016
03:13 PM
@Jitendra Yadav it's under /usr/hdp/2.3.4.7/ranger-hive-plugin/lib/ranger-hive-plugin/ranger-hive-plugin-.... Seems like it exists....
... View more
08-03-2016
02:53 PM
We just installed ranger and turned on hdfs and hive plugin. However, hiveserver2 keeps giving out a Classnotfound exception saying that org.apache.ranger.authorization.hive.authorizer.rangerhiveauthorizerfactory could not be found. This is using HDP v 2.3.4.7 (kerberized) Ranger: 0.5.0.2.3 Any ideas?
... View more
Labels:
07-28-2016
06:24 PM
Assume you have a kerberized cluster using both LDAP and a standalone KDC. In a kerberized cluster, what are the steps that happen for knox to authentication via ldap and kerberos? My understanding is that when you login to knox, knox does an LDAP bind to an LDAP server, once authenticated, Knox would then (now behind the proxy) contact the kerberos service and get a ticket for that user. Question is, 1. Are these assumptions correct? 2. Which user does Knox get a ticket for, the knox user, or the user that is used when binding to LDAP?
... View more
Labels:
05-26-2016
07:26 PM
A few things:
Have you tried restarting your ambari-server and all the ambari-agents? You could try
root@hambarihost> amabri-server restart
And:
root@ambariagenthost> ambari-agent restart
Start option graying in my experience has been because when the components register with Ambari the reverse lookup from ambari wasn't successful for the host. Not sure why things would change after installing ranger, but can you check if you could still successfully do a name resolution and reverse name resolution on your hosts? Tail the /var/log/ambari-agent/amabri-agent.log and see if you could find anything under WARNING or ERROR. Note that sometimes a 'smoking gun' event can be displayed as an INFO. You're also looking for python errors, this will usually have a "Traceback" line. Post the log here and I could take a more indepth look.
... View more
05-26-2016
04:18 AM
2 Kudos
We recently had some issues with Ambari and Kafka alerts. It all started when in HDP 2.3.4, every time we changed the kafka listener port from 6667 to any other port, Ambari would complain and give us an error saying that it couldn’t reach port 6667 even though the broker service is actually running in another port. Here’s the exact error:
“Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667”
This can be quite annoying especially if you have multiple brokers and they’re all reporting CRITICAL and you just can’t seem to get rid of it.
To cut the long story short, here are the steps we did to get rid of the problem. In the section below, we’ll jump into some troubleshooting tips:
1. Get the ID of the kafka broker
> curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "http://localhost:8080/api/v1/clusters/Sandbox/alert_definitions"
2. Get the definitions and save it locally
> curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "http://localhost:8080/api/v1/clusters/Sandbox/alert_definitions/47” > kafka_alerts.json
3. EDIT kafka_alerts.json
Remove the href line.
Change 6667.0 to your new port (e.g. 9092) (Do NOT use decimal or you get a NumberFormatException in the ambari-server.log and no Alerts)
The final JSON file should look like this:
{
"AlertDefinition" : {
"cluster_name" : "Sandbox",
"component_name" : "KAFKA_BROKER",
"description" : "This host-level alert is triggered if the Kafka Broker cannot be determined to be up.",
"enabled" : true,
"id" : 47,
"ignore_host" : false,
"interval" : 1,
"label" : "Kafka Broker Process",
"name" : "kafka_broker_process",
"scope" : "HOST",
"service_name" : "KAFKA",
"source" : {
"default_port" : 9092,
"reporting" : {
"critical" : {
"value" : 5.0,
"text" : "Connection failed: {0} to {1}:{2}"
},
"warning" : {
"text" : "TCP OK - {0:.3f}s response on port {1}",
"value" : 1.5
},
"ok" : {
"text" : "TCP OK - {0:.3f}s response on port {1}"
}
},
"type" : “PORT”,
"uri" : "{{kafka-broker/port}}"
}
}
4.
Upload the file
Do this by running this command in the same directory where you saved kafka_alerts.json file:
> curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT "http://localhost:8080/api/v1/clusters/Sandbox/alert_definitions/47" -d @kafka_alerts.json
It can take up to a minute for Ambari to run the metrics again. To speed things up
you can force ambari to run the alert check:
> curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT "http://localhost:8080/api/v1/clusters/Sandbox/alert_definitions/47?run_now=true”
This
should solve the issue.
Troubleshooting:
If you're still having trouble, these suggestions/tips should help you out.
When uploaded the JSON, make sure the JSON is valid. This is easy to catch as the PUT
will return an error that says invalid structure.
Make sure the default_port is an INTEGER when you upload. This is tricky because if
you keep the decimal (ex. 6667.0), you won't get an error response, but if you look at /var/log/ambari-server/ambari-server.log, you'll get a number format exception.
What's even more tricky is that ambari will start ignoring these metrics all together and you'll end up
with this:
Tail the /var/log/ambari-agents/amabri-agents.log file when you run the PUT commands and have a look out for these types of log entries:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 274, in __json_to_callable
source = json_definition['source']
TypeError: 'NoneType' object is unsubscriptable
This means that your json is invalid, either because of a number format exception or other reasons. Correlate against the ambari-server.log to find out additional information.
Manually trigger the alert and look for these types logs for validation:
INFO 2016-05-25 16:48:33,355 AlertSchedulerHandler.py:374 - [AlertScheduler] Executing on-demand alert kafka_broker_process (1e0e1edc-e051-45bc-8d38-97ae0b3b83f0)
This at least gives you confidence that your alert definition is valid. If instead you get these type of error:
ERROR 2016-05-25 19:40:21,470 AlertSchedulerHandler.py:379 - [AlertScheduler] Unable to execute the alert outside of the job scheduler
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 363, in execute_alert
alert_definition = execution_command['alertDefinition']
KeyError: 'alertDefinition'
Then you know something's wrong with the alert definition.
If things still don't work, try removing the uri from the alert defintion. This will force
ambari to look at the default_port as fall back. Ambari's alert scheduler first looks at the URI,
if it's valid, it uses this, if not, it falls back to using default_port. Remember if you remove
the uri be sure to remove the comma after "PORT"
... View more
- Find more articles tagged with:
- Cloud & Operations
05-24-2016
06:25 PM
1 Kudo
I'm not sure if this question is limited to what can be done from an HDP ecosystem point of view. As another approach, is there a way to protect this using layer 3/4? Specifically, allow only a specific host/IP or subnet and ports to access your hive database. This can be done via firewall rules.
... View more
05-10-2016
01:13 PM
Thanks, very insightful. Interesting to note how zookeeper starts slowing down on writes as zookeeper nodes are added. Makes sense, I didn't realize zookeeper has to write to disk before acknowledgements are sent back to master.
... View more
05-09-2016
07:40 PM
Given the best practice of separating master and slave node configuration, for the sake of argument, if you have a 2/3 master/slave node configuration, is it recommended to have 3 zookeeper masters and have the other zookeeper installed on a slave node or simply install one zookeeper one of the master nodes? Appreciate the input.
... View more
05-06-2016
02:48 PM
1 Kudo
Based on what
@asinghal said, check your configuration object. The code that's running behind the scenes is:
public static String getUpsertStatement(final Configuration configuration) throws SQLException {
Preconditions.checkNotNull(configuration); // <-- THIS IS WHERE you're getting the error
String upsertStmt = configuration.get(UPSERT_STATEMENT);
if(isNotEmpty(upsertStmt)) {
return upsertStmt;
}
...
Just for kicks, try putting a stop break on your code:
final Configuration configuration = HBaseConfiguration.create();
final Job job = Job.getInstance(configuration, "phoenix-mr-job");
and look at your 'configuration' object and make sure the object is valid. HTH
... View more
05-06-2016
02:38 PM
4 Kudos
Just some thoughts here, before writing your own custom input format. Seems like your XML file is incredibly large and you want to prevent the entire XML from being loaded into memory. StAX is definitely one way but you loose parallelism. if you want to preprocess your XML, try: looking into Mahout's XMLInputFormat (https://dzone.com/articles/hadoop-practice), Or you could try using PIG's XML loader (http://hadoopgeek.com/apache-pig-xml-parsing-xpath/). Better yet, you could also look at HDF/Nifi to process XML (https://community.hortonworks.com/articles/25720/parsing-xml-logs-with-nifi-part-1-of-3.html) HTH.
... View more
05-06-2016
02:10 PM
Go to ambari -> Hive -> Services -> Restart all to restart hive server and hive metastore services. You may need tor restart ambari service also. Go to your amabri server and in the command line, type: > ambari-server restart Make sure you restart under the account that you started ambari server (usually root).
... View more
04-22-2016
09:45 PM
Just to add to Ajay, If you're using ranger, check policies for HDFS, hive, and hbase and make sure you have permissions to access the table and hdfs directory. Kerberized clusters will certainly have something to do with this. Make sure the service principals have been added properly. If you kerberized your cluster using ambari, this should be done for you automatically.
... View more
04-22-2016
09:31 PM
Oh and don't forget to put a PutSQL processor to actually execute the sql command that ConvertJSONToSQL processor does for you 🙂
... View more
04-22-2016
09:30 PM
2 Kudos
Make the following changes: Change your connection string to include the database. For example: jdbc:mysql://localhost:3309/[databasename] Take out the schema and the catalog name from the jsontosql processor configuration and just leave the table name. Your schema will come from the connection string And you should be good to go.
... View more
