Member since
07-11-2016
25
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2894 | 08-21-2017 04:57 AM | |
3609 | 08-17-2017 01:13 AM |
10-23-2017
10:30 AM
Hi, When we try to run the spark-submit on our kerberos cluster, we get below mentioned error. spark-submit -v \ --master yarn-client \ --jars $myDependencyJarFiles \ --conf spark.default.parallelism=12 \ --conf spark.driver.memory=4g \ --conf spark.executor.memory=4g \ --conf spark.yarn.security.tokens.hive.enabled=false \ --conf spark.yarn.security.tokens.hbase.enabled=false \ --conf spark.yarn.executor.memoryOverhead=2048 \ --conf spark.kryoserializer.buffer.max=512m \ --conf spark.yarn.appMasterEnv.MASTER=yarn-client \ --conf spark.yarn.appMasterEnv.HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop \ --conf spark.yarn.appMasterEnv.HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce \ --conf spark.hadoop.fs.hdfs.impl.disable.cache=true \ --keytab user.keytab \ --principal user@abc.domain.com --class com.test.spark.spark.testclass \ ERROR: y ERROR - 17/10/23 15:54:48 WARN yarn.ExecutorDelegationTokenUpdater: Error while trying to update credentials, will try again in 1 hour 23-10-2017 15:54:48 GMT test-hourly ERROR - 17/10/23 15:54:48 ERROR util.Utils: Uncaught exception in thread main 23-10-2017 15:54:48 GMT test-hourly ERROR - java.lang.StackOverflowError Also, I have tried by running the spark-submit with principle & keytab and without principle & keytab, but the error was same. Can anybody please help me on the same, if the error is related with having a client mode on kerberos cluster or if you can help me on the possible fix. Thanks, Amit
... View more
Labels:
- Labels:
-
Apache Spark
-
Kerberos
08-31-2017
04:24 AM
I do have same issue, on Kerberos enalbed cluster Impala-shell works fine on one machine but gives the same TTransportException error on another machine of the cluster, can anybody please help on the same.
... View more
08-24-2017
01:03 PM
Hi, I am getting below error in hostmonitor log; Monitor-HostMonitor throttling_logger ERROR ntpdc: ntpdc -np: not synchronized to any server Running ntpstat gives below message, which seems to be synchronized. ntpstat synchronised to unspecified at stratum 11 time correct to within 61 ms polling server every 64 s Can anybody please help me out on the same. Thanks, Amit
... View more
- Tags:
- HostMonitor
Labels:
- Labels:
-
Cloudera Manager
08-21-2017
06:19 AM
Hi, Can anybody please help me on the same. Thanks, Amit
... View more
08-21-2017
04:57 AM
I am able to resolve this issue by setting the verify_cert_dir in /etc/cloudera-scm-agent/config.ini I was missing the root certificate file, which I had download from CA authority and added to the verify_cert_dir. Also, I had executed below command to verify the same. openssl verify -verbose -CAfile <(cat cert_intermediate_ca.pem thawte_root_ca.pem) hostname.pem It gave me message: hostname.pem: OK Thanks, Amit
... View more
08-21-2017
04:53 AM
Hi, I am trying to configure the TLS level 3 encryption using method B from below link; https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_config_tls_agent_auth.html I alreday have a wildcard based public CA. Question: 1] Do I need to generate the CSR request on each of the host and send the CSR to CA i.e. if I have 10 datanode then do I need to create 10 i.e. csr per node OR Since, I have the wildcard public CA, I am not required to do the same and can directly jump to "Step 5: Export the private key from the Java keystore and convert it with OpenSSL for reuse by Agent" of the documentation? Can you please confirm the same, I would be realy thankful for the same. Thank you, Amit
... View more
- Tags:
- TLS Encryption
Labels:
- Labels:
-
Cloudera Manager
-
Security
08-17-2017
01:21 AM
Hi All, I had implemented the Level 1 TLS encryption and which is working. But, when I have implemented the Level 2 TLS encryption as per the steps given in below link https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_config_tls_auth.html#topic_3 I have started getting below error. 1. In cloudera-scm-agent log [17/Aug/2017 07:24:50 +0000] 31094 MainThread agent ERROR Heartbeating to c018-srv1.e8sec.com:7182 failed. Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.1-py2.6.egg/cmf/agent.py", line 1346, in _send_heartbeat self.max_cert_depth) File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.1-py2.6.egg/cmf/https.py", line 132, in __init__ self.conn.connect() File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect self.sock.connect((self.host, self.port)) File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 185, in connect ret = self.connect_ssl() File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl return m2.ssl_connect(self.ssl) SSLError: certificate verify failed 2. In Cloudera-scm-Server Log 2017-08-17 07:51:04,118 WARN 118674289@agentServer-169:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: unknown_ca I have tried by using verify_cert_file as well as by using verify_cert_dir. Can anybody please help me on the same, if I am missing something or anything else needed to be done to fix this issue. I would be really thankful for any help on the same. Thank you, Amit
... View more
- Tags:
- TLS Encryption
Labels:
- Labels:
-
Cloudera Manager
-
Kerberos
-
Security
08-17-2017
01:13 AM
This issue resolved for me when I rebooted my CM machine. Thanks, Amit
... View more
08-16-2017
11:09 AM
Thanks mbigelow, Yes, I have added the public CA certificate to keystore and I have given the user cloudera-scm a full permission on the keystore files like cacerts,jsscacerts, pki folder, x509 folder and jks folder. I have validated the certificate using commands; openssl s_client -showcerts -connect hostname:443 And keytool -list -v -keystore cacerts --alias I have also validated that in the cloudera agent process file /var/run/cloudera-scm-agent/process/1653-cloudera-mgmt-SERVICEMONITOR/cmon.conf I can see some of the ssl entries as per below; <property> <name>scm.server.url</name> <value>https://hostname:7183</value> </property> <property> <name>com.cloudera.enterprise.ssl.client.truststore.location</name> <value>/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/cacerts</value> </property> <property> <name>com.cloudera.enterprise.ssl.client.truststore.password</name> <value>changeit</value> </property> Regarding your point " correct hostname in certificate" do I need to verify anything else, apart from what I mentioned above. Also, I would be really thankful if you can suggest, what else I can do to fix these errors. Thanks, Amit
... View more
08-16-2017
08:38 AM
Hi, I have enabled the TLS level 1 encryption and after the same I am getting few errors in my log as per below; 1] Getting below error in My cloudera-scm-server.log 2017-08-16 14:56:56,261 INFO MainThread:com.cloudera.server.cmf.WebServerImpl: Cipher suite TLS_EMPTY_RENEGOTIATION_INFO_SCSV found. Allowing SSL/TLS renegotiations. 2017-08-16 14:56:56,288 INFO MainThread:com.cloudera.server.cmf.WebServerImpl: TLS web connections will use port: 7183 2017-08-16 14:56:56,292 INFO MainThread:com.cloudera.server.cmf.WebServerImpl: Plaintext web connections will use port: 7180 2017-08-16 14:56:56,337 INFO MainThread:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GenerateCredentials BasicCmdArgs{args=[]}. 2017-08-16 14:56:56,337 INFO MainThread:com.cloudera.server.cmf.Main: Generating credentials (command 4481) at startup 2017-08-16 14:56:56,393 INFO WebServerImpl:com.cloudera.enterprise.JavaMelodyFacade: No JavaMelody class net.bull.javamelody.SessionListener: net.bull.javamelody.SessionListener 2017-08-16 14:56:56,479 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Unable to retrieve remote parcel repository manifest java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused to http://serverip:8000/manifest.json at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297) at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104) at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:399) at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:390) at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:352) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:409) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused to http://serverip:8000/manifest.json at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100) ... 11 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282) ... 3 more 2017-08-16 15:31:35,624 INFO 1922557741@scm-web-39:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-EVENTSERVER-bdec96eb8ea18d0be431197fa05f0a3b' from CMhost 2] Getting below error in my cloudera-scm-agent.log ERROR Heartbeating to CMhostname:7182 failed. Connection refused Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.1-py2.6.egg/cmf/agent.py", line 1346, in _send_heartbeat self.max_cert_depth) File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.1-py2.6.egg/cmf/https.py", line 132, in __init__ self.conn.connect() File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect self.sock.connect((self.host, self.port)) File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 181, in connect self.socket.connect(addr) File "<string>", line 1, in connect error: [Errno 111] Connection refused ERROR [1646-cloudera-mgmt-HOSTMONITOR] Failed to update 3] In Eventserver log file 2017-08-16 13:28:30,475 ERROR com.cloudera.cmf.eventcatcher.server.EventCatcherService: Error starting EventServer org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/lib/cloudera-scm-eventserver/v3/write.lock Can anybody please help me on the same, as I am not able to find out the proper solution for the same. Thank you in advance. Thanks, Amit
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Kerberos
-
Security
08-13-2017
12:32 PM
Hi, Below is my scenario. 1] We already have trusted public CA certificate. 2] We have a new cluster where we want to implement the kerberos and as a pre-requisite enabling the TLS encryption. 3] Since, cluster is new created the cacerts using command as per below on CM host under /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/. keytool -keystore cacerts -importcert -alias aliasname -file /certificatepath/cert_wildcard_full.pem 4] Used the steps mentioned in below link; https://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_create_deploy_certs.html From above link Followed - step[1],step[2] From above link Not Followed - Since, already have the trusted public CA certificate hence not followed the step[3],step[4],step[5] [Please correct me, if I am wrong] And then followed step[6] i.e. Import the Certificate into the Keystore 5] After doing this followed below link steps. https://www.cloudera.com/documentation/enterprise/latest/topics/sg_cms_ssl.html 6] I have also copied the $JAVA_HOME/jre/lib/security/jssecacerts to all nodes on the cluster and created the cacert on all nodes of the cluster using same command given above in #3. However, when I starts the cloudera server I am getting below error consistently; WARN 1671856335@scm-web-4:org.mortbay.log: javax.net.ssl.SSLHandshakeException: no cipher suites in common Can anybody please help me on this, I have tried various things but not able to fix it and due to this CM is not working. If I am missing anything in above steps or doing anything wrong in above steps can you please confirm what would be the right steps to enable the TLS for trusted public CA. Thank you very much in advance. Really looking out for any assistance on the same. Thanks, Amit
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Kerberos
03-30-2017
02:50 AM
Thanks Lars for your help. On Impala DELETE, is their any specific reason to stop DELETE on Impala tables? May be I'm missing something, but as I have the kudu service installed on my cluster. In cloudera Impala configuration, what is the difference between setting kudu service or selecting none as in both cases kudu queries works fine. Thank you for your help. Thanks, Amit
... View more
03-29-2017
06:00 AM
Hi,
I had a cluster[CDH 5.8.2] in which I was using Impala and Kudu.
Impala Parcel is downloaded from - http://archive.cloudera.com/beta/impala-kudu/parcels/latest/
I have upgraded this cluster to CDH 5.10 with cloudera manager 5.10.
Now, running the select verison() query on this upgraded cluster in Impala gives me below details;
impalad version 2.7.0-cdh5.10.0 RELEASE
However, in CDH 5.10 upgrade they mentioned the support for Imapal 2.8. I can not find the parcel for the Impala 2.8.
Also, running the delete command on Impala table give me below error.
"ERROR: AnalysisException: Impala does not support modifying a non-Kudu table: default.impala_testtable"
Questions:
1. Can anybody suggest me how can I upgrade to Impala 2.8? Is there any parcel for the same or the one which I'm currently using is the latest?
2. As running delete command on Impala table gives me the error what is the alternative to delete data from existing impala table? However, the delete command works fine with Kudu tables.
Can anybody please help me on the same.
Thanks,
Amit
... View more
Labels:
- Labels:
-
Apache Impala
-
Apache Kudu
01-09-2017
03:26 AM
Thanks a lot mbigelow. Ldap auth worked for me. I was using openLdap ldap_baseDN like cn=test,ou=xyz,dc=impalabigdata,dc=net, when I changed it as ou=xyz,dc=impalabigdata,dc=net and used the JDBC connection as jdbc:hive2://server:21050/default;","test","ldappwd", it worked for me. Thanks a lot for providing a complete details, as currently these things didn't get clarified in existing documentation. Thanks, Amit
... View more
01-07-2017
09:19 AM
Hi, I need to implement the JDBC connection for Impala with username and password[I guess here username=host user on which Impala daemon is running and Password=corresponding user password] for our client cluster. As per my understanding we do not need to make any specific change in Impala server configs or on server for the same. What I tried: 1. Cluster is without Kerberos. 2, Tried two types of drivers. a] Driver Type 1] JDBC:hive2 - When I uses connection string; DriverManager.getConnection("jdbc:hive2://server:21050/default;","hostuser","password"). It does not work at all and ends in connection timeout. When I uses connection string; DriverManager.getConnection("jdbc:hive2://server:21050/default;auth=noSasl;") It works and gets connected to impala. b] Driver Type 2] JDBC:impala - When I uses connection string; DriverManager.getConnection("jdbc:impala://Server:21050/default;AuthMech=3;UID=hostuser;PWD=password") It does not work at all and ends in connection timeout. When I uses connection string; DriverManager.getConnection("jdbc:impala:// Server :21050/default;AuthMech=0;" ) It works and gets connected to impala. Questions: 1. For Implementing the Impala JDBC username & password based connection, is it mandatory to have either Kerberos or LDAP or Sentry Implented? 2. If not, then using JDBC with Impala local user credentials does need any additional parameters or change at the server level or change in Impala local user permission? 3. Can you please provide any example to implement it, like any server level or client level changes if my implementation is wrong. Thank you in advance. Amit
... View more
Labels:
- Labels:
-
Apache Impala
07-13-2016
02:28 AM
Thank you Vinithra. However, any ided that how soon the kudu support will be available in cloudera director?
... View more
07-11-2016
10:48 AM
Hi, We are using cloudera director 2.1 and trying to install the CM 5.7.1, along with the same below are some sections where we have also added Kudu Parcel reference; products { CDH: 5.7.1 KAFKA: 2.0.0 KUDU: 0.9.1 } parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.7.1/","http://archive-primary.cloudera.com/kafka/parcels/2.0.0/","http://archive.cloudera.com/beta/kudu/parcels/0.9.1/"] services: [HDFS, YARN, ZOOKEEPER,HIVE, HUE, OOZIE, SENTRY, SPARK_ON_YARN, KAFKA,KUDU] Along with KUDU_TSERVER & KUDU_Master Roles. There are two main error which we are getting after running command as per below; sudo cloudera-director bootstrap-remote Bigdatacluster.conf --lp.remote.username=admin --lp.remote.password=admin -- lp.remote.hostAndPort=localhost:7189 1] ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RoleConfigGroupsResource.updateConfig",causeClass=class javax.ws.rs.BadRequestException, causeMessage="HTTP 400 Bad Request"} ----- Not getting any reference for this error, need help on the same. 2] Invalid role type(s) specified. Ignored during role creation: KUDU: KUDU_TSERVER,KUDU_MASTER ------ How can we setup kudu as a part of this config? Can any member please help me on the same. Thanks in advance, Amit
... View more
Labels:
- Labels:
-
Apache Kudu