Member since
10-04-2016
68
Posts
6
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4493 | 03-23-2017 08:41 AM | |
1952 | 01-26-2017 07:22 PM | |
1181 | 12-23-2016 12:07 PM | |
4114 | 12-21-2016 01:54 PM | |
1110 | 12-05-2016 06:37 AM |
03-30-2017
09:41 AM
Switched to JDK 1.7 and got the same issue. It seems that JDK can't pick up from the cache. $ export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true $HADOOP_OPTS" $ export HADOOP_ROOT_LOGGER=TRACE,console; $ export HADOOP_JAAS_DEBUG=true $ hdfs dfs -ls 2> /tmp/hdfsls.txt Java config name: null Native config name: /etc/krb5.conf Loaded from native config [UnixLoginModule]: succeeded importing info: uid = 1000 gid = 1000 supp gid = 4 supp gid = 10 supp gid = 190 supp gid = 1000 Debug is true storeKey false useTicketCache true useKeyTab false doNotPrompt true ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is false principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false Acquire TGT from Cache >>>KinitOptions cache name is /tmp/krb5cc_1000 Principal is null null credentials from Ticket Cache [Krb5LoginModule] authentication failed Unable to obtain Princpal Name for authentication [UnixLoginModule]: added UnixPrincipal, UnixNumericUserPrincipal, UnixNumericGroupPrincipal(s), to Subject
... View more
03-30-2017
06:57 AM
It seems that JDK was not able to load kerberos ticket from cache. $ export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true $HADOOP_OPTS" $ hdfs dfs -ls / 2> /tmp/hdfsls.txt Java config name: null Native config name: /etc/krb5.conf Loaded from native config >>>KinitOptions cache name is /tmp/krb5cc_1000 I should see extra KRB debug, but found none.
... View more
03-30-2017
06:10 AM
P.S. I am on MIT KRB 1.14 version.
... View more
03-29-2017
09:34 PM
I am using CM API installing a CDH cluster on AWS with MIT KDC and JDK 1.8u121. From CM UI, keberos is working fine. I checked CM kerberos encryption types and they match those defined in kdc.conf. $ sudo cat /var/kerberos/krb5kdc/kdc.conf [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] AWS = { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal } $ klist -ef Ticket cache: KEYRING:persistent:1000:1000 Default principal: wzhu@AWS Valid starting Expires Service principal 03/30/2017 00:20:37 03/31/2017 00:20:37 krbtgt/AWS@AWS Flags: FI, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 $ hdfs dfs -ls / ... ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ip-10-1-30-107.us-west-1.compute.internal/10.1.30.107"; destination host is: "ip-10-1-30-107.us-west-1.compute.internal":8020; ...skipping... at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:730) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:726) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Kerberos
-
Security
03-27-2017
02:39 PM
Thanks. For cloud deployment, we definately want SSL/TLS for backend DBs.
... View more
03-23-2017
08:45 AM
I can't find any instruction on this. Is there anyway to enable SSL/TLS between CM services such as CM, HIVE, Navigator, etc. and backend DB such as mysql, postgres and oralce?
... View more
Labels:
- Labels:
-
Cloudera Manager
03-23-2017
08:41 AM
1 Kudo
Resolved by using `%`.* in the grant statement which removes mysql database. AWS RDS will not let us touch this database on PAAS offering.
... View more
03-21-2017
10:49 AM
Since AWS RDS MySQL instance doesn’t grant the initial DB user the real ‘root’ privileges, I need specific instructions on AWS RDS setup. I can’t run the following command to create a temp user. ( https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_installing_configuring_dbs.html#cmig_topic_5_1 😞 mysql> grant all on *.* to 'temp'@'%' identified by 'temp' with grant option; I had to modify with the following: mysql> grant all on `%`.* to 'temp'@'%' identified by 'temp' with grant option; [ec2-user@ip-x ~]$ sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -h $myDB-endpoints -utemp -ptemp --scm-host ip-x.us-west-1.compute.internal scm scm scm JAVA_HOME=/usr/java/jdk1.8.0_121 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.8.0_121/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Unable to login using supplied username/password. [ main] DbCommandExecutor ERROR Error when connecting to database. java.sql.SQLException: Access denied for user 'scm'@'myIP' (using password: YES) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)[mysql-connector-java.jar:5.1.41] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)[mysql-connector-java.jar:5.1.41] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)[mysql-connector-java.jar:5.1.41] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)[mysql-connector-java.jar:5.1.41] at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)[mysql-connector-java.jar:5.1.41]
... View more
Labels:
- Labels:
-
Cloudera Manager
02-21-2017
08:40 AM
Thanks. With data at-rest encryption, we have to add hdfs user access to all encryption zones. Is this by default?
... View more
02-15-2017
01:12 PM
Labels:
- Labels:
-
Cloudera Manager
02-14-2017
07:21 AM
I tried both v2.2 and v2.3. Both don't work with MySQL 5.7.
... View more
02-14-2017
07:20 AM
This is a brand new installation. MySQL 5.7 is installed in EC2 instance by myself, not RDS.
... View more
02-09-2017
05:13 AM
Here is application.log for v2.3. [2017-02-08 12:22:06.301 -0500] INFO [main] - application: Initializing Spring embedded WebApplicationContext ...skipping... at org.flywaydb.core.internal.command.DbMigrate.access$800(DbMigrate.java:46) at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:207) at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:156) at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72) at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:156) at org.flywaydb.core.Flyway$1.execute(Flyway.java:1059) at org.flywaydb.core.Flyway$1.execute(Flyway.java:1006) at org.flywaydb.core.Flyway.execute(Flyway.java:1418) at org.flywaydb.core.Flyway.migrate(Flyway.java:1006) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1713) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1650) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1579) ... 25 common frames omitted Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Invalid default value for 'CREATED_AT' at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.Util.getInstance(Util.java:408) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:943) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
... View more
02-09-2017
05:06 AM
Tried latest 2.3, failed at the same spot.
... View more
02-09-2017
05:03 AM
thanks. I will give a try.
... View more
02-07-2017
06:46 AM
Hue shell is not working since MySQL crendentails are not in /etc/hue/conf/hue.ini.
... View more
02-06-2017
08:23 PM
If I run the command out of box as user hue, I get the exception stating not connecting to DB. Since hue.ini in /etc/hue/conf has no info on MySQL, I tried to use the one .ini used by HUE process. However, the password in .ini was hidden. I could reset the hue user password in MySQL database, but it is a little too much. I don't have the hue admin user and password since I didn't set up it. I am taking over the cluster now and tried to create another super user to replace the existing admin user.
... View more
02-06-2017
10:13 AM
It works if I use embedded DB (http://gethue.com/password-management-in-hue). However, if I am using external DB, /etc/hue/conf/hue.ini doesn't contain the credentails for the external DB. I did "sudo find / -name hue.ini" to locate the latest hue.ini, but it doesn't contain the password either. It actually calls a script to get the password. What should I set before calling "build/env/bin/hue createsuperuser" to make it work?
... View more
Labels:
- Labels:
-
Cloudera Hue
02-04-2017
03:34 PM
Since "Import Kerberos Account Manager Credentials" tab show blank for both User name and Password, I have no clue which user is used. Basically I will create a new account manager then.
... View more
01-26-2017
07:22 PM
Missed JCE configuration. All is good.
... View more
01-26-2017
05:07 PM
NN indicates the authentication failure is due to the following: 2017-01-26 20:04:09,861 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8022: readAndProcess from client 10.3.1.23 threw e xception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechani sm level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]]
... View more
01-26-2017
02:47 PM
I am trying to bootstrap a CDH 5.9.1 cluster with MIT KDC in AWS. The same configuration works without kerberos. The application log didn't show any error. The bootrap command failed as follows: ... * Enabling Kerberos ............................................................................................................................................................ done * Calling firstRun on cluster m7test ... done * Waiting for firstRun on cluster m7test .............................................................................................................................................................. done * Collecting diagnostic data ................................................................................................................ done * Cloudera Manager 'First Run' command execution failed: Failed to perform First Run of services. ... I logged into CM and saw HDFS data nodes all failed to start. DN failed to authenticate with NN.
... View more
Labels:
- Labels:
-
Kerberos
01-25-2017
08:11 AM
I modified application.properties as follows to support MySQL: lp.database.type: mysql lp.database.username: clouderadirector lp.database.password: TopSecret lp.database.host: 10.3.1.197 lp.database.port: 3306 lp.database.name: clouderadirectordb (brand new database) After the modification, Cloudera Director failed to start due to flyway exception on migration: first it said no migration necessary, but later indicated a failed migration. 2017-01-24 17:18:06] INFO [main] - o.f.c.i.dbsupport.DbSupportFactory: Database: jdbc:mysql://10.3.1.197:3306/clouderadirectordb (MyS QL 5.7) [2017-01-24 17:18:06] INFO [main] - o.f.core.internal.command.DbValidate: Validated 5 migrations (execution time 00:00.030s) [2017-01-24 17:18:06] INFO [main] - o.f.core.internal.command.DbMigrate: Current version of schema `clouderadirectordb`: 3.2.0.0.1 [2017-01-24 17:18:06] INFO [main] - o.f.core.internal.command.DbMigrate: Schema `clouderadirectordb` is up to date. No migration neces sary. [2017-01-24 17:18:06] INFO [main] - o.f.c.i.dbsupport.DbSupportFactory: Database: jdbc:mysql://10.3.1.197:3306/clouderadirectordb (MyS QL 5.7) [2017-01-24 17:18:06] INFO [main] - c.c.l.p.u.ReferentialIntegrityService: Disabling referential integrity for MYSQL [2017-01-24 17:18:06] INFO [main] - c.c.l.p.util.PipelinePurgeService: Purging PIPELINES [2017-01-24 17:18:06] INFO [main] - c.c.l.p.util.PipelinePurgeService: Purging PIPELINE_JOBS [2017-01-24 17:18:06] INFO [main] - c.c.l.p.util.PipelinePurgeService: Purging PIPELINE_VALUES [2017-01-24 17:18:06] INFO [main] - c.c.l.p.u.ReferentialIntegrityService: Enabling referential integrity for MYSQL [2017-01-24 17:18:06] INFO [main] - o.f.core.internal.command.DbValidate: Validated 15 migrations (execution time 00:00.057s) [2017-01-24 17:18:06] INFO [main] - o.f.core.internal.command.DbMigrate: Current version of schema `clouderadirectordb`: 3.2.0.0.1 [2017-01-24 17:18:06] WARN [main] - o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext: Exception encountered during context init ialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [com/cloudera/launchpad/config/FlywayMigrationsConfig.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Schema `clouderadirectordb` contains a failed migration to version 3.2.0.0.1 ! [2017-01-24 17:18:06] WARN [main] - c.c.l.c.security.CipherSchemeFactory: Allowing cipher scheme to be set again, prior scheme is com. cloudera.launchpad.common.security.TripleDESCipher@5215cd9a [2017-01-24 17:18:06] INFO [main] - o.e.j.server.handler.ContextHandler: Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@bc57b40{/,file :/tmp/jetty-docbase.8174602001888934659.7189/,UNAVAILABLE} [2017-01-24 17:18:06] ERROR [main] - o.s.boot.SpringApplication: Application startup failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [com/cloudera/launchpad/config/FlywayMigrationsConfig.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Schema `clouderadirectordb` contains a failed migration to version 3.2.0.0.1 ! [2017-01-24 17:18:06] WARN [main] - c.c.l.c.security.CipherSchemeFactory: Allowing cipher scheme to be set again, prior scheme is com. cloudera.launchpad.common.security.TripleDESCipher@5215cd9a [2017-01-24 17:18:06] INFO [main] - o.e.j.server.handler.ContextHandler: Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@bc57b40{/,file :/tmp/jetty-docbase.8174602001888934659.7189/,UNAVAILABLE} [2017-01-24 17:18:06] ERROR [main] - o.s.boot.SpringApplication: Application startup failed
... View more
Labels:
- Labels:
-
Security
01-25-2017
07:31 AM
Thanks. As a workaround, I just disable password validation via ansible script right after mysql initial installation.
... View more
01-24-2017
07:53 PM
MySQL 5.7 is listed as supported DB version. However, Cloudera Director generates passwords not suitable for 5.7 standard. Application log is shown as follows: [2017-01-24 22:46:13] INFO [io-thread-38] - ssh:10.3.1.154: [ main] DbProvisioner ERROR Stack Trace: [2017-01-24 22:46:13] INFO [io-thread-38] - ssh:10.3.1.154: java.sql.SQLException: Your password does not satisfy the current policy requirements [2017-01-24 22:46:13] INFO [io-thread-38] - ssh:10.3.1.154: at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1078)[mysql-connec tor-java.jar:]
... View more
01-05-2017
02:18 PM
Same thing happened to me. Used curl on edge node without any issue.
... View more
01-04-2017
08:39 AM
I am not using iptables, using firewalld instead on Centos 7.x. The error I saw is caused by Hive doing select count(*) on a table, and the log indicates that the communication is between two slave nodes, not between edge node and AM. Please check your test setting. If your range is large enough, some jobs might succeed. With my current setting of 6000 port range, some jobs failed, and some succeeded by retrying and hitting the port within the range.
... View more
01-04-2017
08:20 AM
Redeployed the client configuration from CM. Checked both yarn and hive configuration, both mapred-site.xml files have the correct configuration reflected.
... View more
01-04-2017
07:49 AM
Thanks for your quick response. I did the quick test after putting the client side setting, but it still doesn't work. I still saw MR job failed due to No Route to Host from slave 1 to slave 2 on port not defined within the set range.
... View more
01-04-2017
07:34 AM
Could you check if this JIRA is fixed by Cloudera? https://issues.apache.org/jira/browse/MAPREDUCE-6338
... View more