@bgooley Does oozie email action needs a firewall to be opened between slave nodes to smtp or just the oozie server to smtp? ... View more

var/lib/kms-keytrustee/keytrustee/.keytrustee folder on both the kms hosts should match and unfortunately they do not in our cluster, So if a key create request goes to one kms host and retrieval goes to another kms host the command fails.     [root@host]# md5sum /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg fec74c82e3da7f04f2acd36a937072b5  /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg   [root@host]# md5sum /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg 88483e6a8ee1d245d3c83b740fd43683  /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg     Used bdr tool to take a back up of encrypted zones in the same cluster, purged all keys, dropped all zones. Used rsync to sync /var/lib/kms-keytrustee/keytrustee/.keytrustee on both kms hosts, created all keys, zones and used bdr to restore the data from backup. Everything looks good now!! ... View more

@lhebert It is turning very interesting....I noticed hadoop key list -provider kms://https@host[1-2]:16000/kms both give different results.   hadoop key create test-key, I cannot see test-key on kms host1 if the request hit kms host2.  ... View more

Hi @lhebert   We are using Cloudera licensed KMS. We have manually syncronized .keytrustee folder while setting the KMS. Can we manually sync them again now? and how do we know which one of Active/Passive KMS is corrupted?     ... View more

@bgooley any ideas where to start debug from? ... View more

@bgooley   Fantastic!...Both of the changes did the magic 🙂 Kudos!!!     ... View more

@bgooley    ldaps:// smoothly generated the accounts,  unicodepwd, usersharedcontrol must be needed for AD to create users and for those ldaps is mandatory. I think it should be added to Cloudera documentation.   kinit -kt /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep zookeeper| tail -1`/zookeeper.keytab principal [root@hostname sbalusu_c@example.com]# klist Ticket cache: KEYRING:persistent:0:0 Default principal: zookeeper/hostname.example.com@EXAMPLE.COM Valid starting Expires Service principal 07/12/2018 16:12:00 07/13/2018 02:12:00 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 07/19/2018 16:11:59   I have added udp_preference_limit=1 but still getting the earlier error: 4:17:47.210 PM ERROR QuorumPeerMain Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: ICMP Port Unreachable at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)   Documentation does not talk about ICMP port, I tried enabling debug mode but could not figure out which port it is looking for. Any idea?     Thanks & Regards, Siva       ... View more

@bgooley   I was able to create the principles when I commented out UserAccountControl, But the problem with this is the account getting locked and the below property set:   If use UserAccountControl attribute I get   ldap_add: Server is unwilling to perform (53) additional info: 0000052D: SvcErr: DSID-031A1248, problem 5003 (WILL_NOT_PERFORM), data 0   But when creating the user through AD UI everything is good, That states that there is no access/policy blocking issue. It seems conflict between openldap vs AD. Any suggestions/directions for more testing?   Thanks, Siva   ... View more

@bgooley    kinit -kt /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep zookeeper| tail -1`/zookeeper.keytab zookeeper/host@EXAMPLE.COM kinit: Client's credentials have been revoked while getting initial credentials     I was able to do a kinit as my self from the same host and There are no connection issues. Never spent this much time on enabling kerberos 😛 ... View more

@bgooley Thanks!! We are in the process of enabling ssl, Once done we will switch to ldaps.   I have changed the unicodepwd to userpasswd, Cloudera manager was able to create the principals, But while I was starting the services, zookeeper threw the following error:   5:05:02.212 PM ERROR QuorumPeerMain Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: ICMP Port Unreachable at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79) ... View more