@BorjaRodriguez that looks like a different known issue that we've seen with incremental stats on tables with large numbers of columns. It's fixed in CDH5.13+. ... View more

Yes you can do it in multiple ways. For example you can use Group by or Distinct. If you want to find duplicities on the subset of the columns (i.e. find all rows where customer_id is duplicate) I would recommend to use a Group by. ... View more

Hi @bgooley, Thank you for the response.   Here are the contents - On CM host below CA cert is generated [root@hostname pki]# ls -ltr total 20 -rw-r--r-- 1 root root 1121 Sep 2 12:27 hostname-server.csr -rw-r--r-- 1 root root 1834 Sep 2 12:27 ca-key -rw-r--r-- 1 root root 1314 Sep 2 12:27 ca-cert -rw-r--r-- 1 root root 4198 Sep 2 12:28 hostname.jks lrwxrwxrwx 1 root root 7 Sep 6 10:21 4ba83cc1.0 -> ca-cert [root@hostname pki]# pwd /opt/cloudera/security/pki -> ca cert is imported to keystore along with jsse certs on CM host and same copied to one of the agent hosts. -> copied the contents for ca-cert generate through - openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650 -> created the sym link for the ca-cert, updated the parameters in /etc/cloudera-scm-agent/config.ini along verify_cert_dir -> Here is the output from verification - [root@hostname1.domain cloudera-scm-agent]# openssl s_client -connect hostname:7183 -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem < /dev/null CONNECTED(00000003) depth=0 C = Unknown, ST = Unknown, L = Unknown, O = Unknown, OU = Unknown, CN = hostname verify return:1 --- Certificate chain 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- Server certificate -----BEGIN CERTIFICATE----- MIIDmTCCAoGgAwIBAgIEFdAE2jANBgkqhkiG9w0BAQsFADB9MRAwDgYDVQQGEwdV bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEwHwYDVQQDExhjc2VraGFy MDIucWU5LmppZ3Nhdy5jb20wHhcNMTgwOTAyMTkyNzAyWhcNMjgwODMwMTkyNzAy WjB9MRAwDgYDVQQGEwdVbmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQH EwdVbmtub3duMRAwDgYDVQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEw HwYDVQQDExhjc2VraGFyMDIucWU5LmppZ3Nhdy5jb20wggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQCg4ailB41WoGo6h3fepc6wILw37WaRZ3IApxj0ZTO0 n8po151baw0IEk1X+lYXAkuXloycCKsFUhjiOU2AF2iBbVOYXyA9IB5a9a+b507r EG1NqdJrdN6k4kZrsv+91CizYunjNNheDIgUv/gA7U/sSbvvchXUlN+j4y9LpwKH wcbY/kKLZV9EoNeyrJANccHYiHRQgPLfEcQoHJtDRuWt0TIDqZOR+Xe6fICLfDiC 8WRXqpYKRfFoLVwzFEJuilZXT/J9obzhRgxt9sF7eVV86kFa2EIEZ7aAzZGdIeVT s3RpUO/iSq2X5DxQ7BH8Duc6kgfuaAIOCuANeEGO7trxAgMBAAGjITAfMB0GA1Ud DgQWBBTPmi/hBkF9xJxiN4bvB2QehVuM5TANBgkqhkiG9w0BAQsFAAOCAQEAJEgp 747BfFBIKVPoBCOnxopLM0iJ0jhOTMPo2gld8XoHdH52UOKPlnTsYxqRv9qcU2m8 zjlNxpzT2LMEMGWwk8cP+4ikGrW1vI/bUZOVwFg2pDeho/AE9IOgdn6kd9OQXnL+ D2AB8woJwLWEQ755CEETjvk84kC4BVMMCAB4tDhwAEmzhsehg5B1FzVGfNE5k7fS Ey+ZkNMr/9omWYd4lzHja8pCBCISUCVU7c4ybnEdvLfhyGk/uoNzE2Tf11OsmNtj Fqc4Wbfr9BmcEzDWP6pO5ID6hOu93JaKPmMiW6ofzoJfUdF9Mj/q8pEBc88dqpbf aMvL2RAdfPcFFqSEWA== -----END CERTIFICATE----- subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 1409 bytes and written 415 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5B91D501E15D42E800371EBAFE7BF3FD673EEEDA1A10E30CBEC808C2431F2325 Session-ID-ctx: Master-Key: A0DD11CA122343D962AFE236893EC2F00371D37DF3BDC340AB3F1CCDC25C8E48F4DC28255A6CC1926654D1708FE23B9A Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1536283905 Timeout : 300 (sec) Verify return code: 0 (ok) --- DONE restarted CM server and agent Please advise if I'm missing anything. ... View more

This is related to the JobHistoryServer log reported earlier. Please ensure/perform the following items for JHS and job completions to thoroughly work: First: Ensure that 'mapred' and 'yarn' are part of the 'hadoop' group in common: ~> hdfs groups mapred ~> hdfs groups yarn Both command must include 'hadoop' in their outputs. If not, ensure they are added to that group name. Second, all files and directories under HDFS /tmp/logs aggregation dir (or whatever you've reconfigured it to use) and /user/history/* have their group set to 'hadoop' and not anything else: ~> hadoop fs -chgrp -R hadoop /user/history /tmp/logs ~> hadoop fs -chmod -R g+rwx /user/history /tmp/logs Note: ACLs suggested earlier are not required to resolve this problem. The group used on these dirs is what matters in the default state, and the group setup described above is how YARN and JHS daemon users share information and responsibilities with each other. You may remove any ACLs set, or leave them be as they are still permissive. ... View more

Thank you for your reply! I followed CDH post, then test two scenes: 1. Authentication success 2018-09-03 16:41:13,168 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=xxxxx sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3eb07fd3 Welcome to ZooKeeper! JLine support is enabled [zk: xxxxx(CONNECTING) 0] 2018-09-03 16:41:13,440 [myid:] - INFO [main-SendThread(xxxxx:2181):Login@294] - Client successfully logged in. 2018-09-03 16:41:13,441 [myid:] - INFO [Thread-1:Login$1@128] - TGT refresh thread started. 2018-09-03 16:41:13,445 [myid:] - INFO [Thread-1:Login@302] - TGT valid starting at: Mon Sep 03 16:40:47 CST 2018 2018-09-03 16:41:13,445 [myid:] - INFO [Thread-1:Login@303] - TGT expires: Tue Sep 04 02:40:47 CST 2018 2018-09-03 16:41:13,445 [myid:] - INFO [Thread-1:Login$1@182] - TGT refresh sleeping until: Tue Sep 04 01:10:18 CST 2018 2018-09-03 16:41:13,445 [myid:] - INFO [main-SendThread(xxxxx:2181):SecurityUtils$1@124] - Client will use GSSAPI as SASL mechanism. 2018-09-03 16:41:13,452 [myid:] - INFO [main-SendThread(xxxxx:2181):ClientCnxn$SendThread@975] - Opening socket connection to server xxxxx/xxxxx:2181. Will attempt to SASL-authenticate using Login Context section 'Client' 2018-09-03 16:41:13,456 [myid:] - INFO [main-SendThread(xxxxx:2181):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /xxxxx:33160, server: xxxxx/xxxxx:2181 2018-09-03 16:41:13,462 [myid:] - INFO [main-SendThread(xxxxx:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server xxxxx/xxxxx:2181, sessionid = 0x2659d1248f90274, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null WATCHER:: WatchedEvent state:SaslAuthenticated type:None path:null [zk: xxxxx(CONNECTED) 0] getAcl /znode1 'sasl,'zkcli@xxx : cdrwa 2. Authentication failed 2018-09-03 16:38:48,415 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=xxx sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3eb07fd3 Welcome to ZooKeeper! 2018-09-03 16:38:48,436 [myid:] - WARN [main-SendThread(xxxxx:2181):ClientCnxn$SendThread@957] - SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/etc/zookeeper/conf/jaas.conf' because of a RuntimeException: java.lang.SecurityException: java.io.IOException: /etc/zookeeper/conf/jaas.conf (No such file or directory) Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2018-09-03 16:38:48,438 [myid:] - INFO [main-SendThread(xxxxx:2181):ClientCnxn$SendThread@975] - Opening socket connection to server xxxxx/xxx:2181 WATCHER:: WatchedEvent state:AuthFailed type:None path:null JLine support is enabled 2018-09-03 16:38:48,500 [myid:] - INFO [main-SendThread(xxxxx:2181):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /xxx:33021, server: xxxxx/xxx:2181 2018-09-03 16:38:48,506 [myid:] - INFO [main-SendThread(xxxxx:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server xxxxx/xxxx:2181, sessionid = 0x2659d1248f90271, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: xxx(CONNECTED) 0] getAcl /znode1 'sasl,'zkcli@xxx : cdrwa zookeeper client can still get the znode data if the authentication is failed. Is there any way to check the authentication of session, not the inside znode? ... View more

Hi, I also faced similar issues while applying regex_replace() to only strings columns of a dataframe. The trick is to make regEx pattern (in my case "pattern") that resolves inside the double quotes and also apply escape characters. val pattern="\"\\\\[\\\\x{FFFD}\\\\x{0}\\\\x{1F}\\\\x{81}\\\\x{8D}\\\\x{8F}\\\\x{90}\\\\x{9D}\\\\x{A0}\\\\x{0380}\\\\]\"" val regExpr = parq_out_df_temp.schema.fields.map(x => if(x.dataType == StringType){"regexp_replace(%s, $pattern,'') as %s".format(x.name,x.name)} else x.name) val parq_out=parq_out_df_temp.selectExpr(regExpr:_*)   This worked fine for me !! ... View more