Hi, The below properties are working in hive shell but not in beeline and getting Error: Error while processing statement: Cannot modify mapred.compress.map.output at runtime. It is not in list of params that are allowed to be modified at runtime SET mapred.output.compress SET mapred.compress.map.output How to whitelist these properties for beeline. Thank you. ... View more

Hi, SET hive.warehouse.data.skipTrash = true; Query returned non-zero code: 1, cause: hive configuration hive.warehouse.data.skipTrash does not exists hive version : Hive 1.2 Is this property deprecated? Please let me know. Thanks ... View more

Can someone please let us know whether we can identify the missing and corrupted blocks using REST API or not? ... View more

Hi, Can anyone let us know the support matrix for Ranger KMS Luna SA HSM. Also, let us know whether HW officially support LUNA SA HSM 7 version or not. Currently, we have integrated Ranger KMS (HDP 3.1) with Luna SA HSM(HSM Model -> LunaSA 6.2.2,HSM Firmware Version -> 6.10.9) Thank you. ... View more

Hi, I'm trying to enable Ranger HA where Ranger admin & plugins ssl is enabled. I have configured load-balancer in one of the node and installed additional Ranger admin. But when I try to login to Ranger UI through load-balancer IP, I'm getting below issue. Load-balancer server IP address using https protocol: Node details: Ranger Admin: https://ranger_admin_IP1:6182 Additional Ranger Admin : https://ranger_admin_IP2:6182 Load-balancer : https://LB_IP:443 From load-balancer node, I have executed the below commands and could see the results as below. $curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k https://ranger_admin_IP1:6182/login.jsp 200 $curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k https://ranger_admin_IP2:6182/login.jsp 200 $curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k https://LB_IP/login.jsp 404 I have updated the Ranger External URL as "https://LB_IP:443" and I could see the below alert in Ranger Admin service. Please find the Load-balancer configuration below and seems to be issue with configuration. Self-signed certs(load-balancer node): $ ls -lrt /usr/local/apache2/conf/ -rw-r--r-- 1 root root  1679 Feb 20 07:45 server.key -rw-r--r-- 1 root root  1338 Feb 20 10:22 server.crt httpd.conf: ServerRoot "/usr/local/apache2" Listen 88 LoadModule authn_file_module modules/mod_authn_file.so LoadModule authn_core_module modules/mod_authn_core.so LoadModule authz_host_module modules/mod_authz_host.so LoadModule authz_groupfile_module modules/mod_authz_groupfile.so LoadModule authz_user_module modules/mod_authz_user.so LoadModule authz_core_module modules/mod_authz_core.so LoadModule access_compat_module modules/mod_access_compat.so LoadModule auth_basic_module modules/mod_auth_basic.so LoadModule socache_dbm_module modules/mod_socache_dbm.so LoadModule reqtimeout_module modules/mod_reqtimeout.so LoadModule filter_module modules/mod_filter.so LoadModule mime_module modules/mod_mime.so LoadModule log_config_module modules/mod_log_config.so LoadModule env_module modules/mod_env.so LoadModule headers_module modules/mod_headers.so LoadModule setenvif_module modules/mod_setenvif.so LoadModule version_module modules/mod_version.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule slotmem_shm_module modules/mod_slotmem_shm.so LoadModule ssl_module modules/mod_ssl.so LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so LoadModule unixd_module modules/mod_unixd.so LoadModule status_module modules/mod_status.so LoadModule autoindex_module modules/mod_autoindex.so LoadModule dir_module modules/mod_dir.so LoadModule alias_module modules/mod_alias.so Include conf/extra/httpd-ssl.conf Include /usr/local/apache2/conf/ranger-lb-ssl.conf httpd-ssl.conf: Listen 443 SSLProtocol all -SSLv3 SSLProxyProtocol all -SSLv3 <VirtualHost _default_:443> DocumentRoot "/usr/local/apache2/htdocs" ErrorLog "/usr/local/apache2/logs/error_log" TransferLog "/usr/local/apache2/logs/access_log" SSLEngine on SSLCertificateFile "/usr/local/apache2/conf/server.crt" SSLCertificateKeyFile "/usr/local/apache2/conf/server.key" ranger-lb-ssl.conf: <VirtualHost *:443>         SSLEngine on         SSLProxyEngine on         SSLCertificateFile /usr/local/apache2/conf/server.crt         SSLCertificateKeyFile /usr/local/apache2/conf/server.key         #SSLCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem         #SSLProxyCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem         SSLVerifyClient optional         #SSLVerifyClient require         SSLOptions +ExportCertData         SSLProxyVerify none         SSLProxyCheckPeerCN off         SSLProxyCheckPeerName off         SSLProxyCheckPeerExpire off         ProxyRequests off         ProxyPreserveHost off         Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED         <Proxy balancer://rangercluster>                BalancerMember https://ranger_admin_IP1:6182 loadfactor=1 route=1                BalancerMember https://ranger_admin_IP2:6182 loadfactor=1 route=2                 Order Deny,Allow                 Deny from none                 Allow from all                 ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off         </Proxy>         # balancer-manager         # This tool is built into the mod_proxy_balancer         # module and will allow you to do some simple         # modifications to the balanced group via a gui         # web interface.         <Location /balancer-manager>                 #RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"                 SetHandler balancer-manager                 Order deny,allow                 Allow from all         </Location>        ProxyPass /balancer-manager !        ProxyPass / balancer://rangercluster/        ProxyPassReverse / balancer://rangercluster/ </VirtualHost> $cat /usr/local/apache2/logs/access_log xx.xx.xx.xx - - [20/Feb/2019:17:22:08 +0000] "GET /login.jsp HTTP/1.1" 404 207 xx.xx.xx.xx - - [20/Feb/2019:17:22:57 +0000] "GET /login.jsp HTTP/1.1" 404 207 xx.xx.xx.xx - - [20/Feb/2019:17:23:08 +0000] "GET /login.jsp HTTP/1.1" 404 207 $cat /usr/local/apache2/logs/error_log [Wed Feb 20 12:57:00.255350 2019] [ssl:error] [pid 17473] [client xx.xx.xx.xx:65502] AH02039: Certificate Verification: Error (20): unable to get local issuer certificate [Wed Feb 20 13:28:23.544429 2019] [mpm_prefork:notice] [pid 95071] AH00173: SIGHUP received.  Attempting to restart [Wed Feb 20 13:28:23.559039 2019] [mpm_prefork:notice] [pid 95071] AH00163: Apache/2.4.16 (Unix) OpenSSL/1.0.2k-fips configured -- resuming normal operations [Wed Feb 20 13:28:23.559063 2019] [core:notice] [pid 95071] AH00094: Command line: '/usr/local/apache2/bin/httpd' [Wed Feb 20 13:29:14.414809 2019] [mpm_prefork:notice] [pid 95071] AH00173: SIGHUP received.  Attempting to restart [Wed Feb 20 13:29:14.430214 2019] [mpm_prefo Any help would be greatly appreciated!!!. Thank you. ... View more

Hi, We have installed Ranger KMS and migrated the master key from Ranger KMS Database to HSM. I have a few doubts here. Would you please help to clarify. During the HDFS write operation, HDFS client tells the Name Node that it wants to write a file to the EZ, the Name Node requests the KMS to return a EDEK. The KMS does this by generating a unique DEK and DEK will be encrypted using EZK which is present in Ranger database. This EDEK is returned to the NameNode and stored along with the the file’s metadata. 1. How frequently Ranger KMS(EZK) connects to HSM(master key)? 2. What is the load on HSM from Ranger KMS i.e. no. of crypto operations/sec/hour/day? Whenever, there is a Read/Write operation on HDFS encryption zones, I could see the below methods being invoked in KMS log kms.log Read Operation: INFO KMS - Entering decryptEncryptedKey method DEBUG RangerPluginClassLoader - <== RangerPluginClassLoader.activate() DEBUG RangerKmsAuthorizer - ==> RangerKmsAuthorizer.assertAccess(testkey1, useid@domain (auth:KERBEROS), DECRYPT_EEK) DEBUG RangerKmsAuthorizer - ==> RangerKmsAuthorizer.hasAccess(DECRYPT_EEK, userid@domain (auth:KERBEROS) , testkey1) INFO KMS - Exiting handleEncryptedKeyOp method Write operation: INFO KMS - Entering generateEncryptedKeys method INFO KMS - Entering decryptEncryptedKey method DEBUG RangerKmsAuthorizer - ==> RangerKmsAuthorizer.assertAccess(testkey1, userid@domain (auth:KERBEROS), DECRYPT_EEK) DEBUG RangerPluginClassLoader - ==> RangerPluginClassLoader.activate() DEBUG RangerPluginClassLoader - <== RangerPluginClassLoader.activate() DEBUG RangerKmsAuthorizer - ==> RangerKmsAuthorizer.assertAccess(testkey1, userid@domain (auth:KERBEROS), DECRYPT_EEK) INFO KMS - Exiting handleEncryptedKeyOp method. 3. Once the decryptEncryptedKey method is invoked, request will be sent to HSM for decrypting EZK using master key. Please correct me if I'm wrong. 4. Whenever we perform create new EZ key/Rollover key operation from Ranger KMS UI, request will be sent to HSM for encrypting/decrypting the EZ key using master key. Please correct me if I'm wrong. Thank you. ... View more

Hi, [Ambari 2.7.3, HDP 3.1] In Active Directory Kerberized environment, I'm getting below issue when I try to access Namenode UI, RM UI and Job histroy UI from Ambari Error: HTTP ERROR 403 problem accessing /index.html. Reason: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96) krb5.conf: max_life = 30d default_tgs_enctypes = aes128-cts arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-hmac-sha1 aes256-cts default_tkt_enctypes = aes128-cts arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-hmac-sha1 aes256-cts permitted_enctypes = aes256-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-cbc-md4 allow_weak_crypto = yes klist: $ls -lrt /etc/security/keytabs/spnego.service.keytab -r--r-----. 1 root hadoop 433 Feb 9 11:59 /etc/security/keytabs/spnego.service.keytab $klist -ket /etc/security/keytabs/spnego.service.keytab Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (arcfour-hmac) 0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (des-cbc-md5) 0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (aes256-cts-hmac-sha1-96) 0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (des3-cbc-sha1) 0 02/09/2019 07:40:04 HTTP/hostname_fqdn@Crealm (aes128-cts-hmac-sha1-96) kinit: $kinit -kt /etc/security/keytabs/spnego.service.keytab $(klist -kt /etc/security/keytabs/spnego.service.keytab|sed -n "4p"|cut -d" " -f7) # klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: HTTP/hostname_fqdn@realm Valid starting Expires Service principal 02/09/2019 12:53:05 02/09/2019 22:53:05 krbtgt/realm@realm renew until 02/16/2019 12:53:05 I have re-generated the spnego keytab in all the hosts from ambari UI but did not help. Would you please help this. Thank you. ... View more

Hi I'm getting below issue while starting Timeline Service V2.0 in HDP 3.1 org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /atsv2-hbase-secure1/tokenauth/keys at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:549) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:528) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1199) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1177) 2019-02-07 06:10:29,463 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=6318 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server sl73caeapd044.visa.com,17020,1549478310629 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1487) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) , details=row 'prod.timelineservice.entity' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=sl73caeapd044.visa.com,17020,1549462163537, seqNum=-1 2019-02-07 06:10:33,487 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=7, retries=36, started=10342 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server sl73caeapd044.visa.com,17020,1549478310629 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1487) Please help on this. Thank you. Regards, Sampath ... View more

Hi, To identify the zookeeper leader, I can use the below command in CLI. $echo stat | nc zk_hostname 2181 | grep Mode Mode: follower Is it possible to achieve this using REST API calls? Thank you. Regards, Sampath ... View more

Hi, We have configured tez as hive execution engine by setting hive.execution.engine=tez in ambari. Hive jobs are running fine and are getting submitted and successful in Resource Manager UI. But I am not able to view Job history Server UI of these jobs. However, I can see normal mapreduce jobs in job history server UI in detail. When I click the History from Tracking URL field in Resource Manager UI, it's redirecting to the below URL and showing nothing. Tracking URL: https://hosts_fqdn:8080/#/main/view/TEZ/tez_cluster_instance?viewPath=%2F%23%2Ftez-app%2Fapplication_1544533454600_0047 Cluster is SSL enabled and Kerberized. In the above URL, If I replace port from 8080 to 9443 manually, then I could see the Tez view as in the attached screenshot. Properties set in tez-site.xml: tez.tez-ui.history-url.base=https://hosts_fqdn:9443/#/main/view/TEZ/tez_cluster_instance tez.am.tez-ui.history-url.template=__HISTORY_URL_BASE__?viewPath=%2F%23%2Ftez-app%2F__APPLICATION_ID__ I could see the below warnings in the mapreduce-job-history-server log as well. WARN mortbay.log (Slf4jLog.java:warn(76)) - SSL renegotiate denied: java.nio.channels.SocketChannel[connected local=<ip> remote=<ip> Please help to fix this issue. Thank you. ... View more