Hi ,   I am getting below error when i ran the below command    1st question bash-4.1$ yarn logs -applicationId application_55261206146721_39976 18/05/23 23:10:20 INFO client.RMProxy: Connecting to ResourceManager at xxxxxxxxxxxxxxxxxxxxxxxxx /tmp/logs/yarn/logs/ application_55261206146721_39976 does not exist. Log aggregation has not completed or is not enabled.   i already enabled    yarn.log-aggregation-enable=true yarn.nodemanager.remote-app-log-dir=/tmp/logs yarn.log-aggregation.retain-seconds=3 days     2nd question: does below command syntax  supported in cloudera?    yarn logs -applicationId application_332332323073474_0002 -show_application_log_info yarn logs -applicationId application_1 332332323073474_0002 -show_container_log_info   Regards Jacqueline   ... View more

Hi Kafka Gurus, Please help me in understand the difference between these parameters 1) Send.buffer.bytes vs batch.size vs Max.request.size 2) Queue.buffer.max.ms vs linger.ms Regards JJ ... View more

Can someone distinguish yarn container vs yarn Child Regards JJ ... View more

Hi Gurus. Our Ambari Metrics collector goes down. Our is Kerberos env. Ambari Version is :Version2.4.2.0 i see below errors in ambari-metrics-collector.log i see belwo error FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix. Caused by: org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG Caused by: org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG in hbase-ams-master-xxxxxxxxxx.log 2017-12-23 07:40:57,717 ERROR [xxxxxx,61300,1513971592600_ChoreService_1] master.BackupLogCleaner: Failed to get hbase:backup table, therefore will keep all files org.apache.hadoop.hbase.TableNotFoundException: hbase:backup I implemented below procedure but no luck https://community.hortonworks.com/questions/73577/problem-with-ambari-metrics.html 1) Turn on Maintenance mode 2) Stop Ambari Metrics 3) hadoop fs -rmr /ams/hbase/* 4) rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/* 5) [zk: localhost:2181(CONNECTED) 0] ls / [zk: localhost:2181(CONNECTED) 1] rmr /ams-hbase-unsecure 6) Start Ambari Metrics 7) Turn off Maintenance mode My setting are : hbase.rootdir = hdfs://hpdprod:8020/user/ams/hbase hbase.cluster.distributed=true timeline.metrics.service.operation.mode=distributed Any help is really appreciate Thanks JJ ... View more

Hi, We did our HDP stack install as root user. Little bit concerned about the files and directories -- permissions and ownership. Do we have any fixing scripts, which can set right permission and ownership? Regards JJ ... View more

unix ldapsearch workfine ,but ranger ldaptool is failing below ldapsearch works fine : ldapsearch -h free-ipa-dev-01.uat.txdc.datastax.com -x -b "dc=txdc,dc=datastax,dc=com" -W hadoopadmin but ranger ldaptool is failing : [root@dev-rng-001 ~]# cd /usr/hdp/current/ranger-usersync/ldaptool [root@dev-rng-001 ldaptool]# ./run.sh -d users Ldap url [ldap://ldap.example.com:389]: ldaps://free-ipa-dev-01.uat.txdc.datastax.com:636 Bind DN [cn=admin,ou=users,dc=example,dc=com]: hadoopadmin Bind Password: User Search Base [ou=users,dc=example,dc=com]: dc=txdc,dc=datastax,dc=com User Search Filter [cn=user1]: cn=* Reading ldap properties from input.properties ERROR: Failed to perfom ldap bind. Please verify values for ranger.usersync.ldap.binddn and ranger.usersync.ldap.ldapbindpassword javax.naming.CommunicationException: simple bind failed: free-ipa-dev-01.uat.txdc.datastax.com:636 [Root exception is javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target] Can you please help Regards JJ ... View more

Hi , I came to know that if there are any under-replicated blocks, it will be taken care automatically by HDFS to match up with the replication factor. Question is, If the disk fails, 1) how soon this process of identifying the under replicated block will kick off , 2) When hdfs will kick off the process to replication of under-replicated blocks? 3) Is there any way to halt the auto-correction of under replication blocks. What parameters we need to set. We don't want to kick off the process of auto-correction ,immediately. We would like to start this procedure, say after 5 hrs from the time of disk failure. How to configure it. Thanks JJ ... View more

Right now we have only one topic , with 3 partitions and with a replicaiton factor of 3. My log.dirs has 4 mount point or paths or directories. Only 1 mount point is being used out of 4 . Other 3 mount point or paths or directories , mentioned in log.dirs are not been used. How can i make all the mount / directories , get used. Thanks JJ ... View more

Hi , policymgr_external_url parameter ( External URL ) , when pointed the load balancer vip , Ranger sink with Active Directory isnot happening. We have successfully setup Ranger Admin HA . External URL was pointing to load balancer vip.With this new configuration, Ranger sink with Active Directory is not happening. if we replace the load balance vip with actual host name , able get all the Active Directory user list. is there any problem with setting up load balancer vip ? , i am not getting. http://dev-rag.dataquest.com:6080” is not working for AD user sink. can we have multiple hostname for policymgr_external_url parameter ( External URL ) like“ http://dev-rag-001.dataquest.com,dev-rag-002.dataquest.com:6080” ? please share your thoughts and experiences Regards JJ ... View more

Ranger is not getting updated with all the Active Directory accounts. select * from x_portal_user; ---> does not show Active Directory accounts. ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User Group Info -- Connection reset Kerberos was successfully implemented. After enabling kerberos following values we set for ranger. 1) ranger.usersync.kerberos.keytab : /etc/security/keytabs/rangerusersync.service.keytab 2) ranger.usersync.kerberos.principal : rangerusersync/_HOST@DEV.DATAQUEST.COM 3) ranger.usersync.policymgr.username : rangerusersync But klist for keytab show the principal as : rangerusersync/rng-node1.dev.dataquest.com@DEV.DATAQUEST.COM Note : we do not have any ssl implementation and certificate or keystore adn thruststore is not needed for us. below is the error message from usersync.log ``` 07 Sep 2017 05:02:42 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User Group Info : com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Connection reset at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getUsergroupInfo(PolicyMgrUserGroupBuilder.java:567) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$500(PolicyMgrUserGroupBuilder.java:72) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$2.run(PolicyMgrUserGroupBuilder.java:539) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$2.run(PolicyMgrUserGroupBuilder.java:535) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addUserGroupInfo(PolicyMgrUserGroupBuilder.java:535) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:340) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:327) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:706) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:249) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) ... 15 more 07 Sep 2017 05:02:42 INFO LdapUserGroupBuilder [UnixUserSyncThread] - groupSearch is enabled, would search for groups and compute memberships 07 Sep 2017 05:02:43 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getGroups() completed with group count: 0 ``` Any help is really appreciated. ... View more

Hi Experts, Can i more the 1 Active Name Node in HDP Cluster? . Prefer to have 1 Stand by Name node for multiple Active Name Node. Does this configuration is possible? Thanks JJ ... View more

Hi , How can we modify below properties to new location hadoop_pid_dir_prefix yarn_pid_dir_prefix hive_pid_dir hcat_pid_dir All these parameters are grayed out and does not let me to modify. I want to modify their exisitng configuration to my custom folder / directory. Thanks JJ ... View more

how can we modify the below ranger parameters. •ranger_admin_log_dir •ranger_pid_dir •ranger_usersync_log_dir •ranger.tagsync.logdir Rigth now in ambari these are grayed out and does not let me to modify From the os command prompt ,i tried to modify below two files /usr/hdp/current/ranger-admin/ews/ranger-admin-services.sh /usr/hdp/current/ranger-usersync/ranger-usersync-services.sh example from pidf=/var/run/ranger/rangeradmin.pid to pidf=/var/log/hdp/ranger/run/rangeradmin.pid There after Ranger servers failed to come up Your help is needed ... View more