Cloudera Community
Harsh J
user rank
user rank
Mentor

Re: Cannot pass value from Hive query output direc...

by Expert Contributor in Support Questions
‎08-27-2015 09:17 AM
‎08-27-2015 09:17 AM
I retried the LCE configs after upgrading to CDH 5.4.4 - still doesn't work.  Both Hive-on-Hue and Oozie failed in MR.  One of the nodes have cache directory reconfigured:   drwxr-s--- 4 myao yarn 37 Mar 20 11:30 /s0/yarn/nm/usercache/myao drwxr-s--- 4 myao yarn 37 Mar 20 11:30 /s1/yarn/nm/usercache/myao drwxr-s--- 4 myao yarn 37 Mar 20 11:30 /s2/yarn/nm/usercache/myao   So, I will stick with the $HADOOP_USER_NAME workaround for now.     Cheers, Miles   ... View more

Re: Hbase coonection via Java Application - Cloude...

by Visitor in Support Questions
‎08-27-2015 06:35 AM
‎08-27-2015 06:35 AM
Thanks Harsh, I have upgraded client side Hbase library and client code as per updated hbase-client api version 0.99.1. Now it works fine.   ... View more

Re: Hbase utilizing more storage space while loadi...

by Visitor in Support Questions
‎08-26-2015 12:05 AM
‎08-26-2015 12:05 AM
Thanks . Block encoding and compression together helped to storage utilization. ... View more

Re: Permission denied when trying to open /webhdfs...

by Mentor Mentor in Support Questions
‎08-25-2015 11:42 PM
‎08-25-2015 11:42 PM
You are unfortunately impacted by a limitation described at: https://issues.apache.org/jira/browse/HDFS-5796 You will need to use a custom filter class to avoid this (i.e. to auth browsers separately from regular WebHDFS auth access, probably by checking UserAgent strings/etc.), by implementing a sub-class of https://github.com/cloudera/hadoop-common/blob/cdh5.4.4-release/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java Note that you can instead also opt to use the Hue File Browser, which is also more feature-rich in comparison. ... View more

Re: How Map and Reduce operations are actually car...

by Explorer in Support Questions
‎08-18-2015 01:01 AM
‎08-18-2015 01:01 AM
Thanks Harish ... View more

Re: HBASE ZOOKEEPER CONFIG CDH5.4.4

by Visitor in Support Questions
‎08-17-2015 09:38 AM
‎08-17-2015 09:38 AM
Thank you Harsh . This solution worked for me .   I have 21000 records at my source and I'm doing a UPSERT in Hbase using Java API . I'm not sure of Hbase is making additional connections to Zookeeper and each such instance is a FileDescriptor . Attached part of log . It took 10 minutes for the Job to complete for a Job with 4 reduces tasks .   2015-08-17 16:07:12,577 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2279f558 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 2015-08-17 16:07:12,577 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x2279f5580x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase 2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49332, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181 2015-08-17 16:07:12,580 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863f8, negotiated timeout = 60000 2015-08-17 16:07:12,584 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863f8 2015-08-17 16:07:12,585 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863f8 closed 2015-08-17 16:07:12,585 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down 2015-08-17 16:07:12,685 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5cdeb23b connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 2015-08-17 16:07:12,685 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x5cdeb23b0x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase 2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49335, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181 2015-08-17 16:07:12,689 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fa, negotiated timeout = 60000 2015-08-17 16:07:12,693 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fa 2015-08-17 16:07:12,694 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fa closed 2015-08-17 16:07:12,694 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down 2015-08-17 16:07:12,795 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1dbc8040 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 2015-08-17 16:07:12,795 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x1dbc80400x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase 2015-08-17 16:07:12,798 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-17 16:07:12,799 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49339, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181 2015-08-17 16:07:12,800 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fc, negotiated timeout = 60000 2015-08-17 16:07:12,803 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fc 2015-08-17 16:07:12,804 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fc closed 2015-08-17 16:07:12,804 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down 2015-08-17 16:07:12,905 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x64f98ef1 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 2015-08-17 16:07:12,905 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x64f98ef10x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase 2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49342, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181 2015-08-17 16:07:12,909 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fe, negotiated timeout = 60000 2015-08-17 16:07:12,912 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fe 2015-08-17 16:07:12,914 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fe closed   ... View more

Re: Speeding up deployment of app binaries

by Mentor Mentor in Support Questions
‎08-10-2015 07:15 PM
‎08-10-2015 07:15 PM
Do the binaries change job-to-job, or are they static? If they do not change, you could benefit from the new feature of proper-shared resource caches: https://issues.apache.org/jira/browse/YARN-1492 Otherwise, compression is your answer. The APIs allow you to add "archives" (zips) which will automatically be unwrapped at the attempt's root, when localising. Have you also profiled which part is worrying to you more? Is it the initial upload-to-HDFS period, or the localising (per NM) period instead? ... View more

Re: Oozie custom action deployment on Cloudera 5.4...

by Contributor Contributor in Support Questions
‎08-04-2015 04:39 PM
‎08-04-2015 04:39 PM
The following steps can be done to get/set configurations: ==== Oozie ActionService Executor Extension Classes ==== >>> from cm_api.api_client import ApiResource >>> print ApiResource('nightly54-1.vpc.cloudera.com').get_all_clusters()[0].get_all_services()[4].get_all_roles()[0].get_config(view='full')['oozie_executor_extension_classes'] : oozie_executor_extension_classes = none >>> print ApiResource('nightly54-1.vpc.cloudera.com').get_all_clusters()[0].get_all_services()[4].get_all_roles()[0].update_config({'oozie_executor_extension_classes':'oozie_test.class'}) >>> print ApiResource('nightly54-1.vpc.cloudera.com').get_all_clusters()[0].get_all_services()[4].get_all_roles()[0].get_config(view='full')['oozie_executor_extension_classes'] : oozie_executor_extension_classes = oozie_test.class ==================== ==== Oozie SchemaService Workflow Extension Schemas ==== >>> from cm_api.api_client import ApiResource >>> print ApiResource('nightly54-1.vpc.cloudera.com').get_all_clusters()[0].get_all_services()[4].get_all_roles()[0].get_config(view='full')['oozie_workflow_extension_schemas'] : oozie_workflow_extension_schemas = ssh-action-0.1.xsd,hive-action-0.3.xsd,sqoop-action-0.3.xsd,shell-action-0.2.xsd,shell-action-0.1.xsd >>> ApiResource('nightly54-1.vpc.cloudera.com').get_all_clusters()[0].get_all_services()[4].get_all_roles()[0].update_config({'oozie_workflow_extension_schemas':'ssh-action-0.1.xsd,hive-action-0.3.xsd,sqoop-action-0.3.xsd,shell-action-0.2.xsd,shell-action-0.1.xsd,oozie-test-action.xsd'}) >>> print ApiResource('nightly54-1.vpc.cloudera.com').get_all_clusters()[0].get_all_services()[4].get_all_roles()[0].get_config(view='full')['oozie_workflow_extension_schemas'] : oozie_workflow_extension_schemas = ssh-action-0.1.xsd,hive-action-0.3.xsd,sqoop-action-0.3.xsd,shell-action-0.2.xsd,shell-action-0.1.xsd,oozie-test-action.xsd =================== Hardcoded value used for method such as "get_all_clusters()[0]" for brevity. A for-loop would be needed to parse for specific value and return the object for the next call, etc... [1]. For future reference, all the modules can be found at ".../cm_api/endpoints." [1] http://cloudera.github.io/cm_api/docs/python-client ... View more

Re: How to read the parameter value in mapper code...

by Mentor Mentor in Support Questions
‎07-28-2015 07:06 AM
‎07-28-2015 07:06 AM
Yes, do you not see it working? You'll need to pass the XML property via the workflow.xml under the action's configuration section. ... View more

Re: Impact when rename hbase big table with snapsh...

by Expert Contributor in Support Questions
‎07-28-2015 01:32 AM
‎07-28-2015 01:32 AM
And I would like to make sure that support namespace when rename the table by snapshot. Thanks ... View more
Contact Me
Community Statistics
Member Since ‎07-31-2013 07:21 AM
Last Visited
Posts 1,924
Kudos received 461