About Harsh J

Harsh J · ‎08-25-2015

You are unfortunately impacted by a limitation described at: https://issues.apache.org/jira/browse/HDFS-5796 You will need to use a custom filter class to avoid this (i.e. to auth browsers separately from regular WebHDFS auth access, probably by checking UserAgent strings/etc.), by implementing a sub-class of https://github.com/cloudera/hadoop-common/blob/cdh5.4.4-release/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java Note that you can instead also opt to use the Hue File Browser, which is also more feature-rich in comparison.

Harsh J · ‎08-25-2015

You can attempt to increase your jute.maxbuffer Java System Property on the ZK servers to a value higher than 2-3 GB (in bytes) to overcome this. It appears a very large record was somehow placed into your ZK by an application, which appears to have then caused this issue.

Harsh J · ‎08-24-2015

Thanks for taking the time to update the case. If it did resolve your issue, please do mention the steps taken and mark the thread as resolved, so it helps others with similar errors when they land here!

Harsh J · ‎08-23-2015

What version of HBase libraries is your application referencing? It seems like an old one, not matching the cluster ones.

Harsh J · ‎08-15-2015

How exactly are you invoking your Java application? Is /etc/hbase/conf/ directory on the classpath? If the failure you're quoting is from within the Reducer task log, then ensure you don't do a "new Configuration()"/"HBaseConfiguration.create()" anywhere inside the Reducer class, but instead reuse context.getConfiguration() to obtain the standard job configuration which will carry the ZK properties.

Harsh J · ‎08-11-2015

Check http://stackoverflow.com/a/14540272 perhaps, which includes an example.

Harsh J · ‎08-11-2015

It just means that when the split offset (starting point) is 0, i.e. start of file, we read the first line. Otherwise (non-zero offsets/starting point/mid-points of file) we arbitrarily skip the first line, because we know that the previous split's reader reads always an extra line at the end. Does this help?

Harsh J · ‎08-11-2015

What part of it was not clear specifically? Could you quote, so it can be explained further?

Harsh J · ‎08-11-2015

Please read http://wiki.apache.org/hadoop/HadoopMapReduce

Harsh J · ‎08-10-2015

Do the binaries change job-to-job, or are they static? If they do not change, you could benefit from the new feature of proper-shared resource caches: https://issues.apache.org/jira/browse/YARN-1492 Otherwise, compression is your answer. The APIs allow you to add "archives" (zips) which will automatically be unwrapped at the attempt's root, when localising. Have you also profiled which part is worrying to you more? Is it the initial upload-to-HDFS period, or the localising (per NM) period instead?

Member Since	‎07-31-2013 07:21 AM
Last Visited
Posts	1,924
Kudos received	461

Cloudera Community

Re: S3Guard Suggested to help fix Consistency

Re: Failed to start namenode. java.io.FileNotFound...

Re: sqoop import issue

Re: Efficient ways to store many images files

Re: S3 loading into HDFS

Re: Permission denied when trying to open /webhdfs...

Re: zookeeper error Unexpected exception causing s...

Re: Hbase coonection via Java Application - Cloude...

Re: Hbase coonection via Java Application - Cloude...

Re: HBASE ZOOKEEPER CONFIG CDH5.4.4

Re: How Map and Reduce operations are actually car...

Re: How Map and Reduce operations are actually car...

Re: How Map and Reduce operations are actually car...

Re: How Map and Reduce operations are actually car...

Re: Speeding up deployment of app binaries