Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

Highlighted

I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

Error:

java.io.IOException: Trying to load more than 32 hfiles to one family of one region at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:419) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:313) at com.HBase.BulkLoad.HBaseBulkLoad.doBulkLoad(HBaseBulkLoad.java:35) at com.HBase.BulkLoad.HbaseBulkLoadDriver.run(HbaseBulkLoadDriver.java:82) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at com.HBase.BulkLoad.HbaseBulkLoadDriver.main(HbaseBulkLoadDriver.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

15 REPLIES 15
Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

Hi @venkateswarlu prudhvi

You can increase value of this configuration at client and retry loading then it should work fine.

hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily

Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

Where do we found that property and let me know to what extent I can increase,can you give me the code snippet so that I change it.

Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

I did not find first property in /etc/hbase/conf/hbase-site.xml.

Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

You can add this property with the max number of files you trying to load per column family. It might vary from the data size you are trying to bulkload.

Here is the code snippet from LoadIncrementalHFiles.

{code}

..

if (!checkHFilesCountPerRegionPerFamily(regionGroups)) {

// Error is logged inside checkHFilesCountPerRegionPerFamily.

throw new IOException("Trying to load more than " + maxFilesPerRegionPerFamily

+ " hfiles to one family of one region");

..

}

{code}

{code}

...

maxFilesPerRegionPerFamily = conf.getInt(MAX_FILES_PER_REGION_PER_FAMILY, 32);

..

{code}

Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

Can you also check whats the value of hbase.hregion.max.filesize?

Is it very less that's why that many number of files getting created?

Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

Follwing is my code

Driver code

==========

System.out.println("Before IncrementalLoad:::::"); HFileOutputFormat.configureIncrementalLoad(job, new HTable(configuration,TABLE_NAME)); System.out.println("After IncrementalLoad:::::"); job.waitForCompletion(true); System.out.println("after waitForCompletion:::"); System.out.println("Output Path:::"+outputPath); if (job.isSuccessful()) { System.out.println("Inside if:::"); HBaseBulkLoad.doBulkLoad(outputPath, TABLE_NAME); } else { System.out.println("Inside else::::"); result = -1; }

doBulkLoad code

=============

public static void doBulkLoad(String pathToHFile, String tableName) { try { System.out.println("Inside doBulkLoad::::"); // String hbaseConfDir="/etc/hbase/conf"; Configuration configuration = new Configuration(); // configuration.addResource(new Path("/usr/hdp/2.4.0.0-169/hbase/conf/hbase-site.xml")); configuration.addResource(new Path("/usr/hdp/2.4.2.0-258/hbase/conf/hbase-site.xml")); // configuration.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml")); // configuration.set("mapreduce.child.java.opts", "-Xmx1g"); //new code started System.out.println("Before::::"); // configuration.set("zookeeper.znode.parent", "/hbase-unsecure"); /* configuration.set("zookeeper.znode.parent", "/hbase"); configuration.set("hbase.zookeeper.quorum", "sandbox.hortonworks.com"); configuration.set("hbase.zookeeper.property.clientPort", "2181");*/ //new code ended // HBaseConfiguration.addHbaseResources(configuration); LoadIncrementalHFiles loadFfiles = new LoadIncrementalHFiles(configuration); HTable hTable = new HTable(configuration, tableName); loadFfiles.doBulkLoad(new Path(pathToHFile), hTable); System.out.println("Bulk Load Completed.."); } catch(Exception exception) { exception.printStackTrace(); } } }

Let me know where I need to change and what I need to add

Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

You can increase hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily to value more than no. of files you are trying to load.

And also increase hbase.hstore.blockingStoreFiles to avoid regionTooBusyException.

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

Where I need to find those properties

Highlighted

Re: I am getting java.io.IOException: Trying to load more than 32 hfiles to one family of one region Error when I am doing Bulk Load

I did not find first property in /etc/hbase/conf/hbase-site.xml.

Don't have an account?
Coming from Hortonworks? Activate your account here