Created on 06-30-2017 06:04 AM - edited 09-16-2022 04:52 AM
Hi,
I am seeing this error when running pig jobs. Which parameter has to be tuned.
The map and redude cluster wide memory is 4G for map and 4G for reduce.
We are not settnig any heap while running job.
2017-06-29 20:05:10,664 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2017-06-29 20:05:10,718 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2017-06-29 20:05:12,220 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN,HASH_JOIN,DISTINCT,FILTER
2017-06-29 20:42:02,042 [Service Thread] INFO org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call- Usage threshold init = 698875904(682496K) used = 597759256(583749K) committed = 698875904(682496K) max = 698875904(682496K)
2017-06-29 20:42:02,934 [Service Thread] INFO org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call - Collection threshold init = 698875904(682496K) used = 457643760(446917K) committed = 698875904(682496K) max = 698875904(682496K)
2017-06-29 20:52:26,919 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. Java heap space Details at logfile: xxxxxx
Created 08-24-2017 12:38 PM
Pig supports a number of Java properties that you can use to customize Pig behavior. You can retrieve a list of the properties using the help properties command. All of these properties are optional; none are required.
To specify Pig properties use one of these mechanisms:
Note: The properties file uses standard Java property file format.
The following precedence order is supported: pig.properties > -D Pig property > -P properties file > set command. This means that if the same property is provided using the –D command line option as well as the –P command line option and a properties file, the value of the property in the properties file will take precedence.
To specify Hadoop properties you can use the same mechanisms:
The same precedence holds: hadoop-site.xml > -D Hadoop property > -P properties_file > set command.
Hadoop properties are not interpreted by Pig but are passed directly to Hadoop. Any Hadoop property can be passed this way.
All properties that Pig collects, including Hadoop properties, are available to any UDF via the UDFContext object. To get access to the properties, you can call the getJobConf method.