Member since
10-01-2015
3933
Posts
1150
Kudos Received
374
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2639 | 05-03-2017 05:13 PM | |
2213 | 05-02-2017 08:38 AM | |
2363 | 05-02-2017 08:13 AM | |
2398 | 04-10-2017 10:51 PM | |
1067 | 03-28-2017 02:27 AM |
10-24-2015
01:20 AM
the requirement is to call sqoop command from a shell action, there's looping going on and needs more flexibility than using a sqoop action in Oozie. I am aware of all the possible ways of sqooping in Oozie, for some reason calling a shell action and sqoop within it, throws the error above and my question is what is the fix for that error. Why does it complain about
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException:File does not exist: hdfs://servername:8020/user/username/.staging/job_1444331888071_2109/job.splitmetainf
... View more
10-24-2015
01:02 AM
here's a good example https://github.com/apache/hbase/blob/master/hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/BufferedMutatorExample.java
... View more
10-24-2015
01:00 AM
sqoop is being called from a shell action, no choice in that.
... View more
10-23-2015
08:13 PM
can't find any solution to this error, sqoop in the shell script runs fine on the command line but not in Oozie org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://servername:8020/user/username/.staging/job_1444331888071_2109/job.splitmetainfo
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1568)
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1432)
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1390)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1312)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1080)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://servername:8020/user/username/.staging/job_1444331888071_2109/job.splitmetainfo
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1563)
... 17 more
2015-10-23 15:45:55,263 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Sqoop
10-23-2015
05:33 PM
One thing to keep in mind is that SmartSense will collect core-site.xml so if you have WASB access keys, they will be collected, so to opt-out you need to disable the properties you don't want collected.
... View more
10-23-2015
02:19 PM
1 Kudo
SOLUTION Pig introduced an option to run hcatalog commands in grunt and in scripts. There's more info in pig.properties file in /etc/pig/conf/pig.properties # In addition to the fs-style commands (rm, ls, etc) Pig can now execute # SQL-style DDL commands, eg "sql create table pig_test(name string, age int)". # The only implemented backend is hcat, and luckily that's also the default. # # pig.sql.type=hcat
# Path to the hcat executable, for use with pig.sql.type=hcat (default: null) # hcat.bin=/usr/local/hcat/bin/hcat this is on sandbox 2.3, HDP 2.3 and HDP 2.3.2 by default so running any pig script with hcat commands without -useHCatalog will fail with the following, usually that will happen through Oozie Pig Stack Trace
---------------
ERROR 2997: Encountered IOException. /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties.
java.io.IOException: /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties.
at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1286)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:631)
at org.apache.pig.Main.main(Main.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136) so to fix it change it globally by editing pig.properties in Ambari to point to the following: hcat.bin=/usr/bin/hcat or copy pig.properties to your own location, override the pig.properties with right path (i.e. hcat.bin=/usr/bin/hcat) and execute script like so pig -P pig.properties test.pig or override the property on the fly pig -Dhcat.bin=/usr/bin/hcat test.pig Or even lesser intrusive way: In your pig script put this in the beginning set hcat.bin /usr/bin/hcat;
... View more
10-23-2015
03:16 AM
1 Kudo
Does anyone know if this utility was renamed or deprecated? Is there an equivalent? HBase Client: Group Puts by RegionServer In addition to using the writeBuffer, grouping Puts by RegionServer can reduce the number of client RPC calls per writeBuffer flush. There is a utility HTableUtil currently on TRUNK that does this, but you can either copy that or implement your own verison for those still on 0.90.x or earlier.
... View more
Labels:
- Labels:
-
Apache HBase
10-23-2015
01:55 AM
do I need to set $HCAT_HOME/bin/hcat in pig.properties?
... View more
10-23-2015
01:52 AM
I see it's thrown from this class https://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/tools/grunt/GruntParser.java
... View more
10-23-2015
01:49 AM
getting error when executing a pig script through Oozie. Script has pig and hcat commands.
... View more
Labels: