Member since
10-28-2016
9
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3301 | 11-15-2016 11:40 AM | |
7226 | 11-14-2016 01:20 PM |
11-28-2016
10:04 AM
Hello, I'm getting a timeout error with PowerBI, when I use curl from my Linux desktop all works fine. I'm just using the 'free' PowerBI download, anyone know if this version is fully functional? Any chance anyone's been able to read Parquet formatted data into PowerBI (our bigger question). Thanks, Craig
... View more
11-16-2016
11:52 AM
Simple sparkSQL application works: (In CDH 5.8.0) fine with spark-submit run through oozie, generate below exception: What r we doing wrong? Oozie exception: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient Spark Code:---------------------------- import org.apache.spark._ import org.apache.spark.sql._ import org.apache.spark.sql.hive._ object SQLAction{ def main(args:Array[String]) { val conf = new SparkConf() val sc = new SparkContext(conf) val sqlc = new HiveContext(sc) // cause exception println("sparkSQL success") //not get run sqlc sql "select * from testcase_pos" collect println("select success") sc.stop } } ---------------------------- What we tried: Add hive-site.xml as Job.xml. Hive-site.xml is stored under workspace. Same Exception Add conf.set("hive.metastore.uris","thrift://host:9083") into spark code. Not working We netstat the metastore thrift port but nothing is there …Could some one show me what's underline of oozie sparksql action?
... View more
Labels:
11-15-2016
11:40 AM
Hello, kind of figured out a few things here ... there was a mental block on local vs hdfs paths that the error messages didn't really give clues. the property file defines the oozie.wf.application.path which is the base directory for the supplied jars, the jar path was preceeded by ${nodeName} which is an absolute path. you have to logically put together the Configuration (property) file with the Definition (workflow.xml) to make sure every thing is being addressed properly. Also digging in the spark code yielded the System.exit(101) error ... classNotFound, which was an indicator that an invalid path was being used in our configuration. hope this helps, Craig
... View more
11-15-2016
06:55 AM
Executing a real simple spark job, just print one line. The jar file is generated by sbt. It works fine with spark-submit command line. When executing from oozie (BY hue's UI, only specified the jar file, no arguments, no 3-party jars): Always return System.exit(101). No matter the spark.core dependency is included or not. Does some knows the meaning of error 101? Can't find it in the source code in SparkMain or SparkSubmit. Stdout: .... --conf spark.yarn.security.tokens.hive.enabled=false --conf spark.yarn.security.tokens.hbase.enabled=false --verbose /user/mrt1/test/oozie-test.jar ================================================================= >>> Invoking Spark class now >>> Intercepting System.exit(101) <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [101] Oozie Launcher failed, finishing Hadoop job gracefully Oozie Launcher, uploading action data to HDFS sequence file: hdfs://.....:8020/user/mrt1/oozie-oozi/0000142-160913091926881-oozie-oozi-W/spark-57f6--spark/action-data.seq Oozie Launcher ends
... View more
11-14-2016
01:20 PM
1 Kudo
Hello, this finally got resolved, I was missing the <file> tag in the workflow.xml <exec>/user/mrt1/oozie-oozi/craigTest.sh</exec> <file>/user/mrt1/oozie-oozi/craigTest.sh#craigTest.sh</file> A colleague suggested adding the file tag and trying it, low and behold it worked. I then went back to Hue and saw the "files+" section added the full path there and it worked. The error message kind of makes sense, but I would think Hue wouldn't let me create the shell action without the <file> tag. I guess it means never create a workflow.xml with only the <exec> tag. Craig
... View more
11-10-2016
08:39 AM
hello, I'm a oozie first time user trying to automate some of our spark jobs. They were failing so I figured I'd start with a simple script to establish we have things configured correctly. Bash cmd line: $ echo -e "A|1|2|3\nB|4|5|6" | hadoop fs -put - /ARisk/tmp/test.txt works as expected. I created an HDFS file: /user/hue/oozie/workspaces/workflows/craigTest.sh with only the above cmd line in there. $ hadoop fs -chmod 777 /user/hue/oozie/workspaces/workflows/craigTest.sh created an oozie workflow, shell script, click and select path to craigTest.sh above. When I save and run the job I get a failure, here is the log stdout: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory java.io.IOException: Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:102) at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:61) so I try to navigate to see where the job would run .... [root@wrlmr1 usercache]# pwd; ls -al /data6/yarn/nm/usercache total 8 drwxr-xr-x 2 yarn yarn 4096 Jul 25 16:03 . drwxr-xr-x 5 yarn hadoop 4096 Jul 25 16:43 .. nothing in the dir here .... do I have some kind of permission issue? any pointers appreciated, this is CDH 5.8 out of the box. Thanks, Craig
... View more
- Tags:
- CDH5.8
10-28-2016
12:16 PM
Hello,
I've got HUE working like a champ, this is CDH 5.8 in our Lab.
but I'm getting lots of failures or hangs with curl using various commands
hangs:
http://localhost:50070/webhdfs/v1/tmp/test1?op=MKDIRS
I'm trying to figure out what Port HUE is using, my hue.ini doesn't show it.
This is commented out in hue.ini
# Use WebHdfs/HttpFs as the communication mechanism. # Domain should be the NameNode or HttpFs host. # Default port is 14000 for HttpFs. ## webhdfs_url=http://localhost:50070/webhdfs/v1
is there some other param or config I should be referencing?
Regards,
Craig
... View more
10-28-2016
10:19 AM
Vale, no worries, thanks for the response. All seems well with my services page, and the logs don't appear to be funky or anything. I do get some connect failure messages from my curl request: curl -i "http://127.0.0.1:1400/webhdfs/v1/user?op=GETCONTENTSUMMARY" .... <!--Title--> <table class='titleTable' > <tr> <td class='titleData'> Cannot Connect </td> </tr> </table> <!--/Title--> <!--Content--> <table class="contentTable"> <tr> <td class="contentData"> The proxy could not connect to the destination in time. </td> </tr> </table> <!--/Content-->
... View more
10-28-2016
08:45 AM
Hi, Getting same erorr as user above, my service is enabled so there seems to be something dumb i'm doing. Any pointers? Thanks, Craig The requested URL could not be retrieved While trying to retrieve the URL: ****REMOVED**** The following error was encountered: We can not connect to the server you have requested. This means that: •The server might be busy at this time. •The server is not reachable. Please try later to see if you can go through
... View more