Member since
10-02-2017
8
Posts
0
Kudos Received
0
Solutions
10-25-2017
10:16 AM
Sean, Not clear what you mean "... you should be able to use HDFS paths.... ". Yeah, I know that for sure HDFS path. However, I am refering to S3 path, somthing like "s3://bucket/....". Is S3 path possible ?
... View more
10-24-2017
04:06 PM
Sean, Two more questions about Oryx 1: (1) It has "model.local-data=false" I set it to false to use HDFS I am thinking "model.instance-dir=..." will point to HDFS. My question is whether it supports S3? Something like " model.instance-dir=s3://bucket/..." (2) Is it running with Java 7 as well (in local computation mode) ? Thanks.
... View more
10-06-2017
10:53 AM
I maven built from sources of Oryx 1 and ran well. Good to know the link, though. Thanks
... View more
10-04-2017
08:19 PM
Sean, Got it and downloaded it. It would be nice if you can revive the repo. I used Oryx 1.0 before and it worked very well. If you cannot make the repo in short term, can you pass along the latest release jar files ? Thanks.
... View more
10-04-2017
08:03 PM
Sean, Thanks for the quick reply. That links show GitHub 404. Perhaps it's not public available any more. Can you fix it ? Thanks.
... View more
10-04-2017
07:49 PM
Sean, I recall there is a local computation mode in Oryx 1.x This is useful for us to play ALS with smaller data set and see how it performs without going into all the Spark, Kafka setup. However; I cannot locate the releases (or git source). Where can I find it ? Thanks.
... View more
Labels:
- Labels:
-
Oryx
10-04-2017
07:30 PM
Sean, I got the same error. I am using kafka 0.11.0.1 and Oryx 2.5. I am using AWS EMR (hadoop 2.7.3; spark 2.2.0 and zk 3.4.10) It looks cannot locate kafka related jar file. Is any way to pass in the jar file ? Thanks. ======= 17/10/05 02:25:17 INFO YarnClientSchedulerBackend: Shutting down all executors 17/10/05 02:25:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 17/10/05 02:25:17 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 17/10/05 02:25:17 INFO YarnClientSchedulerBackend: Stopped 17/10/05 02:25:17 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/10/05 02:25:17 INFO MemoryStore: MemoryStore cleared 17/10/05 02:25:17 INFO BlockManager: BlockManager stopped 17/10/05 02:25:17 INFO BlockManagerMaster: BlockManagerMaster stopped 17/10/05 02:25:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/10/05 02:25:17 INFO SparkContext: Successfully stopped SparkContext Exception in thread "main" java.lang.NoClassDefFoundError: kafka/admin/RackAwareMode at com.cloudera.oryx.lambda.AbstractSparkLayer.buildInputDStream(AbstractSparkLayer.java:179) at com.cloudera.oryx.lambda.batch.BatchLayer.start(BatchLayer.java:105) at com.cloudera.oryx.batch.Main.main(Main.java:33) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: kafka.admin.RackAwareMode at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 12 more 17/10/05 02:25:17 INFO ShutdownHookManager: Shutdown hook called 17/10/05 02:25:17 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-73fdbc88-0da1-498e-aece-7abc3856aad1
... View more
10-02-2017
11:27 AM
Sean, Is CDH required for Oryx 2 ? According to matrix summary, it seems required (along with Kafka parcel). But, the top of the page not indicates CDH (but it lists Hadoop 2.7.0 and Kafka 0.10.2...). http://oryx.io/docs/admin.html Can I just use Hadoop and Kafka (not Cloudera Kafka parcel) ? Thanks
... View more
Labels:
- Labels:
-
Oryx