It looks like you didn't install some package that your notebook requires. ... View more

You can make a DataFrame over all the files and then filter out the lines you don't want. You can make a DataFrame for just the files you want, then union them together. Both are viable.   If you're saying different data types are mixed into sections of each file, that's harder, as you need to use something like mapPartitions to carefully process each file 3 times.     ... View more

Not sure what you're trying to do there, but looks like you have a simple syntax error. bucketBy is a method. Please start with the API docs first. ... View more

I think you mean something like df.write.mode(SaveMode.Overwrite).saveAsTable(...) ? Depends on what language this is. ... View more

I'm not sure what you're asking here. I have verified the project build works and all tests pass. Follow the tutorial at http://oryx.io/docs/endusers.html to get a working instance and take it from there. ... View more

You have to have an initial model before anything works. After that, of course, model scoring happens in real time and updates happen in near-real-time.   I'm not sure what you mean in your second point.   The word count example is correct. It's counting unique co-occurrences of words. If there is just one word on a line, there are no co-occurrences to count. ... View more

No need to ping. As far as I know nobody certifies pandas-Spark integration. We support Pyspark. It has a minimal integration with pandas (e.g. the toPandas method). If there were a Pyspark-side issue we'd try to fix it. But we don't support pandas. ... View more

No, because the speed layer also can't produce model updates unless it first sees a model.   What do you mean that you can't see the data -- in what place? ... View more

It could be a lot of things, but keep in mind that you will not see any model updates unless you have a batch layer running, and it has had time to compute a model and send it to the serving layer. If the batch layer is running, check if it is able to produce a model. It would not with only 2 data points. ... View more

I'll answer in the other duplicated post. ... View more

If you're asking about EMR, this is the wrong place -- that's an Amazon product. ... View more

If you observe there are no jobs starting for an extended period of time, then I'd figure the driver is stuck listing S3 files. If you find the stage that reads the data from S3 is taking a long time, then I'd figure it's reading from S3. It's not certain that's the issue but certainly the place I'd look first.   Because the S3 support is basically from Hadoop, you might look at https://wiki.apache.org/hadoop/AmazonS3 Are you using an s3a:// URI? ... View more

You're almost certainly bottlenecked on reading for S3, or listing the S3 directories, or both. You'd have to examine the stats in the Spark UI to really know. Given your simple job, it's almost certainly nothing to do with memory or CPU, or the shuffle. Try parallelizing more with more, smaller input files? splitting across more workers to make more bandwidth available?   I'm also not clear if you're using a Cloudera cluster here. ... View more

Have a look at https://github.com/sryza/spark-timeseries for time series on Spark.  ... View more

CDSW just uses the Jupyter kernel, so much of what you do in the Jupyter notebook would work here too. Magics will work. There's also an option to suppress this output in the output that is shared in the "Share" link, but that's not quite what you mean. ... View more

I'm not sure how you would do that. We support spark-submit and the Workbench, not Jupyter. It's clear how to configure spark-submit, and you configure the workbench with spark-defaults.conf. You can see your Spark job's config in its UI, in the environment tab. ... View more

This has nothing to do with CM. It has to do with your app's memory configuration. The relevant settings are right there in the error. ... View more

If you have 4 topics with 3 partitions each then you need 12 executor slots to process fully in parallel. You have only 3 slots. If you are using receiver based streaming you may need 1 more, too. Also, 1 core per executor is generally very low. Your result is therefore not surprising and your second config much more reasonable. ... View more

That class was added in Kafka 0.10, so normally I'd say you have a version mismatch, and you're using a version of Oryx built for Kafka 0.10+ with older Kafka. However you say you're using Kafka 0.11. But I think the issue is that something else is bringing older Kafka client libs onto your classpath.   For example, I think the Spark examples JAR file brings it in? that could be a source of old Kafka 0.8 libs. Or somehow the classpath EMR sets up is bringing in Spark's Kafka 0.8 and 0.10 integration or something?   This part is definitely tricky. I know it works on CDH, mostly because its Spark 2 distro only supports Kafka 0.10. ... View more