It looks like you didn't install some package that your notebook requires. ... View more

You can make a DataFrame over all the files and then filter out the lines you don't want. You can make a DataFrame for just the files you want, then union them together. Both are viable.   If you're saying different data types are mixed into sections of each file, that's harder, as you need to use something like mapPartitions to carefully process each file 3 times.     ... View more

Not sure what you're trying to do there, but looks like you have a simple syntax error. bucketBy is a method. Please start with the API docs first. ... View more

I think you mean something like df.write.mode(SaveMode.Overwrite).saveAsTable(...) ? Depends on what language this is. ... View more

I'm not sure what you're asking here. I have verified the project build works and all tests pass. Follow the tutorial at http://oryx.io/docs/endusers.html to get a working instance and take it from there. ... View more

You have to have an initial model before anything works. After that, of course, model scoring happens in real time and updates happen in near-real-time.   I'm not sure what you mean in your second point.   The word count example is correct. It's counting unique co-occurrences of words. If there is just one word on a line, there are no co-occurrences to count. ... View more

No need to ping. As far as I know nobody certifies pandas-Spark integration. We support Pyspark. It has a minimal integration with pandas (e.g. the toPandas method). If there were a Pyspark-side issue we'd try to fix it. But we don't support pandas. ... View more

No, because the speed layer also can't produce model updates unless it first sees a model.   What do you mean that you can't see the data -- in what place? ... View more

It could be a lot of things, but keep in mind that you will not see any model updates unless you have a batch layer running, and it has had time to compute a model and send it to the serving layer. If the batch layer is running, check if it is able to produce a model. It would not with only 2 data points. ... View more

I'll answer in the other duplicated post. ... View more

If you're asking about EMR, this is the wrong place -- that's an Amazon product. ... View more

If you observe there are no jobs starting for an extended period of time, then I'd figure the driver is stuck listing S3 files. If you find the stage that reads the data from S3 is taking a long time, then I'd figure it's reading from S3. It's not certain that's the issue but certainly the place I'd look first.   Because the S3 support is basically from Hadoop, you might look at https://wiki.apache.org/hadoop/AmazonS3 Are you using an s3a:// URI? ... View more

You're almost certainly bottlenecked on reading for S3, or listing the S3 directories, or both. You'd have to examine the stats in the Spark UI to really know. Given your simple job, it's almost certainly nothing to do with memory or CPU, or the shuffle. Try parallelizing more with more, smaller input files? splitting across more workers to make more bandwidth available?   I'm also not clear if you're using a Cloudera cluster here. ... View more

Have a look at https://github.com/sryza/spark-timeseries for time series on Spark.  ... View more

CDSW just uses the Jupyter kernel, so much of what you do in the Jupyter notebook would work here too. Magics will work. There's also an option to suppress this output in the output that is shared in the "Share" link, but that's not quite what you mean. ... View more

I'm not sure how you would do that. We support spark-submit and the Workbench, not Jupyter. It's clear how to configure spark-submit, and you configure the workbench with spark-defaults.conf. You can see your Spark job's config in its UI, in the environment tab. ... View more

This has nothing to do with CM. It has to do with your app's memory configuration. The relevant settings are right there in the error. ... View more

If you have 4 topics with 3 partitions each then you need 12 executor slots to process fully in parallel. You have only 3 slots. If you are using receiver based streaming you may need 1 more, too. Also, 1 core per executor is generally very low. Your result is therefore not surprising and your second config much more reasonable. ... View more

You don't, in general. The cluster is where you run rather than create software. It is more like the JRE env than JDK. Lots of compiler tools aren't available and generally on purpose. ... View more

Caching is OK in that Spark won't use more than it's allowed for caching, and you can turn that fraction down, if your app is heavily using memory for other things. Have a look at  spark.memory.fraction or spark.memory.storageFraction. However that is only the issue if you're running out of memory on executors.   If you're running out of driver memory try retaining a lot fewer job history details. Turn spark.ui.retained{Jobs,Stages,Tasks} way down to reduce that memory consumption.   But the answer may simply be that you need more memory. I don't see evidence that 7G is necessarily enough, depending on what you are doing. ... View more

You're simply running out of memory. A large portion of memory is dedicated to caching data in Spark, of course, and that explains why a lot of memory has cached data. That's not necessarily the issue here. You may be retaining state on lots of jobs in the driver and that's eating memory in the driver (wasn't clear whether that's the heap you're showing). You can just incease memory, or look for ways to reduce memory usage. ... View more