Basic but have you tried restarting the history-server? ./sbin/start-history-server.sh restart ... View more

@Sachin Ambardekar, The doc above may be slightly dated. Rule of thumb, 4GB per core seems to be the sweet spot for memory intensive workloads which are getting more common nowadays. ... View more

Correct. However these are not necessarily physical disk ... View more

I had a similar use case recently. You have to approach this understanding that it's different paradigm: You can't do I/Os the old fashion way; whatever dataset you're manipulating must be distributed; ie your log file should be on HDFS. So first step, opening the log file and creating a RDD would look something like this: spark = SparkSession\ .builder\ .appName("CheckData")\ .getOrCreate() lines = spark.read.text("hdfs://[servername]/[path]/Virtual_Ports.log").rdd.map(lambda r: r[0]) You don't programmatically iterate on the data per say, instead you supply a function to process each value (lines in this case). So your code where you iterate on lines could be put inside a function: def virtualPortFunction(line): #Do something, return output process of a line virtualPortsSomething = lines.flatMap(lambda x: x.split(' ')) \ .map(lambda x: virtualPortFunction(x)) This is very simplistic way to put it but this will give you a starting point if you decide to go down the PySpark route. Also look at the pyspark samples part of the Spark distribution. Good place to start. https://github.com/apache/spark/blob/master/examples/src/main/python/wordcount.py ... View more

Vitaly, Looks like these are the maven artifacts for packaging the HDP distribution. I suggest using those from Maven Central: http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.spark%22 ... View more

Vishal, you should be able to access VORA's with HANA's JDBC driver. You just need to then map it to a PySpark or SParkR context. Look it up, it's easy to find online... or let me know if you're stuck. ... View more

This is pretty cool Simon, thank you. I've been playing with the idea of building a "podcast indexer" using Nifi, Voice-to-text, Solr. What if I could search all podcasts of the last week mentioning "election" and get audio link and seek position.... 🙂 ... View more

Great article cduby. Thanks! ... View more

Very cool. Thanks jfrazee. ... View more

Great post George. Thank you! ... View more