Basic but have you tried restarting the history-server? ./sbin/start-history-server.sh restart ... View more

@Sachin Ambardekar, The doc above may be slightly dated. Rule of thumb, 4GB per core seems to be the sweet spot for memory intensive workloads which are getting more common nowadays. ... View more

Correct. However these are not necessarily physical disk ... View more

I had a similar use case recently. You have to approach this understanding that it's different paradigm: You can't do I/Os the old fashion way; whatever dataset you're manipulating must be distributed; ie your log file should be on HDFS. So first step, opening the log file and creating a RDD would look something like this: spark = SparkSession\ .builder\ .appName("CheckData")\ .getOrCreate() lines = spark.read.text("hdfs://[servername]/[path]/Virtual_Ports.log").rdd.map(lambda r: r[0]) You don't programmatically iterate on the data per say, instead you supply a function to process each value (lines in this case). So your code where you iterate on lines could be put inside a function: def virtualPortFunction(line): #Do something, return output process of a line virtualPortsSomething = lines.flatMap(lambda x: x.split(' ')) \ .map(lambda x: virtualPortFunction(x)) This is very simplistic way to put it but this will give you a starting point if you decide to go down the PySpark route. Also look at the pyspark samples part of the Spark distribution. Good place to start. https://github.com/apache/spark/blob/master/examples/src/main/python/wordcount.py ... View more

Vitaly, Looks like these are the maven artifacts for packaging the HDP distribution. I suggest using those from Maven Central: http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.spark%22 ... View more

Vishal, you should be able to access VORA's with HANA's JDBC driver. You just need to then map it to a PySpark or SParkR context. Look it up, it's easy to find online... or let me know if you're stuck. ... View more

Houssam, dynamic allocation should help. How many executors are you currently using? ... View more

This is pretty cool Simon, thank you. I've been playing with the idea of building a "podcast indexer" using Nifi, Voice-to-text, Solr. What if I could search all podcasts of the last week mentioning "election" and get audio link and seek position.... 🙂 ... View more

Great article cduby. Thanks! ... View more

I have not recently but you raised my curiosity :). Doing some research, their sandbox appears to built on top of HDP 2.2, so the functionaries HDP sandbox provides should also be available (Ambari, Hive etc..). I'm going to download it and take a look, I'll keep you posted. If you're looking for an entity discovery/analytics/lineage product in general, I would suggest looking into Novetta, another partner of ours. https://hortonworks.com/wp-content/uploads/2014/05/Novetta-Entity-Analytics-and-Hortonworks-Solution-Overview.pdf They have a sandbox available on AWS marketplace also, built on HDP. It comes with a comprehensive tutorial as well. ... View more

Manoj, here's a helpful webcast on HDP and WaterLine can work together for implementing a data lake with data governance. http://bit.ly/1XbjOzq ... View more

Very cool. Thanks jfrazee. ... View more

Great post George. Thank you! ... View more