Basic but have you tried restarting the history-server? ./sbin/start-history-server.sh restart ... View more

@Sachin Ambardekar, The doc above may be slightly dated. Rule of thumb, 4GB per core seems to be the sweet spot for memory intensive workloads which are getting more common nowadays. ... View more

Thanks man, It wasn't clear if CB 1.6 could supports ADLS. A better option than WASB shards with DASH for sure.. ... View more

Quick update, DASH is a package available from MSFT that allows "sharding" accross multiple accounts: https://github.com/MicrosoftDX/Dash/tree/master/DashServer ... View more

I had a similar use case recently. You have to approach this understanding that it's different paradigm: You can't do I/Os the old fashion way; whatever dataset you're manipulating must be distributed; ie your log file should be on HDFS. So first step, opening the log file and creating a RDD would look something like this: spark = SparkSession\ .builder\ .appName("CheckData")\ .getOrCreate() lines = spark.read.text("hdfs://[servername]/[path]/Virtual_Ports.log").rdd.map(lambda r: r[0]) You don't programmatically iterate on the data per say, instead you supply a function to process each value (lines in this case). So your code where you iterate on lines could be put inside a function: def virtualPortFunction(line): #Do something, return output process of a line virtualPortsSomething = lines.flatMap(lambda x: x.split(' ')) \ .map(lambda x: virtualPortFunction(x)) This is very simplistic way to put it but this will give you a starting point if you decide to go down the PySpark route. Also look at the pyspark samples part of the Spark distribution. Good place to start. https://github.com/apache/spark/blob/master/examples/src/main/python/wordcount.py ... View more

Assuming you're using HDP 2.5 sandbox, another option would be to deploy Zeppelin service in Amabari. The modules above are also included in Zeppelin. ... View more

This is pretty cool Simon, thank you. I've been playing with the idea of building a "podcast indexer" using Nifi, Voice-to-text, Solr. What if I could search all podcasts of the last week mentioning "election" and get audio link and seek position.... 🙂 ... View more

Good point Tim. Each "SQL on Hadoop" implementation obviously has pros and cons... general rule of thumbs: SparkSQL --> good for iterative processing, access existing Hive tables, given results fits in memory HAWQ --> good for "traditional" BI-like queries, star schemas, cubes OLAP HIVE-LLAP --> good for petabyte scale mixed with smaller tables requiring sub-second queries. Phoenix --> A good way to interact with HBase tables, good with time series, good indexing Drill, Presto --> Query federation-like capabilities but limited SQL syntax. Performance varies quite a bit. ... View more

Good point Tim. Each "SQL on Hadoop" implementation obviously has pros and cons... general rule of thumbs: SparkSQL --> good for iterative processing, access existing Hive tables, given results fits in memory HAWQ --> good for "traditional" BI-like queries, star schemas, cubes OLAP HIVE-LLAP --> good for petabyte scale mixed with smaller tables requiring sub-second queries. Phoenix --> A good way to interact with HBase tables, good with time series, good indexing Drill, Presto --> Query federation-like capabilities but limited SQL syntax. Performance varies quite a bit. ... View more