By Ambari platform you mean? You can use either zeppelin or superset which you were using. Zeppelin has a lot of interpreters and it can connect to hive/spark/mysql. https://zeppelin.apache.org/supported_interpreters.html Visualization in superset is easier, you can create hive tables using that csv or create mysql using that. You can then add that database in superset and add existing tables. https://superset.incubator.apache.org/tutorial.html#connecting-to-a-new-database ... View more

@Ftoon Kedwan I think you've got the concept wrong In superset, you add datasources/databases and tables assuming they are already present in your environment. (it doesn't create those for you). For example, you'll have a mysql db somewhere and you'll have to provide an sqlalchemy url to add it. You can then go to add tables and add tables which already exist in the database. While adding database/datasource, there's no check whether the physical entity is present. (unless can do explicit test connection while adding), so you were able to add it (not create it). ... View more

@Ketan Kirange This seems to be correct API. Is the metrics collector installed on the same host? (192.168.56.101), 6188 is the default port for AMS. If you have Ambari, you can search for timeline.metrics.service.webapp.address in Metrics collector configs, alternatively you can search for this property in ams-site.xml in the node where it's installed. Or there's a problem in running collector daemon, you can check /var/log/ambari-metrics-collector/ambari-metrics-collector.log/out to check it. https://cwiki.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification is the standard reference. ... View more

You can set -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps flags in YARN_NODEMANAGER_OPTS and then view nodemanager logs in GC visualizer like gceasy.io. This error occurs when all objects are referenced/live and subsequent GC cycles can't reclaim > 2% of heap space. https://plumbr.io/outofmemoryerror/gc-overhead-limit-exceeded ... View more

Check whether SPARK_HOME in interpreter settings points to correct pyspark. Is it set to below value? SPARK_HOME /usr/hdp/current/spark2-client/ Where are you setting spark properties, in spark-env.sh or via Zeppelin? Check this thread: https://issues.apache.org/jira/browse/ZEPPELIN-295 Do spark.driver.memory=4G, spark.driver.cores=2. Check spark.memory.fraction (If it's set to 0.75, reduce it to 0.6) https://issues.apache.org/jira/browse/SPARK-15796 Check logs-> do tail -f /var/log/zeppelin/zeppelin-interpreter-spark2-spark-zeppelin-{HOSTNAME}.log in zeppelin host. ... View more

I also tried that once and it didn't seem to work for some reason. Please try using 'screen' utility. https://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/ Use Ctrl+a+c to create a new one, run your script there without nohup, that is ./run_beeline_hql.sh and detach from that session by using Ctrl+a+d. The process will keep running in the background which you can check by ps. ... View more

Try including the option --driver com.mysql.jdbc.Driver in the import command. ... View more

Good to hear that anobi. I could not find how to restrict sessions to a particular value. However if you set this spark.sql.hive.thriftServer.singleSession true. Only 1 session can be run. This doesn't scale very well. Please run spark.conf.getAll(), you may find other properties related to num sessions. Also please accept/upvote any answers if they helped you in concept. Thank You ... View more

@anobi do Did you try setting spark.sql.thriftServer.incrementalCollect true? I am not running multiple queries at a time, so maybe because of that I'm not seeing this, Try decreasing number of simultaneous sessions after setting incremental to true. ... View more