There is no additional charge from Hortonworks for selecting sandbox VM. However, after 30 days of free trial, you may get billed from Azure for any usage as a resource usage time (not for selecting this specific Hortonworks sandbox) ... View more

You can take a look at http://hortonworks.com/blog/hortonworks-sandbox-azure/ If you are new to azure, you will get one month free trial that you can use to try hortonworks sandbox on azure. ... View more

You can take a look at http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/ yarn.nodemanager.local-dirs: This is a comma separated list of local-directories that one can configure to be used for copying files during localization. The idea behind allowing multiple directories is to use multiple disks for localization – it helps both fail-over (one/few disk(s) going bad doesn’t affect all containers) and load balancing (no single disk is bottlenecked with writes). Thus, individual directories should be configured if possible on different local disks. You can follow same approach for container logs as well. ... View more

Generally not a good idea to use /var/log for yarn.nodemanager.log-dirs which are container logs. Typically, we direct these logs to all the Data mount points (like /grid/N/yarn/log). Same thing for yarn local (/gird/N/yarn/local). This can help with reducing all your IO going to your OS disk (where you typically have /var/log) ... View more

You can add a comma separated list of local/log directories that go to different disks instead of a single folder there to avoid location getting full. It is not a good idea to try to write it to hdfs (even if its possible which I doubt) ... View more

Correct. ... View more

We had a comparative run between hive on tez and spark sql and have run into multiple outliers on sparksql that took a long time. Are you seeing these issues with a single query or have run into this on multiple runs? If you are using ORC, you can set spark.sql.orc.filterPushdown to true. You can also try increasing executor memory. But you need to look at logs to see where its taking this time and if there are any GC issues. ... View more