I'm trying to install Spark 1.6.0 on an HDP 2.4 cluster. I've managed to successfully remove Spark 1.6.1 using the ambari API, but now I cannot find a way to replace the current spark version with 1.6.0. I thought I'd find a similar guide to the one explaining how to install Zeppelin (https://community.hortonworks.com/articles/34424/apache-zeppelin-on-hdp-242.html) that will allow me to erase the current Spark repo from the ambari server and replace it with another version of Spark, but I can't seem to find the Spark repo I can use for this. Any help or guidance for doing this process? ... View more

I'm trying to upload files into an HDP cluster running on AWS from a separate EC2 instance. What is the best practice to achieve this? I can make the EC2 instance authenticate with the HDP datanode using ssh, but I would rather just upload the file directly to HDFS using the CLI, but I'm not sure how to do this. ... View more

I'm trying to use Hive view to access our Hbase DB in our HDP cluster. When I go to Hive view the database explorer is empty, what should I do to allow Hbase to appear as one of the DBs available for querying? I have tried following this guide - http://hortonworks.com/blog/hbase-via-hive-part-1/ but found it a bit unclear and perhaps out dated. Thanks. ... View more

I am running an HDP cluster on AWS using Ambari, and I find myself struggling with an architectural dilemma. Since our HDFS volumes are mounted on EBS, my computation and storage are separated by default, and so the closest I can bring these two units together (e.g a spark job and it's destination HDFS node) is running the job on the same EC2 instance the EBS volume is attached to. Now, in order to be able to scale my computational components and my storage components separately, I have divided the default blueprint provided by Ambari to instances containing the datanode and hbase_regionserver components, and instances containing only the nodemanager component (and metrics_monitor on all nodes, naturally). The purpose of this setup is to be able to launch spark jobs on machines designed for computation purposes only, allowing me to downscale quickly without having to redistribute my HDFS data, and scaling my HDFS nodes up only when I'm running low on space. Does this design make any sense, or does it go against HDP best-practices? Thanks, Yaron. ... View more

I'm following all the steps in this page - https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_ambari_reference_guide/content/ch_changing_the_jdk_version_on_an_existing_cluster.html Still, after following through all the steps mentioned, default java on all of my nodes stays 1.7. My cluster is running HDP 2.4.2.0-258 and was launched using Cloudbreak 1.3. I also can't find a method to restart nodes from within the Ambari UI, So I do this through my EC2 console. Any help would be received with gratitude, Yaron. ... View more

We are launching our cluster on AWS using Cloudbreak, using EBS volumes as our data store for hadoop. I wanted to know if it's possible to detach these volumes and attach them to another cluster, in case a need to move to a new cluster arises (in order to change intsance types, move to another region or any other reason that might come up) Also, are there any best practices for such a scenario? Thanks. ... View more

Trying to create a cluster on AWS using Cloudbreak AMI. After a profile has been configured with proper AWS creds and using mostly resources that came predefined with Cloudbreak cluster creation fails with the following error - /cbreak_cloudbreak_1 | 2016-07-17 14:13:16,231 [reactorDispatcher-85] accept:38 INFO c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Sending notification with data: CloudbreakEvent{id=22, eventType='CREATE_FAILED', eventTimestamp=Sun Jul 17 14:13:16 UTC 2016, eventMessage='Ambari cluster could not be created. Reason: com.sequenceiq.cloudbreak.orchestrator.exception.CloudbreakOrchestratorFailedException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Duplicate key 10.0.10.140', owner='40cc932b-8177-4429-95f9-cf54999cb3ef', account='seq1234567', cloud='AWS', region='us-east-1', blueprintName='hdp-spark-cluster', instanceGroup='null', blueprintId=2', stackId=2', clusterId=2', clusterName=horton-cluster-new} Any further details I've omitted will be gladly supplied, Thanks. ... View more