1. is for unsecured access 2. is for the Hive JDBC driver and looks mostly correct (comment below) 3. is for the Cloudera Hive JDBC driver JDBC connections will always be to the HS2 and not HMS. The hostname portion of the principal needs to be the FQDN of the HS2 server you are connecting to. Try the URI below. jdbc:hive2://<Servername.domain.com>:10000;AuthMech=1;KrbRealm=HADOOP.yyy.XXXX.COM;KrbHostFQDN=<Servername.domain.com>;KrbServiceName=hive ... View more

I don't know HDFS quotas well enough but should fit the bill. https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html In CM you can configure alerts to notify you when disk and HDFS is nearing capacity. ... View more

If it is still looking for node4 after it goes down then the standby is not in a valid state, could be a quorum issue between the journal nodes, or a state/fencing problem with ZK. Review the logs for these services during the test to try to glean more information. ... View more

impala-shell -i haproxy1:21000 -k --ssl Are you using the FQDN in the impala-shell command? i.e. impala-shell -i haproxy.company.local -k -ssl Is there an ssl certificate for the HAProxy and is it configured to use it. Is the CA cert for it in the PEM file that Impala is configured to use? ... View more

That shouldn't matter. I am using an ELB that is completely separate from the CDH cluster. Did you specify the FQDN in that setting and does the principal contain the FQDN? ... View more

After making this change did you Generate Missing Credential in the CM Security windows or manually create the account and SPN. I haven't done Impala but for HS2, after adding the LB info in the Hive configs it through a configuration warning that credentials were missing. I generated them, the warning disappeared, and the LB worked. ... View more

I can't seem to find anything but I thought you could change the prefix. I feel sure you can for MR jobs, but not sure for Hive. But if it is a MR property you could set that in your Hive session. The other thing to talk about here is that *_copy_1 is part of the Hive code for dynamic partitions. It checks before hand if 0000_0 already exist, possible from another reducer or another Hive process. It then appends _copy_# to protect the data. This will remain regardless of the prefix. So in theory, even if you went down to the millisecond, you could end up with identical files with the same name. Changing the prefix should help your case though, so try finding something on changing the output file prefix. ... View more

I think you are just missing the file extension for your jar file. Try changing /home/cloudera/Desktop/SparkMllib/target/SparkMllib-0.0.1-SNAPSHOT-jar-with-dependencies to /home/cloudera/Desktop/SparkMllib/target/SparkMllib-0.0.1-SNAPSHOT-jar-with-dependencies.jar The error is complaining that you did not specific a JAR file which should be the last entry prior to any application arguments. ... View more

The SparkSQL is because it is not using Hive. The rest of the errors and fact that it works in spark-shell tells me that it is likely a classpath issue. The spark-shell is getting all of the correct classes for Hive but your project in Intellij does not. ... View more

How did you set up NN AH? Can you provide the context of the two NN, their names, which one you took down so that make it a more clear on what is going wrong here? The error itself is complaining that the host cannot be resolved. This could be a config issue with the NN HA setup as there should be a service name that resolves, through the config files, to both host. So when NN1 goes offline it will try the NN2. This could be a host name resolution error as well. If DNS or hosts files are not set up correctly then it is simply not able to resolve NN2 from the client. ... View more

The versioning for Spark2 is odd and I won't try to explain as I don't understand it. Spark2 is separate from CDH though. I have Spark2 2.1 installed on CDH 5.11.1 and will be upgrading to Spark2 2.2 soon. I haven't seen any dependency information between Spark2 and CDH. Anyway, your issue is that Spark2 can only be installed using parcels which are specific to CM. So without CM you are not going to be able to install Spark2 from Cloudera. In theory you could get the packages from Apache or elsewhere and manually install it. ... View more

Did you check for new Parcels? Do you have an Anaconda category? Have you checked the CM server logs for any errors? Does the CM host have outbound internet access? ... View more

Look for the --netrc switch for your curl command. You can use this file to pass the username and password to the command. This keeps it out of the ps output and history but the file security must be maintained. The format should be the below but check man if needed. machine host.domain.com login myself password secret ... View more

The bug I mentioned is in CDH 5.11.1 and hasn't been fixed yet. The workaround is to change the number of threads available to the input listing process. Change this value and restart HS2. You should stop seeing such high number of threads. hive.exec.input.listing.max.threads = 1 ... View more

The reason the first query works is because it does not need any MR or Spark jobs to run. The HS2 or Hive client just read the data directly. The second query requires MR or Spark jobs to be ran. This is key to remember when testing or troubleshooting the cluster. Are you able to run Spark jobs out side of Hive? Try the below command but swap out to your jar version. spark-submit --class org.apache.spark.examples.SparkPi --master yarn --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 /opt/cloudera/parcels/SPARK/lib/spark/examples/jars/spark-examples_*.jar Also access the Spark History server to get to the driver and executor logs to try to get more details on the failure. ... View more

Yes. It may cause issues depending on how you are using it when ingestion data into it. What are you trying to do? I wonder if a Hive view would be better then a separate table. ... View more

I haven't set HA Proxy up for Impala, but I think you need a service principal for impala/<HAProxyHost>@REALM.COM in your KDC. The error is that the server is not found in the Kerberos database. ... View more

Can you share the commands you ran for each step? ... View more

In the Spark2 configs, ensure that the Hive service is enabled. This will include the Hive client configs for the Spark2 service. This will allow the SparkSession created by spark2-shell to have Hive support for the HMS on the cluster. I haven't tested actual Spark2 applications but with the above setup it should be as simple as using the .enableHiveSupport in the SparkSession builder method. Outside of that you would probably need to include the hive-site.xml or Hive HMS settings in the Spark Context configuration object and then us .enableHiveSupport. ... View more

Do you mean you have apps installed outside of Hadoop and that they are not install on all only on certain nodes? First, I do not recommend that. The worker nodes do the work and anything that detracts from that decreases the value they bring to the cluster. In the event that you need an app or binary so be used by jobs, you should have it on all workers. Then you can keep the configs the same across all nodes. The title and description make me a bit confused. So I am assuming that you have some apps installed on a worker nodes that are not part of Hadoop or CDH. If that is correct, the best way to handle it is to have two separate host templates for YARN that will give less resources to YARN on the host(s) that have less resources to give due to the external applications. On the topic of limiting jobs, you might be able to use Node labels. These are used to ensure certain types of jobs run on specific nodes. So you could use them to control what runs on that box, but it isn't as cut and dry as I think you expect. ... View more

You need to add the Anaconda repo to the Parcel Settings Remote Parcels Repo URL list. https://repo.continuum.io/pkgs/misc/parcels/ ... View more

There was a bug added, I thought it was added in CDH 5.11.1, but I may be off on that. It was added when the community added parallelism to the input path listing process. Threads are leaked and after some time this will slow down HS2. There is a workaround that I have confirmed works. Please review the below post and subsequent JIRA. https://community.cloudera.com/t5/Batch-SQL-Apache-Hive/HiveServer2-has-more-than-10K-waiting-daemon-threads-named-quot/td-p/57430 ... View more

The other time I have seen this the issue was that Spark2 had not been added as a service to the cluster and therefor the Spark2 gateway was not in place. This would include environmental and configuration settings for clients like spark2-shell. At a cluster screen click the Action menu and select 'Add a Service'. Follow the wizard and add Spark2 from the list of services available. ... View more

They should be able to set the executor memory size when launching the job. This will give it more heap. You can also set this globally in the Spark configs so that all Spark apps have the larger container size. ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --executor-memory 20G \ <------this setting /path/to/examples.jar \ 1000 spark.executor.memory ... View more

You can leave the password blank, it will then prompt the user for the password. curl -u user: https;//.. ... View more

It is possible that AWS is throwing an error that leads Director to state that it can't get the instance in time. The minimum required is only 1. You could test quickly by launching the same size in the same region to see if there is an issue on the AWS side. ... View more

Best practice is to not spread across AZ. This is not a support configuration and will cause issues. I was just migrating between AZ's and had issues trying to check by having some DNs in a different AZ and letting it replicate data over that way. This did not work and many timeouts happened. I could've tried tweaking the configs to get it to work but it wasn't worth it and having such high values would not be advisable in production. With that all said, I would have everything in 1 AZ in a PG and randomly assign racks to provide boundaries for replication. I say randomly as you don't know physically which nodes are closer, but so you don't benefit from reduce network traffic and reducing congestion but it will ensure that the second and third are assigned to rack other than the first replica was written. In all honesty, though, it matters little as even with all nodes in a single rack it will still assign the replicas to other nodes. The risk of AWS having a rack failure and it taking out the 3 nodes with all replicas for a single block is present in either scenario. Note: You may not be able to add additional nodes to a PG after it has been set up. It depends on AZ node availability for that instance type and AWS's black magik to be able to move instances around to get them in. I have had issues with not being able to add more nodes to an existing PG. You will likely have to do a full migration to grow the cluster (or be good friends with AWS). ... View more