About mbigelow

mbigelow · ‎10-13-2017

Kerberos service principals have three parts, the service name, the hostname, and the domain name. The hostname must be in the formation of fully qualified domain name. That is why the service is looking for it in that format while the keytab does not contain an entry for that principal. Recreate the keytab file with the principal in the correct format and you should be good.

mbigelow · ‎10-13-2017

I don't have anything to back up but by my experience this is expected. Weights are a percentage and overall the Fair Scheduler works to ensure equal share. There are other settings you can use to ensure that the high queue gets more but with just this I would expect that both jobs get roughly half of the resources. This would obviously change if more jobs were added to either queue or if you have more than just two queues.

mbigelow · ‎10-13-2017

That error typically means that it never was able to make a connection. The IP address listed in the Host field has two octets with invalid values: 10.92.869.876. The maximum for each octet is 255.

mbigelow · ‎10-13-2017

It is a group. By default Hadoop create the user hdfs in the group hdfs. The first statement does make it confusing but assumes the defaults as that is the only user in the group. You could add users to the group as well (not recommended). The last portion referencing the Kerberos principal is just pointing out that it isn't enough to have a user in the superusergroup/supergroup they also need a valid Kerberos principal. In reality, the users in the group you assign to that property will have Kerberos principals already. I also recommend, as Cloudera does, to not use the default hdfs group.

mbigelow · ‎10-13-2017

What user do you have set as the Sentry 'god'? sentry.service.admin.group A user in this list or in a group in the list will be the only one that is able to create the initial roles until some delegation is granted.

mbigelow · ‎10-13-2017

https://spark.apache.org/docs/1.6.2/monitoring.html This is old but still useful. http://hadooptutorial.info/yarn-web-ui/ I know you can get the info from the RM UI. You should be able to get it from the Spark History UI through curl commands as well. The information could then be fed into Ganglia.

mbigelow · ‎10-04-2017

To date it is not possible. Cloudera did announce Cloudera SDX which claims it can be done with on-prem already and SDX provides the same in the cloud through Altus.

mbigelow · ‎10-04-2017

It is an issue with the installation. I don't know precisely what is the issue though. You can disable it, or set it to permissive, complete the installation, and then revert it back. I have always just kept it off, but presumably, you would need to repeat this for each upgrade.

mbigelow · ‎09-25-2017

That appears to be between the ZK servers. Anything useful in the SM logs?

mbigelow · ‎09-24-2017

Looking for something that shows which znode it is accessing as that will give us a clue as to what it is doing. I would try INFO before DEBUG but you may have to go to that level. Also, try the SM logs since it is the client and may have the info there. As an example the below line shows that the HBase client was trying to access the znode /hbase/hbaseid This is what I am thinking should be in the logs to help us figure this out. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

mbigelow · ‎09-24-2017

Those are the defaults and should be good. Hmm, the Service Monitor is what is running the canary test and the likely source of the connections. We need to figure out what SM is doing. Either go through the ZK logs to determine what node it is accessing. This may require you to increase the logging level. I want to make sure that I got this right. The Canary test is disabled, since being disabled you have restarted the Service Monitor, and since restarting the connections from SM to ZK have climbed until hitting the maximum of 1000. Do I have that right?

mbigelow · ‎09-24-2017

I have never seen such and issues with ZK and SM. What do you have set for the SM timeout values in ZK. I may have not seen this as I have these set. This alone should prevent SM from consuming all of the connections to ZK. ZooKeeper Canary Connection Timeout ZooKeeper Canary Session Timeout ZooKeeper Canary Operation Timeout

mbigelow · ‎09-20-2017

The RM UI and the Spark History are what you are looking for. The yarn command can also be used in a pinch.

mbigelow · ‎09-14-2017

It is late so I am not recalling the specifics but yes, I recommend always creating the Hive table definition outside of Spark. I vaguelyl recall that if you let Spark create it other services can't use it and Spark having issues translating the metadata correctly to the Hive specification.

mbigelow · ‎09-14-2017

These can be set globally, try searching for just spark memory as CM doesn't always include the actual setting name. These can be set per job as well. Spark-submit --executor-memory https://spark.apache.org/docs/1.6.0/submitting-applications.html

mbigelow · ‎09-14-2017

You need to increase the HS2 heap size as whatever it is at is too low to process and return that much data for your query. In CM, browse to the Hive service Configuration tab and search for 'Java Heap Size of HiveServer2 in Bytes'. I don't know what you have but increase it by 1 GB and test.

mbigelow · ‎09-13-2017

1. To install the new license you don't need to have internet access. It can upload the file from your local machine. 2. I only skimmed but I think it only requires that as it assumes that Cloudera Express clusters are using the embedded database. You are using an external DB already so it shouldn't be needed. 3. It should be non-destructive as it is just updating the licensing and unlocking features within CM.

mbigelow · ‎09-13-2017

I just tried it. It is just a POST to the /api/v10/users endpoint. curl -u uname:passw -H "Content-Type: application/json" -X POST -d '{ "items" : [ { "name" : "matt", "password" : "test" } ] }' http://cm_host:7180/api/v15/users

mbigelow · ‎09-13-2017

I think 'No storage dirs specified.' is referencing your dfs.data.dirs. Also, it is possible that the env vars like HADOOP_CONF_DIR are not set correctly for the session you are running that command in. As for the JN error, it seems that it is trying to format the NN but data already exists in the JN edits directory. Was NN HA working prior to Kerberos being enabled? If you are cool with formatting the NN then you are likely fine with manually removing the data in the JN edits directory. I would back it up in case and then remove it and see if the NN can come online. Also, did you have NN HA enabled and then disabled it? This is the only time I have seen data already in place in the JN edits directory. Rolling back NN HA in CM does not clear out this data.

mbigelow · ‎09-13-2017

The only way I can think of would be to have the Spark2 gateway installed on a node that doesn't have the Spark1 gateway or any Spark1 roles. Then create a symlink of spark2-submit to spark-submit.

mbigelow · ‎09-13-2017

Real world experience may differ from the docs. Right now you are seeing a large number of GC pauses. The likely cause is running out of heap space. You can try tuning the GC setting as well.

mbigelow · ‎09-13-2017

It is likely just that you have enough service entities to require a larger heap size. The default of 1 GB Java Heap should be increased.

mbigelow · ‎09-13-2017

Can you share graphs showing the thread leak? Also, I don't see this patch for the bug in any CDH release to date. So you will need to apply the workaround listed in the patch notes or in the other community thread. https://issues.apache.org/jira/browse/HIVE-16949

mbigelow · ‎08-16-2017

When CM creates a principal it sets the password and creates the keytab file. This can be found in the running process directory for the process, /var/run/cloudera-scm-agent/process. Why do you want to auth as this process's user?

mbigelow · ‎08-16-2017

I would test with the hdfs command first to ensure that HDFS with Kerberos is good. On .a node with the HDFS Gateway installed: kinit <enter password> hdfs dfs -ls / Can you share you jaas.conf file? For the Java program, I believe there are a few more config settings that tell a client to use Kerberos. I don't recall them off the top of my head. I would try just using the hdfs and core site files in the configuration object.

mbigelow · ‎08-16-2017

Log into the postgresql instance you installed for CM and Hive, and create the hive user and give it access to the metastore db. Then update the Hive configuration to use this account.

mbigelow · ‎08-16-2017

Did you do these steps prior to Level 1? https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_sg_tls_browser.html#xd_583c10bfdbd326ba-7dae4aa6-147c30d0933--7a61 Did you check that your keystore contains the CM certificate and has the correct hostname? Is the keystore file readable by the CM process user?

mbigelow · ‎08-07-2017

You need to add the Spark2 remote parcel URL to Cloudera Manager. After that you should have the option to download it and distribute it to the cluster.

mbigelow · ‎08-02-2017

Run 'jar -tf /opt/spark/yarn/spark-2.1.0-yarn-shuffle.jar | grep -i YarnShuffleService' This will tell you if the jar file contains the class with the correct name.

mbigelow · ‎08-02-2017

Why won't it work? Have you tried /tmp and /tmp/hive/<user.name>. The alternative if quotas can't be applied to /tmp or its subdirs is to set alerts for HDFS capacity or disk space on the disks hosting the DFS directories.

Online	Offline
Last Visited	‎03-25-2019 05:55 PM

Date Registered	‎08-16-2016 08:51 PM
Last Visited	‎03-25-2019 05:55 PM
Total Messages Posted	642
Total Kudos Received	128

Re: Configuring the HDFS superuser in Kerberos

Re: Hive process crash

Re: Upgrade from CDH 5.11 Express to Enterprise

Re: Adding user to Cloudera Manager using REST AP...

Re: Running in non-interactive mode, and data appe...

Re: Issue during kerberos deployment

Re: Fair scheduler preemption and resource pools w...

Re: Impala through ODBC from PERL SSL Connection E...

Re: Configuring the HDFS superuser in Kerberos

Re: HUE with openLDAP, SENTRY enabled (OpenLdap in...

Re: View resources allocated to particular applica...

Re: Single Sentry for two clusters

Re: Why I do need to turn off SElinux?

Re: Zookeeper connections are taken up by service ...

Re: Zookeeper connections are taken up by service ...

Re: Zookeeper connections are taken up by service ...

Re: Zookeeper connections are taken up by service ...

Re: View resources allocated to particular applica...

Re: pyspark <dataframe>.write.saveAsTable not crea...

Re: spark.yarn.executor.memoryOverhead

Re: Hive process crash

Re: Upgrade from CDH 5.11 Express to Enterprise

Re: Adding user to Cloudera Manager using REST AP...

Re: Running in non-interactive mode, and data appe...

Re: Use SparkLauncher with Spark2

Re: Service Monitor Timeout

Re: Service Monitor Timeout

Re: CDH5.11.0 have bug in hiveserver2

Re: SPNEGO service on an existing node

Re: Unable to access HDFS after enabling kerberos ...

Re: HIVE metastore failed to start (password authe...

Re: Issue After enabling the TLS level 1 encryptio...

Re: Problem with spark2 CSD

Re: Namenodes don't start after add spark_suffle t...

Re: HDFS /tmp filesystem is filling up rapidly and...