Member since
08-16-2016
642
Posts
131
Kudos Received
68
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3976 | 10-13-2017 09:42 PM | |
| 7472 | 09-14-2017 11:15 AM | |
| 3798 | 09-13-2017 10:35 PM | |
| 6032 | 09-13-2017 10:25 PM | |
| 6600 | 09-13-2017 10:05 PM |
06-18-2017
06:57 PM
I don't think it is creating the keytab file under that directory, which won't fail. The subsequent chmod command is failing as the keytab file doesn't exist. Strangely, I don't see the add princ ktutil command before the write to the keytab file. So I feel like it is not creating the keytab file because this is missing. This is easy to test an verify though. Using the account that the CM server process is running under, either cloudera-scm* or root, run ktutil and the wkt command as shown. See if it does write it. If it doesn't, try adding 'add princ principal@realmn' and then wkt. That should narrow down where the issue is coming from.
... View more
06-16-2017
08:08 PM
1 Kudo
It took a bit to track it down but here is the info you are looking for. There are really just two steps, download and put the jar in the Hadoop classpath, as you have already. The other step is to add the gs settings to the core-site.xml so that HDFS is aware of it as a additional filesystem (i.e. knows the class name and impl name). https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md
... View more
06-16-2017
07:17 PM
I would try loading the CA cert that signed the certs for the Kudu master and worker nodes into the JSSE truststore. Do this on the host you are running the Kudu app on. Then any Java applications will use it when negotiating TLS handshakes. You could also create a truststore and use that when launching the Kudu app. https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_sg_create_key_trust.html
... View more
06-16-2017
07:07 PM
Ok, the hosts file on the cluster nodes doesn't impact the CM UI. Check the CM logs for any errors (/var/log/cloudera-scm-server/). Get the IP address from the CM host and use that in the URL (http://<cm_ip>:7180). You can also verify that CM is runnning (ps -ef | grep cloudera-scm-server) and that it is listening on the right port (sudo netstat -alpn | grep :7180). You can also try access it from the CM host itself (curl http://localhost:7180). That will help to see if it is working without messing with name resolution.
... View more
06-16-2017
04:24 PM
It is part of the magic that happens when you install a gateway roles on a nodes using CM. It installs the binaries, download the client configs, sets env vars in such a way that no users on that nodes need to do it. It just works. I have not dug into it to determine how. I expected that it is wrapped up in scripts somewhere that get called when using the clients. Without it, you will need to do it all yourself. And JAVA_HOME isn't the only one, HADOOP_HOME, HADOOP_CONF, etc. The Hadoop docs should have some information on it. Or... you can add the machine in question to CM, add the gateway role, and then remove it from the cluster. You shouldn't need to uninstall the gateway role to remove it from the cluster. No license is required for the gateway roles either.
... View more
06-16-2017
04:15 PM
I'll be... it doesn't care about './'. I have only ever seen that used to launch scripts/executables without specifying the interpreter. Anyway, I am glad you got it worked out but something is not working as it should. The --files option should upload any files listed, also files in the working directory when spark-submit was ran, and any --jars should all be distributed across the cluster, either local or hdfs. You should not have to distributed files manually beforehand. Especially the keytab file as a compromise to it can be devastating. The keytab file should have been copied to the AM containers working directory. I think the issue there was that you listed the original full path in the jaas.conf. From the Cloudera Spark docs: "The full path to the file that contains the keytab for the principal. This keytab is copied to the node running the ApplicationMaster using the Secure Distributed Cache, for periodically renewing the login tickets and the delegation tokens. For information on setting up the principal and keytab, see Configuring a Cluster with Custom Kerberos Principalsand Spark Authentication." https://www.cloudera.com/documentation/enterprise/5-5-x/topics/sg_spark_auth.html
... View more
06-16-2017
11:50 AM
1 Kudo
Yeah, that is normal. Beeline is another client that connects to HS2 where the Hive cli connects directly to HMS. The latter has some security issues and is not scalable. In Hive 2.x the Hive cli will be a wrapper around beeline. You can ignore the message for now or start using beeline right out the gate.
... View more
06-16-2017
11:15 AM
2 Kudos
NTP or Chrony should work. From the Clock Offset Host Health check (CM 5.8.2). "This is a host health test that checks if the host's system clock appears to be out-of-sync with its NTP server(s). The test uses the 'ntpdc -np' (if ntpd is running) or 'chronyc sources' (if chronyd is running) command to check that the host is synchronized to an NTP peer and that the absolute value of the host's clock offset from that peer is not too large. If the command fails, NTP is not synchronized to a server, or the host's NTP daemon is not running or cannot be contacted, the test will return "Bad" health. The 'ntpdc -np' or 'chronyc sources' output contains a row for each of the host's NTP servers. The row starting with a '*' (if ntpdc) or '^*' (if chronyc) contains the peer to which the host is currently synchronized. No row starting with a '*' or '^*' indicates that the host is not currently synchronized. Communication errors and too large an offset between the peer and the host time are examples of conditions that can lead to a host being unsynchronized. Make sure that UDP port 123 is open in any firewall that is in use. Check the system log for ntpd or chronyd messages related to configuration errors. If running ntpd, use 'ntpdc -c iostat' to verify that packets are sent and recieved between the different peers. More information about the conditions of each peer can be found by running the command 'ntpq -c as'. The output of this command includes the association ID that can be used in combination with 'ntpq -c "rv "' to get more information about the status of each such peer. The command 'ntpq -c pe' can also be used to return a summary of all peers and the reason why they are not in use. If running chronyd, use 'chronyc activity' to check how many NTP sources are online/offline. More information about the conditions of each peer can be found by running the command 'chronyc sourcestats'. To check chrony tracking, issue the command 'chronyc tracking'. If NTP is not in use on the host, this check should be disabled for the host using the configuration options shown below. Cloudera recommends using NTP for time synchronization of Hadoop clusters. A failure of this health test can indicate a problem with the host's NTP service or configuration."
... View more
06-16-2017
11:12 AM
@Laith I see that the docs states this but I recall the health test description for time synchronization references both ntp and chrony. My understanding is that either may be used and the requirement is just that time be synchronized. I have only used NTP to date but that is due to familiarity with it. Anyway, to the OP, the only thing I can think of that may break is the CM health tests. If it only uses ntp commands like ntpq to validated that time is in synch then those test would report failures or unknown.
... View more