Either oozie or cron should work. As for how to tell it the env, my first inclination is to look for something at the OS level that you can check like hostname. If the hostname has something that identifies the environment, read, parse, and detect. ... View more

I have been able to use multiple aliases with a single host in CDH. Not multiple hostnames though. What did you hosts file look. It probably is related to how Hadoop does hostname lookups. It used specific Java methods that aren't identical to the hostname CMD, at least across all platforms. ... View more

I'm not terrible familiar with Oozie but I believe the launcher was desperately from the actual job. Also, from the log "-Xmx4096m -Xmx4608m" it is launching with 4 GB container size and the heap is set to 3 GB. Is it set in the Oozie job settings? ... View more

Ok, I added as a friend and updated my privacy to display my email to friends. Let me know if you can't get to it. Do you want me to review it and make sure it is kosher? ... View more

It will work. This will diminish the network throughput and could impact the cluster performance if the typical workload is Network IO bound. In my experience, with predominantly 10 Ge networks, I have not been bound by the network running at the default 1500. ... View more

The properties files is just a list of k/v pairs that then get applied in the workflow.xml. To achieve what you want you will need to write a script (Bash, Python, Perl, etc.) that detects the environment and then updates the nameNode and jobTracker values (those are the two big ones I recall that are cluster specific). ... View more

Track down container container_e29_1484466365663_87038_02_000001. It is most likely a reducer. I say that since you said both the Map and AM container size was set to 2 GB. Therefor the Reduce container size must be 3 GB. Well, in theory the user launching it could have overridden any of them. What is the value of mapreduce.reduce.memory.mb? Lets try another route as well, in the RM UI, in the job in question, does it have any failed maps or reducers? If yes, drill down to the failed one and view the logs. If not, then the AM container OOM'd. From my recollection though, that is the line the AM logs concerning one of the containers it is responsible for. Anyway, the short of it is, either the Reduce container size is 3 GB or the user set their own value to 3 GB as the values in the cluster configs are only the defaults. ... View more

Does this MR job access HBase at all? This error indicates that the Region trade_all was not accessible. Any errors on the HBase RegionServers? Access the HBase Master UI to see what RS are serving this region and split. ... View more

What is default_realm set to in your krb5.conf? What is in the new keytab? klist -kt dreeves_local.keytab And try kinit as follows: kinit -C username or kinit -C username@domain From the Cloudera doc you linked. "Make sure all hosts in the cluster have a Linux user account with the same name as the first component of that user's principal name. For example, the Linux account joe should exist on every box if the user's principal name is joe@YOUR-REALM.COM. You can use LDAP for this step if it is available in your organization." So if your principal and AD account is dreeves@REALM.COM, you need to ensure that the account dreeves exist on all nodes, either using a local Linux account or using LDAP integration. This account also needs to have the same UID/GID across all nodes. You have this in place now for dreeves so we just need to workout authenticating using the with kinit and you should be good. The output of the above commands should help. The expectation is that either kinit command works and I suspect that a non-valid principal or no principals were added to the keytab file. You could use the original keytab you made that works but try it under the dreeves account you made. ... View more

It is a common misconception. Unless you config Hadoop to use LDAP to look up users it will use the default shell based lookup. So you still need local users or implement LDAP for RHEL. It looks like you have that pariltially in place. UID and gids are in place in AD. I don't know how you did it but you need to see how you can present the account without the domain name. ... View more

Go to the node that the mapper or reducer failed and run 'id dreeves'. This need to return a user. If it does not the worker is not able to operate as the user. I don't know why exactly the other commands worked. Did the correct ownership get applied to the file or does it just show a UID and gid. I have seen that in the case with a user is present on the client, edge node, and Namenode but are different. ... View more

It is hitting a timeout while monitoring the status. This is probably similar to the 10 minute timeout for MR, so if a task doesn't provide a progress update for 10 minutes the tasks fails. I haven't confirmed this for Spark. The bigger question is why is it not getting a status update. What does the RM UI show for where the AM is running compared to where the tasks are running? I see that you are using the Hive CLI. Can you try from Beeline or another JDBC connection? ... View more

I don't know of any way. Hadoop in general doesn't care how long it takes; it is more concerned an auto-recover of the platform so that jobs can finish no matter what. You can limit the number of queries or jobs by user or group, you can limit the resources to users or groups. I just don't think there is a way to automatically kill jobs or queries running longer than X. I know other products, like Pepperdata, can track and alert you. It still require manually intervention. Can we step back and you explain what your issue is with long running jobs? As maybe the root cause can be addressed there so job do not run for so long or hold back others. ... View more

Can you post the container logs for one of the containers that was killed? In the RM UI drill down through the job until you get the list of Mappers/Reducers that succeeded or failed. Click through to a failed task and then open the logs. You should find an exception in it on the reason. The code mentioned usually does indicate a heap issue but I have seen it reported for other reason a container was killed, such as when preemption strikes. ... View more

Unfortunately, there isn't an option to use the local usernames without something else. The closest would be PAM. http://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_ig_hue_config.html#topic_15_6_1 ... View more

Yes, create a local account and group on all nodes, create a HDFS user directory, assign permissions.   HDFS supports POSIX like permissions.  You can enable HDFS ACLs as well to allow for more control.  Hive and Impala will recognize and use these.  The caveat is that without Kerberos or LDAP authentication enforcement is friviously.  It is really easy to spoof another account to get around the ACLs.   Note: HBase does not support any form of authorization without Kerberos. ... View more

If you need concurrent users, external is the way to go. Also, it eases management as the different services use different embedded dbs. On the DB HA front, CDH doesn't presume which DB you choose and therefore this isn't a focus on it. Find the DB HA docs for the DB you choose. ... View more

Blast it all my other response didn't make it. The short of it is that Hadoop by default and even with Kerberos is using a shell based group mapper. This means that it still does a group and user look up regardless of the auth mechanism. There is a n LDAP group mapping for Hadoop. I, and Cloudera, do not recommend it. Either integrate LDAP at the OS level or manage the accounts manually. ... View more

1. Yes it could.  I personally don't like the threshold.  It is not a great indicator of there being a small file issue.   2. The number reported by the DN is for all the replicas.  It could mean a lot of small files or just a lot of data.  At the defaults it could mean that the DN heap could use a boost although I always end up bumping it sooner.   3. Yes.   4. Yes.  Each file takes up one or more blocks.  The NN has to track it and it's replicas in its memeory.  So a lot of small files can chew through the NN heap quickly.  The DN heap is less concerned with Metadata associated with a block as it is related to the blocks being read,written, or replicated.   5. I'd worry less on the block count and more on the heap. ... View more

The cmd will use the instance profile from where it is launched. So if you want access but not for all you need to specify the key in the S3 URI. ... View more

The users and groups need to be available on the local OS of all nodes. This can be through LDAP integration or managed manually. ... View more

Can you share anything else like the krb5.conf or kdc.conf? What do you have set for the trusted realmn? The hostname seem to be in the realm hadoop.com.sg. I want to say that the krbtgt principal should be krbtgt/HADOOP.COM.SG/HADOOP.COM. Also try adding -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext to the command and posting the output. It will be a lot but should help nailed down where it is going wrong. ... View more

Later version of Hue, I think 3.7+, are not case insensitive for log in purposes. Linux is case sensitive though, john and John are different identities. What is your auth backend for Hue? I have not found any value. It was better all around to force lowercase names using the rule mapping. RULE:[1:$1]/L RULE:[2:$1]/L ... View more

You need to create them. None exist by default. You could replicate to the same cluster but a different folder but mostly it is used to replicate data and metadata to another cluster. ... View more

I have seen this when reinstalling the CM Agent without dropping and removing the agent from the cluster and CM. The reason this happens is that each agent generation a GUID to represent the host and agent. This is unique to each install. CM views the agents still registered to it as the old ones and not the new ones even though it is the same host and it prevents the new ones from registering to CM. In the CM host page, stop any existing roles, delete them, then deleted the agent. Then the new Host wizard should show the agents now reporting in correctly as unmanaged. ... View more

Yes check there. I don't know the HIve source code but I do know that HDFS still does a username/group lookup against the OS. ... View more

Ah that will do it as all new tables inherent the DB path unless specified in the Create table statement. There is no way to alter it through HIve/Impala. You will need to log into the metastore DB and change it there. You can find it in the <metastore_db_name>.DBS and I believe the column is just called LOCATION. Find the id for the default DB and run something like 'update DBS set LOCATION = 'hdfs://NN_URI:8020/user/hive/warehouse' where id = <default_db_id>;' ... View more

Does the user 'administrator' exist on the HS2 node, and preferable the rest of the nodes. Does the user have a HDFS user directory, /user/administrator, with full access to it? These items are what is needed for users to access the cluster and run jobs regardless of the means of authentication. ... View more

This is going to be rough. You could manually copy the data from the CM server over to each node. You could also deploy a new cluster to those some nodes. I got a feeling that either way the old configs will not be present any longer. Before doing anything I would try to take a backup of the cluster using the CM API. Then you can try to restore the configs from that if you end up with a new cluster with default configs. https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_intro_api.html ... View more

This may be a silly question, but does the test table exist prior to running the CTAS statement? ... View more