Member since
07-31-2013
1924
Posts
462
Kudos Received
311
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1969 | 07-09-2019 12:53 AM | |
| 11877 | 06-23-2019 08:37 PM | |
| 9141 | 06-18-2019 11:28 PM | |
| 10126 | 05-23-2019 08:46 PM | |
| 4576 | 05-20-2019 01:14 AM |
05-01-2015
10:56 AM
Is the host you run the command on carrying a YARN gateway role, i.e. valid RM configs under /etc/hadoop/conf/yarn-site.xml? Do you see the word 'LocalJobRunner' in the output logs of the DistCp command when its running?
... View more
04-22-2015
02:51 AM
You can use on the to pass any variable name and substituted values around.
... View more
04-14-2015
08:35 AM
Thank you for following up with the found solution here! It will benefit others looking for similar info. We also recommend use of the S3A connector going forward, via the s3a:// scheme.
... View more
04-08-2015
05:59 AM
1 Kudo
Every block's replica location allocation gets a different set of DNs. They may all have one common DataNode if the writer client is running on a host that also runs a DataNode, but the other replicas will be randomly selected (within or outside of racks, depending on topology).
... View more
04-08-2015
05:49 AM
A pipeline is created to write multiple replicas for every single under-construction block. For every block's end, the pipeline is completed and closed, and a subsequent new pipeline is opened for the next block (if there is one). There's no notion of "number of pipelines" given that its a sequential operation - why do you seek such a number?
... View more
04-01-2015
07:20 AM
You can force the LCE to impersonate actual users in non-secure mode by turning "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" to false (default is true, which causes it to limit all users to "nobody"). Note though that enabling this ability requires that your users' accounts exist on all nodes in the cluster at the Unix level (i.e. id must return a valid ID for their jobs to work). If you instead just wanted to change the "nobody" user to some other static user, the config for that is "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" (default: "nobody"). If you go this way, you need to leave the earlier mentioned config as its default of true. These configs need to be changed in CM's YARN configuration page (either via direct fields if available, or via the yarn-site.xml), and they must reach the NodeManager's configs to apply. These configs are also documented in the yarn-default.xml: http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml Does this help?
... View more
04-01-2015
12:30 AM
Oozie actions (such as shell, java, etc.) are launched as regular (1-map) MR jobs, so YARN (or JobTracker, if MR1) usually takes care of resource management for Oozie workflows in the cluster.
... View more
03-30-2015
07:19 PM
You are correct in that Oozie does not yet offer a supported way to configure LDAP based authentication over its Web Server. However, it does ship with a simple 'example' LDAP auth filter to demonstrate its ability to accept custom login modules, which can be read and configured via http://archive.cloudera.com/cdh5/cdh/5/oozie/ENG_Custom_Authentication.html#Login_Server_Example. Does this help?
... View more
03-30-2015
05:22 AM
If you update the workflow.xml at the same location the coordinator is configured to look for it, it will automatically pick up the newer XML contents for the future workflow instances.
... View more
03-25-2015
06:04 AM
1 Kudo
The superuser for HDFS is 'hdfs' in CDH installations. Please run the command as the 'hdfs' user instead.
... View more