Hi,
thanks for your quick responses.
I want to use it for backup so I need the preserve flag.
Further on I do not want to care about users and permissions. The copy has to have the same user and permission settings.
Like using the Hadoop FS shell commands. These commands are quite fast for smaller data, like 100 times a 100 MB file,
but quite slow for large files.
I want to exploit yarn to use all datanodes of the cluster for the backup. Each datanode has mounted the backup volume via nfs.
This test is just on a single node.
The user how runs the backup needs root permission to execute the chown.
The only chance I see is using LinuxContainerExecutor.
But super users are banned from running Yarn
jobs.
-rw-r--r-- 1 root hadoop 1.1K Oct 1 22:02 container-executor.cfg
yarn.nodemanager.local-dirs=/var/hadoop/yarn/local
yarn.nodemanager.log-dirs=/var/hadoop/yarn/log
yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=hdfs,yarn,mapred,bin
min.user.id=500
It found an example using
allowed.system.users=mapr
Is this recommended on Hortonworks ? Does not look like as Ambari overwrites the settings.
Mqureshi, I tried your approach as well. Does not help Thank you anyway.
[datameer@dn190 wconf]$ diff core-site.xml.ori core-site.xml
34a35,54
> <name>hadoop.proxyuser.yarn.groups</name>
> <value>*</value>
> </property>
>
> <property>
> <name>hadoop.proxyuser.yarn.hosts</name>
> <value>*</value>
> </property>
>
[datameer@dn190 ~]$ time hadoop --config wconf distcp -prbugpct -m 100 -overwrite /wol/h1 file:/tmp/backup1/h1
17/10/02 12:34:15 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=true, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=100, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[REPLICATION, BLOCKSIZE, USER, GROUP, PERMISSION, CHECKSUMTYPE, TIMES], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[/wol/h1], targetPath=file:/tmp/backup1/h1, targetPathExists=true, filtersFile='null'}
17/10/02 12:34:15 INFO impl.TimelineClientImpl: Timeline service address: http://dn190.pf4h.local:8188/ws/v1/timeline/
17/10/02 12:34:15 INFO client.RMProxy: Connecting to ResourceManager at dn190.pf4h.local/192.168.239.190:8050
17/10/02 12:34:16 INFO client.AHSProxy: Connecting to Application History server at dn190.pf4h.local/192.168.239.190:10200
17/10/02 12:34:16 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 97; dirCnt = 0
17/10/02 12:34:16 INFO tools.SimpleCopyListing: Build file listing completed.
17/10/02 12:34:16 INFO tools.DistCp: Number of paths in the copy list: 97
17/10/02 12:34:16 INFO tools.DistCp: Number of paths in the copy list: 97
17/10/02 12:34:17 INFO impl.TimelineClientImpl: Timeline service address: http://dn190.pf4h.local:8188/ws/v1/timeline/
17/10/02 12:34:17 INFO client.RMProxy: Connecting to ResourceManager at dn190.pf4h.local/192.168.239.190:8050
17/10/02 12:34:17 INFO client.AHSProxy: Connecting to Application History server at dn190.pf4h.local/192.168.239.190:10200
17/10/02 12:34:17 INFO mapreduce.JobSubmitter: number of splits:97
17/10/02 12:34:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1506885218764_0007
17/10/02 12:34:17 INFO impl.YarnClientImpl: Submitted application application_1506885218764_0007
17/10/02 12:34:17 INFO mapreduce.Job: The url to track the job: http://dn190.pf4h.local:8088/proxy/application_1506885218764_0007/
17/10/02 12:34:17 INFO tools.DistCp: DistCp job-id: job_1506885218764_0007
17/10/02 12:34:17 INFO mapreduce.Job: Running job: job_1506885218764_0007
17/10/02 12:34:23 INFO mapreduce.Job: Job job_1506885218764_0007 running in uber mode : false
17/10/02 12:34:23 INFO mapreduce.Job: map 0% reduce 0%
17/10/02 12:34:26 INFO mapreduce.Job: Task Id : attempt_1506885218764_0007_m_000000_0, Status : FAILED
Error: ExitCodeException exitCode=1: chown: changing ownership of `/tmp/backup1/h1/_SUCCESS': Operation not permitted
[datameer@dn190 ~]$ ll /tmp/backup1/h1/
total 895M
-rw-r--r-- 1 yarn hadoop 100M Oct 2 12:34 part-m-00007
-rw-r--r-- 1 yarn hadoop 100M Oct 2 12:34 part-m-00001
-rw-r--r-- 1 yarn hadoop 100M Oct 2 12:34 part-m-00000
-rw-r--r-- 1 yarn hadoop 100M Oct 2 12:34 part-m-00004
:
