Support Questions
Find answers, ask questions, and share your expertise

How to cp multiple folders use distcp action ?

Highlighted

How to cp multiple folders use distcp action ?

Contributor

I want to copying from multiple sources, like this:

    • with sources
      • hdfs://nn1:8020/aa/dt=20160222
      • hdfs://nn1:8020/bb/dt=20160222
      • hdfs://nn1:8020/cc/dt=20160222
      • hdfs://nn1:8020/dd/dt=20160222
      • .... .... .... ....
      • hdfs://nn1:8020/zz/dt=20160222
    • and destination
      • hdfs://nn1:8020/user/aa/dt=20160222
      • hdfs://nn1:8020/user/bb/dt=20160222
      • hdfs://nn1:8020/user/cc/dt=20160222
      • hdfs://nn1:8020/user/dd/dt=20160222
      • .... .... .... ....
      • hdfs://nn1:8020/user/zz/dt=20160222

    and consider a legal copy using command line:

    distcp -update hdfs://nn1:8020/*/dt=20160222   /user/*/dt=20160222

    now how can i to do it using distcp action in oozie ?

    6 REPLIES 6
    Highlighted

    Re: How to cp multiple folders use distcp action ?

    Highlighted

    Re: How to cp multiple folders use distcp action ?

    Contributor

    my distcp action like below:

      <action name="distcp-node">
          <distcp xmlns="uri:oozie:distcp-action:0.1">
              <job-tracker>${jobTracker}</job-tracker>
              <name-node>${nameNode}</name-node>
              <prepare>
                  <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/user/*
              </prepare>
              <configuration>
                  <property>
                      <name>mapred.job.queue.name</name>
                      <value>${queueName}</value>
                  </property>
              </configuration>
              <arg>hdfs://nn1:8020/*/dt=20160222</arg>
              <arg>/user/*/dt=20160222</arg>
          </distcp>
          <ok to="end"/>
          <error to="fail"/>
      </action>  
    

    but it can not work, and error log like below:

    2369-error.png

    Highlighted

    Re: How to cp multiple folders use distcp action ?

    Highlighted

    Re: How to cp multiple folders use distcp action ?

    Contributor

    my workflow.xml context is :

    2402-workflow.png

    if replace "*" to a floder name , for example "myfloder", it worked ok.

    Highlighted

    Re: How to cp multiple folders use distcp action ?

    Highlighted

    Re: How to cp multiple folders use distcp action ?

    Contributor

    how can i resolved the issue ?