Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

oozie accessing hive metastore issue

oozie accessing hive metastore issue

Explorer

Hello,

I want to run the Hcatalog listener for change in partition and subsequently call to some shell commands. I am getting this below error while running the coordinator job. Please help regarding this error.

ERROR CoordActionInputCheckXCommand:517 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-180119101824715-oozie-oozi-C] ACTION[0000005-180119101824715-oozie-oozi-C@2] XException, 
org.apache.oozie.command.CommandException: E1021: Coord Action Input Check Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [No FileSystem for scheme: thrift]
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:237)
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.


java:63)
	at org.apache.oozie.command.XCommand.call(XCommand.java:287)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [No FileSystem for scheme: thrift]
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.pathExists(CoordOldInputDependency.java:232)
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.checkListOfPaths(CoordOldInputDependency.java:200)
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.checkPullMissingDependencies(CoordOldInputDependency.java:172)
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedInput(CoordActionInputCheckXCommand.java:323)
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:173)
	... 7 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [No FileSystem for scheme: thrift]
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:561)
	at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:184)
	at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:100)
	at org.apache.oozie.command.coord.CoordCommandUtils.pathExists(CoordCommandUtils.java:877)
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.pathExists(CoordOldInputDependency.java:220)
	... 11 more
Caused by: java.io.IOException: No FileSystem for scheme: thrift
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2793)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
	at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:553)
	at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:551)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:551)
	... 15 more

coordinator.xml

<coordinator-app name="coordinator-shell-${environment}"
                 frequency="* * * * *"
                 start="${startDate}" end="${endDate}"
                 timezone="UTC"
                 xmlns="uri:oozie:coordinator:0.4">
    <parameters>
        <property>
            <name>startDate</name>
        </property>
        <property>
            <name>endDate</name>
        </property>
        <property>
            <name>jobTracker</name>
        </property>
        <property>
            <name>nameNode</name>
        </property>
        <property>
            <name>hcatNode</name>
        </property>
        <property>
            <name>environment</name>
        </property>
        <property>
            <name>db</name>
        </property>
        <property>
            <name>table</name>
        </property>
        <property>
            <name>country</name>
        </property>
        <property>
            <name>scriptPath</name>
        </property>
    </parameters>


    <datasets>
        <dataset name="raw-logs" frequency="${coord:minutes(1)}" initial-instance="2018-01-18T00:00Z" timezone="UTC">
            <uri-template>thrift://sandbox.hortonworks.com:9083/${db}/${table}/dt=${YEAR}-${MONTH}-${DAY};country=${country}</uri-template>
        <done-flag></done-flag>
        </dataset>
    </datasets>


    <input-events>
        <data-in name="input" dataset="raw-logs">
            <instance>${coord:current(0)}</instance>
        </data-in>
    </input-events>


    <action>
        <workflow>
          <app-path>${nameNode}/user/oozie</app-path>
          <configuration>
            <property>
              <name>oozie.service.HCatAccessorService.hcat.configuration</name>
              <value>/${nameNode}/user/oozie/lib/hive-site.xml</value>
            </property>
            <property>
              <name>jobTracker</name>
              <value>${jobTracker}</value>
            </property>
            <property>
              <name>nameNode</name>
              <value>${nameNode}</value>
            </property>
            <property>
              <name>environment</name>
              <value>${environment}</value>
            </property>
            <property>
              <name>scriptPath</name>
              <value>${scriptPath}</value>
            </property>
          </configuration>
        </workflow>
    </action>
</coordinator-app>


workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="shell-wf-${environment}">
    <parameters>
      <property>
          <name>jobTracker</name>
      </property>
      <property>
          <name>nameNode</name>
      </property>
      <property>
          <name>environment</name>
      </property>
      <property>
          <name>scriptPath</name>
      </property>
    </parameters>


    <start to="shell-node"/>


    <action name="shell-node">
      <shell xmlns="uri:oozie:shell-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>${nameNode}/user/oozie/lib/hive-site.xml</job-xml>
        <configuration>
          <property>
            <name>oozie.libpath</name>
            <value>/user/oozie/share/lib</value>
          </property>
          <property>
            <name>oozie.launcher.mapred.job.queue.name</name>
            <value>${environment}</value>
          </property>
          <property>
            <name>mapreduce.job.queuename</name>
            <value>${environment}</value>
          </property>
          <property>
            <name>oozie.launcher.mapreduce.map.memory.mb</name>
            <value>512</value>
          </property>
          <property>
            <name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>
            <value>512</value>
          </property>
          <property>
             <name>oozie.action.sharelib.for.hive</name>
             <value>hive,hcatalog</value>
        </property>
        <property>
            <name>hive.metastore.uris</name>
            <value>thrift://sandbox.hortonworks.com:9083</value>
        </property>
        </configuration>
        <exec>sample.sh</exec>
        <file>${scriptPath}/sample.sh</file>
        <file>lib/hive-site.xml</file>
        <capture-output/>
      </shell>


      <ok to="end"/>
      <error to="fail"/>
    </action>


    <kill name="fail">
      <message>Shell failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>


calling command:

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie \
    -Doozie.coord.application.path hdfs://sandbox.hortonworks.com:8020/user/oozie/coordinator.xml \
    -Doozie.use.system.libpath true \
    -DjobTracker sandbox.hortonworks.com:8032 \
    -DnameNode hdfs://sandbox.hortonworks.com:8020 \
    -DhcatNode thrift://sandbox.hortonworks.com:9083 \
    -Ddb=default \
    -Doozie.action.sharelib.for.hive=hive,hcatalog \
    -Dtable=employee\
    -Dcountry=US\
    -Denvironment default \
    -DstartDate 2018-01-19T12:38Z \
    -DendDate 2020-01-17T13:00Z \
    -DscriptPath script \
    -verbose \
    -debug \
    -run


Thank you!

1 REPLY 1

Re: oozie accessing hive metastore issue

Explorer

Can anyone please help me in this ...

Don't have an account?
Coming from Hortonworks? Activate your account here