Support Questions
Find answers, ask questions, and share your expertise

oozie accessing hive metastore issue

Explorer

Hello,

I want to run the Hcatalog listener for change in partition and subsequently call to some shell commands. I am getting this below error while running the coordinator job. Please help regarding this error.

ERROR CoordActionInputCheckXCommand:517 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-180119101824715-oozie-oozi-C] ACTION[0000005-180119101824715-oozie-oozi-C@2] XException, 
org.apache.oozie.command.CommandException: E1021: Coord Action Input Check Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [No FileSystem for scheme: thrift]
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:237)
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.


java:63)
	at org.apache.oozie.command.XCommand.call(XCommand.java:287)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [No FileSystem for scheme: thrift]
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.pathExists(CoordOldInputDependency.java:232)
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.checkListOfPaths(CoordOldInputDependency.java:200)
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.checkPullMissingDependencies(CoordOldInputDependency.java:172)
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedInput(CoordActionInputCheckXCommand.java:323)
	at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:173)
	... 7 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [No FileSystem for scheme: thrift]
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:561)
	at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:184)
	at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:100)
	at org.apache.oozie.command.coord.CoordCommandUtils.pathExists(CoordCommandUtils.java:877)
	at org.apache.oozie.coord.input.dependency.CoordOldInputDependency.pathExists(CoordOldInputDependency.java:220)
	... 11 more
Caused by: java.io.IOException: No FileSystem for scheme: thrift
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2793)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
	at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:553)
	at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:551)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:551)
	... 15 more

coordinator.xml

<coordinator-app name="coordinator-shell-${environment}"
                 frequency="* * * * *"
                 start="${startDate}" end="${endDate}"
                 timezone="UTC"
                 xmlns="uri:oozie:coordinator:0.4">
    <parameters>
        <property>
            <name>startDate</name>
        </property>
        <property>
            <name>endDate</name>
        </property>
        <property>
            <name>jobTracker</name>
        </property>
        <property>
            <name>nameNode</name>
        </property>
        <property>
            <name>hcatNode</name>
        </property>
        <property>
            <name>environment</name>
        </property>
        <property>
            <name>db</name>
        </property>
        <property>
            <name>table</name>
        </property>
        <property>
            <name>country</name>
        </property>
        <property>
            <name>scriptPath</name>
        </property>
    </parameters>


    <datasets>
        <dataset name="raw-logs" frequency="${coord:minutes(1)}" initial-instance="2018-01-18T00:00Z" timezone="UTC">
            <uri-template>thrift://sandbox.hortonworks.com:9083/${db}/${table}/dt=${YEAR}-${MONTH}-${DAY};country=${country}</uri-template>
        <done-flag></done-flag>
        </dataset>
    </datasets>


    <input-events>
        <data-in name="input" dataset="raw-logs">
            <instance>${coord:current(0)}</instance>
        </data-in>
    </input-events>


    <action>
        <workflow>
          <app-path>${nameNode}/user/oozie</app-path>
          <configuration>
            <property>
              <name>oozie.service.HCatAccessorService.hcat.configuration</name>
              <value>/${nameNode}/user/oozie/lib/hive-site.xml</value>
            </property>
            <property>
              <name>jobTracker</name>
              <value>${jobTracker}</value>
            </property>
            <property>
              <name>nameNode</name>
              <value>${nameNode}</value>
            </property>
            <property>
              <name>environment</name>
              <value>${environment}</value>
            </property>
            <property>
              <name>scriptPath</name>
              <value>${scriptPath}</value>
            </property>
          </configuration>
        </workflow>
    </action>
</coordinator-app>


workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="shell-wf-${environment}">
    <parameters>
      <property>
          <name>jobTracker</name>
      </property>
      <property>
          <name>nameNode</name>
      </property>
      <property>
          <name>environment</name>
      </property>
      <property>
          <name>scriptPath</name>
      </property>
    </parameters>


    <start to="shell-node"/>


    <action name="shell-node">
      <shell xmlns="uri:oozie:shell-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>${nameNode}/user/oozie/lib/hive-site.xml</job-xml>
        <configuration>
          <property>
            <name>oozie.libpath</name>
            <value>/user/oozie/share/lib</value>
          </property>
          <property>
            <name>oozie.launcher.mapred.job.queue.name</name>
            <value>${environment}</value>
          </property>
          <property>
            <name>mapreduce.job.queuename</name>
            <value>${environment}</value>
          </property>
          <property>
            <name>oozie.launcher.mapreduce.map.memory.mb</name>
            <value>512</value>
          </property>
          <property>
            <name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>
            <value>512</value>
          </property>
          <property>
             <name>oozie.action.sharelib.for.hive</name>
             <value>hive,hcatalog</value>
        </property>
        <property>
            <name>hive.metastore.uris</name>
            <value>thrift://sandbox.hortonworks.com:9083</value>
        </property>
        </configuration>
        <exec>sample.sh</exec>
        <file>${scriptPath}/sample.sh</file>
        <file>lib/hive-site.xml</file>
        <capture-output/>
      </shell>


      <ok to="end"/>
      <error to="fail"/>
    </action>


    <kill name="fail">
      <message>Shell failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>


calling command:

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie \
    -Doozie.coord.application.path hdfs://sandbox.hortonworks.com:8020/user/oozie/coordinator.xml \
    -Doozie.use.system.libpath true \
    -DjobTracker sandbox.hortonworks.com:8032 \
    -DnameNode hdfs://sandbox.hortonworks.com:8020 \
    -DhcatNode thrift://sandbox.hortonworks.com:9083 \
    -Ddb=default \
    -Doozie.action.sharelib.for.hive=hive,hcatalog \
    -Dtable=employee\
    -Dcountry=US\
    -Denvironment default \
    -DstartDate 2018-01-19T12:38Z \
    -DendDate 2020-01-17T13:00Z \
    -DscriptPath script \
    -verbose \
    -debug \
    -run


Thank you!

1 REPLY 1

Explorer

Can anyone please help me in this ...

; ;