Created 08-03-2016 08:43 AM
Hi all,
I've created an Oozie coordinator with synchronous dataset. The time in the cluster is set to CEST (GMT+2). I'm using flume to collect data and create a directory in HDFS in this format:
/flume/%Y/%m/%d/%H
coordinator.properties:
nameNode=hdfs://vm1.local:8020
jobTracker=vm1.local:8050
queueName=default
exampleDir=${nameNode}/user/root/oozie-wait
oozie.use.system.libpath = true
start=2016-08-03T08:01Z
end=2016-08-03T12:06Z
workflowAppUri=${exampleDir}/app
oozie.coord.application.path=${exampleDir}/appcoordiantor.xml:
<coordinator-app name="every-hour-waitForData" frequency="${coord:hours(1)}" start="${start}" end="${end}" timezone="UTC"
xmlns="uri:oozie:coordinator:0.1">
<datasets>
<dataset name="ratings" frequency="${coord:hours(1)}" initial-instance="${start}" timezone="Europe/Paris">
<uri-template>hdfs://vm1.local:8020/user/root/flume/${YEAR}/${MONTH}/${DAY}/${HOUR}</uri-template>
</dataset>
</datasets>
<input-events>
<data-in name="coordInput1" dataset="ratings">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<action>
<workflow>
<app-path>${workflowAppUri}</app-path>
<configuration>
<property>
<name>wfInput</name>
<value>${coord:dataIn('coordInput1')}</value>
</property>
<property>
<name>jobTracker</name>
<value>${jobTracker}</value>
</property>
<property>
<name>nameNode</name>
<value>${nameNode}</value>
</property>
<property>
<name>queueName</name>
<value>${queueName}</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
When running this example flume creates the directory
/user/root/flume/2016/08/03/10/
But the coordinator is waiting for /user/root/flume/2016/08/03/08
Does any one knows how to make Flume creates the directory in UTC or the coordinator reads the correct directory .
Thanks.
Created 08-03-2016 02:49 PM
Okay, I found a workaround, I added:
-Duser.timezone=GMT
which changes the the JVM timezon. The final Flume-ng command will be as following:
flume-ng agent --conf-file spool1.properties --name agent1 --conf $FLUME_HOME/conf -Duser.timezone=GMT
The needed directory for the oozie coordiantor is now being created.
Created 08-03-2016 02:49 PM
Okay, I found a workaround, I added:
-Duser.timezone=GMT
which changes the the JVM timezon. The final Flume-ng command will be as following:
flume-ng agent --conf-file spool1.properties --name agent1 --conf $FLUME_HOME/conf -Duser.timezone=GMT
The needed directory for the oozie coordiantor is now being created.