Created 08-03-2016 08:43 AM
Hi all,
I've created an Oozie coordinator with synchronous dataset. The time in the cluster is set to CEST (GMT+2). I'm using flume to collect data and create a directory in HDFS in this format:
/flume/%Y/%m/%d/%H
coordinator.properties:
nameNode=hdfs://vm1.local:8020 jobTracker=vm1.local:8050 queueName=default exampleDir=${nameNode}/user/root/oozie-wait oozie.use.system.libpath = true start=2016-08-03T08:01Z end=2016-08-03T12:06Z workflowAppUri=${exampleDir}/app oozie.coord.application.path=${exampleDir}/app
coordiantor.xml:
<coordinator-app name="every-hour-waitForData" frequency="${coord:hours(1)}" start="${start}" end="${end}" timezone="UTC" xmlns="uri:oozie:coordinator:0.1"> <datasets> <dataset name="ratings" frequency="${coord:hours(1)}" initial-instance="${start}" timezone="Europe/Paris"> <uri-template>hdfs://vm1.local:8020/user/root/flume/${YEAR}/${MONTH}/${DAY}/${HOUR}</uri-template> </dataset> </datasets> <input-events> <data-in name="coordInput1" dataset="ratings"> <instance>${coord:current(0)}</instance> </data-in> </input-events> <action> <workflow> <app-path>${workflowAppUri}</app-path> <configuration> <property> <name>wfInput</name> <value>${coord:dataIn('coordInput1')}</value> </property> <property> <name>jobTracker</name> <value>${jobTracker}</value> </property> <property> <name>nameNode</name> <value>${nameNode}</value> </property> <property> <name>queueName</name> <value>${queueName}</value> </property> </configuration> </workflow> </action> </coordinator-app>
When running this example flume creates the directory
/user/root/flume/2016/08/03/10/
But the coordinator is waiting for /user/root/flume/2016/08/03/08
Does any one knows how to make Flume creates the directory in UTC or the coordinator reads the correct directory .
Thanks.
Created 08-03-2016 02:49 PM
Okay, I found a workaround, I added:
-Duser.timezone=GMT
which changes the the JVM timezon. The final Flume-ng command will be as following:
flume-ng agent --conf-file spool1.properties --name agent1 --conf $FLUME_HOME/conf -Duser.timezone=GMT
The needed directory for the oozie coordiantor is now being created.
Created 08-03-2016 02:49 PM
Okay, I found a workaround, I added:
-Duser.timezone=GMT
which changes the the JVM timezon. The final Flume-ng command will be as following:
flume-ng agent --conf-file spool1.properties --name agent1 --conf $FLUME_HOME/conf -Duser.timezone=GMT
The needed directory for the oozie coordiantor is now being created.