Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 08-03-2016 08:43 AM
Hi all,
I've created an Oozie coordinator with synchronous dataset. The time in the cluster is set to CEST (GMT+2). I'm using flume to collect data and create a directory in HDFS in this format:
/flume/%Y/%m/%d/%H
coordinator.properties:
nameNode=hdfs://vm1.local:8020
jobTracker=vm1.local:8050
queueName=default
exampleDir=${nameNode}/user/root/oozie-wait
oozie.use.system.libpath = true
start=2016-08-03T08:01Z
end=2016-08-03T12:06Z
workflowAppUri=${exampleDir}/app
oozie.coord.application.path=${exampleDir}/appcoordiantor.xml:
<coordinator-app name="every-hour-waitForData" frequency="${coord:hours(1)}" start="${start}" end="${end}" timezone="UTC"
xmlns="uri:oozie:coordinator:0.1">
<datasets>
<dataset name="ratings" frequency="${coord:hours(1)}" initial-instance="${start}" timezone="Europe/Paris">
<uri-template>hdfs://vm1.local:8020/user/root/flume/${YEAR}/${MONTH}/${DAY}/${HOUR}</uri-template>
</dataset>
</datasets>
<input-events>
<data-in name="coordInput1" dataset="ratings">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<action>
<workflow>
<app-path>${workflowAppUri}</app-path>
<configuration>
<property>
<name>wfInput</name>
<value>${coord:dataIn('coordInput1')}</value>
</property>
<property>
<name>jobTracker</name>
<value>${jobTracker}</value>
</property>
<property>
<name>nameNode</name>
<value>${nameNode}</value>
</property>
<property>
<name>queueName</name>
<value>${queueName}</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
When running this example flume creates the directory
/user/root/flume/2016/08/03/10/
But the coordinator is waiting for /user/root/flume/2016/08/03/08
Does any one knows how to make Flume creates the directory in UTC or the coordinator reads the correct directory .
Thanks.
Created 08-03-2016 02:49 PM
Okay, I found a workaround, I added:
-Duser.timezone=GMT
which changes the the JVM timezon. The final Flume-ng command will be as following:
flume-ng agent --conf-file spool1.properties --name agent1 --conf $FLUME_HOME/conf -Duser.timezone=GMT
The needed directory for the oozie coordiantor is now being created.
Created 08-03-2016 02:49 PM
Okay, I found a workaround, I added:
-Duser.timezone=GMT
which changes the the JVM timezon. The final Flume-ng command will be as following:
flume-ng agent --conf-file spool1.properties --name agent1 --conf $FLUME_HOME/conf -Duser.timezone=GMT
The needed directory for the oozie coordiantor is now being created.