I have a hdfs directory where files will be created periodically. I need to get notified when there is a file created in its subdirectory so that I can start reading the file from that hdfs location and send the contents to kafka topic. I found this can be done using HDFS iNotify. But there are few disadvantages like instead of a hdfsdirectory, it notifies when the file is created anywhere in the hdfs. It captures all events happening in the hdfs. public class HdfsINotifyExample {
public static void main(String[] args) throws IOException, InterruptedException, MissingEventsException {
long lastReadTxid = 0;
if (args.length > 1) {
lastReadTxid = Long.parseLong(args[1]);
}
System.out.println("lastReadTxid = " + lastReadTxid);
HdfsAdmin admin = new HdfsAdmin(URI.create(args[0]), new Configuration());
DFSInotifyEventInputStream eventStream = admin.getInotifyEventStream(lastReadTxid);
while (true) {
EventBatch batch = eventStream.take();
System.out.println("TxId = " + batch.getTxid());
for (Event event : batch.getEvents()) {
System.out.println("event type = " + event.getEventType());
switch (event.getEventType()) {
case CREATE:
CreateEvent createEvent = (CreateEvent) event;
System.out.println(" path = " + createEvent.getPath());
System.out.println(" owner = " + createEvent.getOwnerName());
System.out.println(" ctime = " + createEvent.getCtime());
break;
default:
break;
}
}
}
}
} Instead of monitoring all events type , is there any better way to look only for create file event in a particular hdfs directory. Any shell script or through any other mechanisms?
... View more
