@Artem Ervits sounds good... how about also using sequence file format before merging.. or i beleive storing in zip or bzip format would be the effective storage i guess. so we can store any file formats not just jpeg or png's effectively without wasting block size space or disk space in zip or bzip format.. Am i correct. I heard MAPR has some other file system which is best used for storing small files compared to hortonworks. How about zip files which are larger than 64mb are these splitted in hdfs or in Nifi we write a processor so that the zip files wont exceed 64mb??? ... View more

thanks @Aldrin Piri. ExecuteProcess processor fetches data. But i ll look for creating a custom processor. Thanks for the help. ... View more

@Aldrin Piri Interesting i thought we would have an out of the box processor as In internet of things where nifi is best suited i think we need an out of the box processor. SO if we are trying to stream well logs or sensor data using nifi which processors do we use? We have GetTwitter to stream tweets which is realtime.. Anyway thanks for the tips for using alternate processors for proof of concept. Thank you. ... View more

@jpercivall. thanks for the reply. So i beleive the ExtractImageMetadata processor gets the png and jpeg image files metadata rather than system level metadata. SO if GetFile is getting most of the attributes then why we need ExtractMetadataProcessor please? Just wondering it would be good if you can share the GetFile and ExtractImageMetadata nifi template which you said you have implemented please. Thank you ... View more

@Aldrin PiriI am trying to use InvokeHttp processor for the follwoing datasift https url. SSlcontext is now been setup. In the properties of InvokeHttp i gave https url and gave basic username and password which i was using for curl. I gave Auth in basic username and password. It starts up but doesnt pull any data. How do we stop it apart from restarting nifi. Also once the connection is made i want to keep it alive and doesn want it to get time out like in gethttp..?? . https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958 -H 'Auth: suri:dba37513923299cbb5bcbff766bacd3d ... View more

thanks @Aldrin Piri it was really helpful. My ssl issue is not sorted. But just wondering when i was streaming the data usinf puthdfs it errors saying json file already exists but i just created a new json file before i start nifi and then streaming data to that file in hdfs. Do i have write any expression language to say if the file is of certain bytes then create a new file or whats the best way forward? thank you ... View more

For the datasift i have a curl https url i.e. https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958 -H 'Auth: suri:dba37513923299cbb5bcbff766bacd3d'. when i do curl command it works but when i use the same url in Gethttp it throws ssl error and invokehttp processor wont fetch anything.. any ideas? ... View more