@PJ When you are having huge data in orc table then it will take a while to convert all the results and store them as csv file. Here is what i tried:- foo is an orc table hive#select * from foo; +---------+--+ | foo.id | +---------+--+ | 1 | | 2 | | 3 | | 4 | +---------+--+ bash#hive-e "select * from foo1">>foo1.txt bash# cat foo1.txt +----------+--+ | foo1.id | +----------+--+ | 1 | | 2 | | 3 | | 4 | +----------+--+ When we are having small set of data it will be done very quickly. if the number of records are really big then Ideal way to do this is as follows hive#INSERT OVERWRITE DIRECTORY '<Hdfs-Directory-Path>' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE SELECT * FROM default.foo; or else you can write the data to local directories also just add local hive#INSERT OVERWRITE LOCAL DIRECTORY '<Local-Dir-Path>' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE SELECT * FROM default.foo; Also, note that the insert overwrite directory basically removes all the existing files under the specified folder and then create data files as part files and this may create multiple files and you may want to concatenate them on the client side after it's done exporting. Using this approach means you don't need to worry about the format of the source tables, can select your own delimiters and output formats. ** I would suggest try to avoid saving large file to local directory if possible, Use insert overwrite directory and store the results to HDFS directory ** For more details refer to this link. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingdataintoHiveTablesfromqueries ... View more

@Karl Fredrickson Yes ListSFTP processor only look for new files that got created after the state that processor stored. State value is max time stamp of the file created in that directory. Example:- lets assume that listsftp processor has listed all the files in the directory until 4:10 then processor scheduled to run for every 10 minutes next run is at 4:20. There are new files(test1.txt,test2.txt) got created at 4:11 then these new files(test.txt,test2.txt) will only be listed at 4:20 run(because processor runs for every 10 mins) and then processor updates the state with the 4:11 time stamp.(you can view by right clicking on the processor and click on view state). Although flow files got created at 4:11 still they will be listed only at 4:20 run, because in this run processor checks for the new files that got created after state value. If you configure this processor to less frequent i.e less than 10 minutes then processor will looks for new files that got created more often. ... View more

@Narasimma varman if you want only nifi-app_2017 log files then Input Directory property <directory-path> change the File Filter property to nifi-app_2017.*\.log if you want any logs that having app_2017 in the filename then use .*app_2017.*\.log Keep in mind get file processor is configured KeepsourceFile property is set to be false by default, once it pulls the files then it deletes them on the directory If you don't want to delete the files from the directory then change property KeepsourceFile to true then the processor wont delete the files once they got pulled. if you set Recursive SubDirectories property to true then make sure nifi having access to your input Directory. in addition if you want to do any tailing on the logs then follow the below links to how to configure processors to tail the logs https://community.hortonworks.com/questions/141403/in-nifi-tailing-multiple-directories-with-the-same.html?childToView=141479#comment-141479 https://community.hortonworks.com/questions/141502/how-tailfile-works-with-multiple-files.html?childToView=141517#answer-141517 ... View more

@Arsalan Siddiqi One way to do this is extract all the content of the json file as attributes associated with the flowfile. Then once you make invoke http extract the results of invokehttp to attributes again. By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile. In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb. Flow:- 1.EvaluateJsonProcessor //extract the content to flowfile attribute 2.invokehttp 3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute 4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message. Example:- Input Json file:- { "id": 1, "name": "HCC", "age": 20 } extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now. Evaljsonpath configs:- Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it. use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again. lets consder invokehttpresult will have { "dept": "community"} extract dept as attribute. Right now your flowfile will have id,name,age,dept as attributes to the flowfile. you can do as many invokehttps as you want after each invoke http extract the content as attribute. Now you need to prepare a json message use AttribiutestoJSON processor Change the property attributes list as id,name,age,dept This processor now prepares new json message with all the attributes you listed in the processor. { "id": 1, "name": "HCC", "age": 20,"dept": "community"} Use the json result from Attributestojson processor to store into mongoDB. AttributesToJSON Configs:- ... View more

Hi @Mohan Sure, We can get results as you expected by using EvaluateXquery //we can keep all the required contents as attributes of flowfile. UpdateAttribute //update the contents of attributes that got extracted in evaluatexquery processor. ReplaceText //replace the flowfile content with attributes of flowfile PutHDFS //store files into HDFS EvaluateXquery Configurations:- Change the existing properties 1.Destination to flowfile-attribute 2.Output: Omit XML Declaration to true Add new properties by clicking + sign 1.author //author 2.book //book 3.bookstore //bookstore Input:- <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="myfile.xsl" ?> <bookstore specialty="novel"> <book style="autobiography"> <author> <first-name>Joe</first-name> <last-name>Bob</last-name> <award>Trenton Literary Review Honorable Mention</award> </author> <price>12</price> </book> </bookstore> Output:- As you can see in screenshot all the content are as attributes(book,bookstore,author) to the flowfile. EvaluateXquery Processor configs screenshot:- Update Attribute Processor:- 1.author ${author:replaceAll('<author>([\s\S]+.*)<\/author>','$1')} updating the author attribute input to updateattribute processor:- <author> <first-name>Joe</first-name> <last-name>Bob</last-name> <award>Trenton Literary Review Honorable Mention</award> </author> Output:- <first-name>Joe</first-name> <last-name>Bob</last-name> <award>Trenton Literary Review Honorable Mention</award> 2.book ${book:replaceAll('<book\s(.*)>[\s\S]+<\/author>([\s\S]+)<\/book>','$1$2')} Input:- <book style="autobiography"> <author> <first-name>Joe</first-name> <last-name>Bob</last-name> <award>Trenton Literary Review Honorable Mention</award> </author> <price>12</price> </book> Output:- style="autobiography" <price>12</price> 3.bookstore ${bookstore:replaceAll('.*<bookstore\s(.*?)>[\s\S]+.*','$1')} Input:- <bookstore specialty="novel"> <book style="autobiography"> <author> <first-name>Joe</first-name> <last-name>Bob</last-name> <award>Trenton Literary Review Honorable Mention</award> </author> <price>12</price> </book> </bookstore> Output:- specialty="novel" Configs:- ReplaceText Processor:- Cchange the properties of Replacement Strategy to alwaysreplace and use your attributes bookstore,book,author in this processor and we are going to overwrite the existing contents of flowfile with the new content. add 2 more replacetext processors for book and author attributes. Output:- <first-name>Joe</first-name> <last-name>Bob</last-name> <award>Trenton Literary Review Honorable Mention</award> PutHDFS processor:- Configure the processor and give the directory name where you want to store the data. Flow Screenshot:- For testing purpose i have use generate flowfile processor but in your case generate flowfile processor will be the source processor from where you are getting this xml data. ... View more

@mayki wogno Can you use hive escape character in your select statement Hive escape character:- ` Use the above character to escape from your select statement and try to run select HiveQL again New Select HIVEQL statement:- select country,year,`delta-1` from <table> Select HiveQL Processor config Screenshot:- ... View more