Hi All, I have an use case where I want to find number of occurrences of the word and want to perform an action on it. example: 1. I have multiple flow files coming in 2. I want to extract a word (say, user_name) using extracttext processor 3. count the word 4. if user_name_count =10 5. do replacetext 10 as 1 6. putemail to user_name that user_name count is 10. Can you please let me know which processors can be helpful for the usecase. Suggestions are appreciated. ... View more

Hi All, I have a 10GB file every minute coming to a location (/dir), and there is an external table for that location. The file is as below karlon,n_d_1,26,6234,2019-09-08,1536278400 d'lov,research,20,1001,2019-09-08,1536278400 kris'a,b_x_3,20,4532,2019-09-08,1536278400 external table name: ex_t name department age id date time karlon n_d_1 26 6234 2019-09-08 1536278400 d'lov research 20 1001 2018-09-08 1536278400 I have puthiveql processor in my flow which gets data from external table and inserts in to multiple ORC table. ORC : table_1, table_2, table_3,table_4,table_5, table_6 Every table(orc table) has same columns. name(string),department (string),age (int),id (int),date (string),partition_value (int) The puthiveql processor has multiple insert queries in it. INSERT INTO table_1 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'research' AND time='1536278400'; INSERT INTO table_2 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'n_d_1' AND time='1536278400'; INSERT INTO table_3 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'b_x_3' AND time='1536278400'; INSERT INTO table_4 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'research' AND time='1536278400'; INSERT INTO table_5 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'x_in_1' AND time='1536278400'; INSERT INTO table_6 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'z_e_3' AND time='1536278400'; The above is sent as a flowfile to puthiveql, which is scheduled every minute, as the file arrives every minute. Puthiveql is very slow process the above and the inserts are not happening frequently. Can you please suggest how to improve the performance of the puthiveql, I have increased the concurrent processor but it did not help, some times the flowfiles(which have insert statements) get queued and never execute. Suggestions are highly appreciated. ... View more

Hi All, I have json data in multiple small files (some times only one line in a file). I want to merge all small files in to single large file. I am getting a large file in an unexpected format. ex: file 1: {"code"="1", "color"="green"} {"code"="2", "color"="blue"} {"code"="3", "color"="orange"} file 2: {"code"="4", "color"="yellow"} {"code"="5", "color"="red"} I am getting the below output after using MergeContent {"code"="1", "color"="green"} {"code"="2", "color"="blue"} {"code"="3", "color"="orange"}{"code"="4", "color"="yellow"} {"code"="5", "color"="red"} Expected output {"code"="1", "color"="green"} {"code"="2", "color"="blue"} {"code"="3", "color"="orange"} {"code"="4", "color"="yellow"} {"code"="5", "color"="red"} ... View more

Hi, I am getting a plain json stream with '\n' delimiter through TCP. I am listening to TCP using listenTCP, set batchsize to 10000. My json is with variable values, ex: {"a":"20180831","b":"b"} {"a":"20180831","b":"b","c":"c"} I want to add a partition_value attribute to every line in json stream at once, which should look like The attribute a is always present in json, so I want to use partition_value in a {"a":"20180831","b":"b","partition_value":"20180831"} {"a":"20180831","b":"b","c":"c","partition_value":"20180831"} I have used "UpdateRecord" processor below are the configuration UpdateRecord JsonTreeReader AvroSchemaRegistry AvroRecordSetWriter I used UpdateRecord -> jsontreereader ->avroschemaregistry |_________ -> avrorecordsetwriter Then I have used avrotojson I am getting only one line as output, can you please suggest where it is happening wrong or let me know if there is a better way to do it Thank you {"a":"20180831","b":"b","c":null,"partition_value":"20180831"} ... View more