Hi All, I need help to get the below result. I have two tables table name: match +-----------------------------------+----------------+--+ | hint | remarks | +-----------------------------------+----------------+--+ | 1.1.1.1 | ip | | 123456789 | contact | | http://123123123123123123.some_n | url | +-----------------------------------+----------------+--+ table name : t1 +-------------------------------------------------------------------------------+-------------------+--+ | t1.text | t1.b | +-------------------------------------------------------------------------------+-------------------+--+ | This ip is found 1.1.1.1 and is matched with match | table name match | | This ip is found 1.1.1.2 and is matched with match | table name match | | This contact is found 123456789 and is matched with match | table name match | | This contact is found 123456789123456789 and is matched with match | table name match | | This url is found http://123456789123456789.some_n and is matched with match | table name match | +-------------------------------------------------------------------------------+-------------------+--+ I want to search hint column of match table in text column of t1 table and get complete text column values. so, basically I want to do a query like select t1.text from t1 join match where t1.text contains (any value in match.hint); It will be helpful if this can be done in hive or I can live with pyspark, so pyspark help is also welcome P.S: table t1 is a big table and match is a small table with limite values(say 1500). Thank you ... View more

Hi All, I have an use case where I want to find number of occurrences of the word and want to perform an action on it. example: 1. I have multiple flow files coming in 2. I want to extract a word (say, user_name) using extracttext processor 3. count the word 4. if user_name_count =10 5. do replacetext 10 as 1 6. putemail to user_name that user_name count is 10. Can you please let me know which processors can be helpful for the usecase. Suggestions are appreciated. ... View more

Hi All, I have a 10GB file every minute coming to a location (/dir), and there is an external table for that location. The file is as below karlon,n_d_1,26,6234,2019-09-08,1536278400 d'lov,research,20,1001,2019-09-08,1536278400 kris'a,b_x_3,20,4532,2019-09-08,1536278400 external table name: ex_t name department age id date time karlon n_d_1 26 6234 2019-09-08 1536278400 d'lov research 20 1001 2018-09-08 1536278400 I have puthiveql processor in my flow which gets data from external table and inserts in to multiple ORC table. ORC : table_1, table_2, table_3,table_4,table_5, table_6 Every table(orc table) has same columns. name(string),department (string),age (int),id (int),date (string),partition_value (int) The puthiveql processor has multiple insert queries in it. INSERT INTO table_1 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'research' AND time='1536278400'; INSERT INTO table_2 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'n_d_1' AND time='1536278400'; INSERT INTO table_3 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'b_x_3' AND time='1536278400'; INSERT INTO table_4 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'research' AND time='1536278400'; INSERT INTO table_5 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'x_in_1' AND time='1536278400'; INSERT INTO table_6 PARTITION(partition_value) SELECT name, department, age, id, date, cast(regexp_replace(date,'-','') as int) AS partition_value FROM ex_t WHERE department = 'z_e_3' AND time='1536278400'; The above is sent as a flowfile to puthiveql, which is scheduled every minute, as the file arrives every minute. Puthiveql is very slow process the above and the inserts are not happening frequently. Can you please suggest how to improve the performance of the puthiveql, I have increased the concurrent processor but it did not help, some times the flowfiles(which have insert statements) get queued and never execute. Suggestions are highly appreciated. ... View more

Hi All, I have json data in multiple small files (some times only one line in a file). I want to merge all small files in to single large file. I am getting a large file in an unexpected format. ex: file 1: {"code"="1", "color"="green"} {"code"="2", "color"="blue"} {"code"="3", "color"="orange"} file 2: {"code"="4", "color"="yellow"} {"code"="5", "color"="red"} I am getting the below output after using MergeContent {"code"="1", "color"="green"} {"code"="2", "color"="blue"} {"code"="3", "color"="orange"}{"code"="4", "color"="yellow"} {"code"="5", "color"="red"} Expected output {"code"="1", "color"="green"} {"code"="2", "color"="blue"} {"code"="3", "color"="orange"} {"code"="4", "color"="yellow"} {"code"="5", "color"="red"} ... View more

Hi, I am getting a plain json stream with '\n' delimiter through TCP. I am listening to TCP using listenTCP, set batchsize to 10000. My json is with variable values, ex: {"a":"20180831","b":"b"} {"a":"20180831","b":"b","c":"c"} I want to add a partition_value attribute to every line in json stream at once, which should look like The attribute a is always present in json, so I want to use partition_value in a {"a":"20180831","b":"b","partition_value":"20180831"} {"a":"20180831","b":"b","c":"c","partition_value":"20180831"} I have used "UpdateRecord" processor below are the configuration UpdateRecord JsonTreeReader AvroSchemaRegistry AvroRecordSetWriter I used UpdateRecord -> jsontreereader ->avroschemaregistry |_________ -> avrorecordsetwriter Then I have used avrotojson I am getting only one line as output, can you please suggest where it is happening wrong or let me know if there is a better way to do it Thank you {"a":"20180831","b":"b","c":null,"partition_value":"20180831"} ... View more

Hi, I am recieving data from TCP as a json stream using pyspark. I want to save the files(append files and basically a file is a minute based ex:yyyyMMddHHmm (file), so all messages in one min should go to the corresponding file) and parallelly I want to save the json to orc hive table. I have two questions involved 1. *[path : '/folder/file'] When I receive data in Dstream I flatMap and split("\n") and then repartition(1).saveAsTextfile(path,"json") lines = ssc.socketTextStream("localhost", 9999) flat_map = lines.flatMap(lambda x: x.split("\n")) flat_map.repartition(1).saveAsTextFiles(path,"json") The above saves to the path given, but instead of giving one single file per minute and save to the folder, this makes three folders with a _SUCCESS file and a part_00000 file in every folder, which is not expected. Please help me how to solve this as expected : basically one folder per day and one file per minute under the folder? 2. If I want to save the json to orc hive table.. can I do it from a dstream? or I have to change the dstream to rdd and then perform some processing to save it to orc? as I am new to pyspark please help with the above or with some examples. ... View more

Hi All, I am beginner to spark and wanted to do the below. a port 55500 is trying to send jsons as a stream (ex: {"one":"1","two":"2"}{"three":"3","four":"4"}). I have a orc table in hive with columns given below one, two,three,four,spark_streaming_startingtime,spark_streaming_endingtime,partition_value I want to load the streaming values in to hive orc table. Can you please guide me how to achieve this. Thank you for your support. ... View more