@Sami Ahmad This hive connection pool configs are varies depending on the setup,unfortunately there are no standard properties that i can share with you. HiveConnectionPool configs:- I use the below connection pool configs Database Connection URL jdbc:hive2://sandbox.hortonworks:2181,sandbox.hortonworks:2181,sandbox.hortonworks:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-binary;principal=hive/_HOST@HDP.LOCAL Hive Configuration Resources /etc/hdp/hive-site.xml,/etc/hdp/core-site.xml,/etc/hdp/hdfs-site.xml Database User nifi Password No value set Max Wait Time 500 millis Max Total Connections 8 Validation query select current_timestamp Please refer to below links to get your jdbc connection url and if your environment is kerberized then you need to mention kerberos principal,keytab in the connection pool service. https://community.hortonworks.com/questions/107945/hive-database-connection-pooling-service.html https://community.hortonworks.com/articles/4103/hiveserver2-jdbc-connection-url-examples.html https://community.hortonworks.com/questions/64609/cannot-create-hive-connection-pool.html ... View more

@Vaibhav Kumar we need to change the name tag in the xml file i.e open the .xml file in any editor like Notepad++ ..etc then <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <template encoding-version="1.2"> <description></description> <groupId>3b738dba-0161-1000-c808-f7d38f21fcab</groupId> <name>176561_jsontoCSV</name> <snippet> edit the name inside the .xml file then you can import same template again. As NiFi keeps the name of the template based on the name tag in the .xml file, if we do rename for the xml file it won't change the name tag filename so NiFi able to see duplicated template names and throws already exists error. ... View more

@Vaibhav Kumar, i think while uploading template you are facing the below issue if so that means you have already uploaded the template and again you are trying to upload the template that having same name again, at this time NiFi throws an already exists error. To make sure what are the available templates in your NiFi instance then go to top right corner Global Menu Click on Templates Option then you can view what are the available Templates in the NiFi instance. For More details please refer to below link https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_user-guide/content/templates.html what exactly happens at the background when we check/un-check relation checkboxes ? Splitjson:- Lets assume your input json record [ { "member_id": 1, "loan_amnt": 5000 },{ "member_id": 2, "loan_amnt": 1000 } ] In split json processor we are having three relationships 1.split --> as we are having a json array with 2 records as shown above, this relation gets the splitted records i.e 2 in our case. 2.original --> as this relation states original i.e what ever the input json that split json processor that will be routed to this relation, in our case json array(i.e same as input record that shown above code snippet) that having 2 records in it. 3.failure --> the FlowFile is not valid JSON or the specified path does not exist), it will be routed to this relationship, mostly used to get notifications if the process failed. So when we feed split,original,failure relations to EvaluateJsonPath processor 1.from split relation as we are having 2 splitted records and we can get member_id,loan_amnt attribute values 1,5000.. from these flowfiles.Replace text processor gets all the values in the csv file. 2.from original relation we are going to have our input json array message to EvaluateJsonPath processor for this flowfile we are not able to get the attribute values for member_id,loan_amnt because it is inside array, So for this flowfile we will have empty attribute values and when we did replace text processor we are going to have empty value for the attributes. 3.As we are having a valid json so we are not going to have any flowfile routed to failure relation. Let us know if you having any other questions ..!! ... View more

@Saikrishna Tarapareddy, if you are using NiFi 1.2+ then you can try with ConvertRecord processor instead of ConvertJSONToAvro processor, as ConvertRecord processor takes array of json and converts as avro. Keep convertrecord processor configs record reader as Json reader and writer as avro set writer. ... View more

@Bob T You can achieve that by using three successive replace text processors with the below Configs:- Replacetext1:- Search Value \]\" Replacement Value ] Maximum Buffer Size 1 MB Replacement Strategy Regex Replace Evaluation Mode Entire Text Replacetext2:- Search Value \"\[ Replacement Value [ Replacement Strategy Regex Replace Evaluation Mode Entire text ReplaceText3:- in this processor we are trying to replace \(backslash) with empty string. Search Value \ Replacement Value Empty string set Replacement Strategy Regex Replace Evaluation Mode Entire text The output from the last replacetext processor would be { "page_id": "page_bb51c576-a885-482f-bf19-648f0b654745", "group_id": "MFGRPPA000054", "provider": "{"addresses":["1405 Shady Avenue, Pittsburgh, PA"],"education":[{"name":"TEXAS A&M UNIVERSITY COLLEGE OF MEDICINE","graduation_date":""}],"last_name":"Oandasan","affiliations":"","middle_name":"","suffix":"","certifications":["ADULT PSYCHIATRY, CHILD & ADOLESCENT PSYCHIATRY – AMERICAN BOARD OF PSYCHIATRY AND NEUROLOGY"],"degrees":["MD"],"internships":[],"fellowships":[],"phone_numbers":["4124202400","41242024001405","8007320999"],"specialties":["CHILD & ADOLESCENT PSYCHIATRY"],"residencies":["GENERAL PSYCHIATRY, GEORGETOWN UNIVERSITY"],"name":"Aileen Oandasan","state":"PA","first_name":"Aileen"}", "org_name": "CHILDREN'S INSTITUTE OF PITTSBURGH, THE", "scrape_id": "scrape_42d77b5d-19c1-4760-94bc-1fdf4c3b1d79" } Flow:- If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Vaibhav Kumar I have tried with your input Json data and processor configs,all your configs looks good. i'm able to get the flowfile with "1","1000" from replacetext processor. Can you make sure the feeding connections to the processors are correct based on the below screenshot. If the issue still exists then share your flow screenshot or xml so that we can help you ASAP. ... View more

@Sami Ahmad Sad to hear that 😞 I tried to create the same flow and it works as expected. Here is what i tried.. i have create a table in hive bucketed,partitioned,orc format,transactional. hive ddl:- hive> create table default.tmp(id string,name string) partitioned by(ts timestamp) clustered by (id) into 2 buckets stored as orc TBLPROPERTIES ("transactional"="true"); hive> desc tmp; +--------------------------+-----------------------+-----------------------+--+ | col_name | data_type | comment | +--------------------------+-----------------------+-----------------------+--+ | id | string | | | name | string | | | ts | timestamp | | | | NULL | NULL | | # Partition Information | NULL | NULL | | # col_name | data_type | comment | | | NULL | NULL | | ts | timestamp | | +--------------------------+-----------------------+-----------------------+--+ i have created the tmp table with id,name columns as string types and partitioned by ts as timestamp type. Now PutHiveStreaming Processor Configs:- select data from tmp table now:- hive> select * from tmp; +-----+-------+--------------------------+--+ | id | name | ts | +-----+-------+--------------------------+--+ | 1 | hdf | 2015-01-02 20:23:20.0 | | 1 | hdf | 2015-01-02 20:23:20.0 | | 1 | hdf | 2018-03-12 19:11:51.217 | | 1 | hdf | 2018-03-12 19:11:51.24 | +-----+-------+--------------------------+--+ hive> show partitions tmp; +---------------------------------+--+ | partition | +---------------------------------+--+ | ts=2015-01-02 20%3A23%3A20.0 | | ts=2018-03-12 19%3A11%3A51.217 | | ts=2018-03-12 19%3A11%3A51.24 | +---------------------------------+--+ For testing purpose use select hiveql processor with below configs:- HiveQL Select Query select int(1) id,string('hdf')name,timestamp(current_timestamp)ts Output Format Avro Configs:- Test Flow:- try this flow once and make sure you are able to load using PutHiveStreaming processor into hive.once you are able to load the data into hive then try with your scenario. If issue still exists then share your hive create table statement and 10 records of sample data so that we can recreate and help you out ASAP. In addition if you don't want to create partitions then create table statement would be hive> create table default.tmp(id string,name string,ts string) --with ts as timestamp type also works clustered by (id) into 2 buckets stored as orc TBLPROPERTIES ("transactional"="true"); ... View more