@Vivek Singh Does the answer helped, then please Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Yassine Specify type as BigInt which is Equivalent to long type,hive don't have long datatype. hive> alter table table change col col bigint; for more reference https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-IntegralTypes(TINYINT,SMALLINT,INT/INTEGER,BIGINT) ... View more

@siva vulli Does the answer helped to resolve your issue, Click on Accept button below to accept the answer,to close the thread and it would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Mark McGowan Does the answer helped to resolve your issue, Click on Accept button below to accept the answer,to close the thread and it would be great help to Community users to find solution quickly for these kind of issues. ... View more

<br> @Omer Shahzad By using PutHiveQL processor we can execute Hive DDL/DML statements, As you need to copy data between hive tables so we need to prepare hive statements and execute them using PutHiveQL processor. Example:- Let's take i need to copy 5 source tables in hive to target database in hive for this case keep all your insert/insert overwrite statements in Generate Flowfile processor. GenerateFlowfile Processor configs:- im having 5 target tables which are having different partiton columns dt,yr,ts and source data table also in hive having same partition columns, in the below statements i'm selecting a specific partition and inserting data into final table for the specific partition. Hive DML staements as follows:- insert into target_db_name.target_table_name1 partition(dt) select * from source_db_name.tablename1 where dt='partition_value' insert into target_db_name.target_table_name2 partition(yr) select * from source_db_name.tablename2 where yr='partition_value' insert into target_db_name.target_table_name3 partition(dt) select * from source_db_name.tablename3 where dt='partition_value' insert into target_db_name.target_table_name4 partition(ts) select * from source_db_name.tablename4 where ts='partition_value' insert into target_db_name.target_table_name5 partition(dt) select * from source_db_name.tablename5 where dt='partition_value' Schedule this processor either using cron (or) timer driven to run daily (or) based on your use case. If you want to change the partition column value based on time then use expression language to change the value dynamically based on time. example:- dt='${now():format("yyyy-MM-dd")}' output dt='2018-04-01' Refer to below this link for more date manipulations in NiFi. Then use Split Text processor:- As in the above generate flowfile processor we are having all the statements in one flowfile, now by using split text processor split the contents of flowfile by each line. now we are splitting the contents of flowfile into each line to new flowfile as we are having 5 insert statements so we will get 5 flowfile each having insert staetments in them. PutHiveQl Processor:- Put hive ql processor expects the content of an incoming FlowFile is expected to be the HiveQL command to execute. We are having insert statements as our flowfile content so use the Splits relation from split text processor relation and connect to Puthive Ql processor. each statement will be executed by using put hive ql processor. By using this method if you need to add new table to copy data from source to target then just add the appropriate insert staement in generate flowfile processor and we are re using the same flow for copy data from source to final. in addition if you want to distribute the flowfiles then use Remote processor group after Split text processor for distribution. Following link explains how to configure remote processor group https://community.hortonworks.com/articles/16461/nifi-understanding-how-to-use-process-groups-and-r.html Flow:- ... View more

@Stefan Constantin I tried with your sample record and i'm able to get the csv data as i have mentioned in the comment above and also i have attached my xml you can save and upload to your NiFi instance and compare both find where are you missing some configs. Could you share your Convert Record processors Record Reader, Record Writer,Avro Schema Registry Config screenshots. ... View more

@Stefan Constantin If you want header with lower and camel case names then in csvsetwriter controller change the below property to false. Include Header Line false now we are not going to get the header line included in the output flowfile, then use Replace text processor to add our own header to the file Replace text configs:- Search Value (?s)(^.*$) Replacement Value inv_idn,inv_number,usr_mdf,invIssDat,invCliCod,invCliNam,invCli_RegCountry,invoiceClass,corCliCod,ofoChiNmb,bsnCtgDsc,bsnCtgCod,bsnCtgDspCod,invTyp,invStu,invOrgCod,cryCod,amnWthVat,amnWthVatInEur,mediaFeesInEur,kcFeesInEur Character Set UTF-8 Maximum Buffer Size 1 MB //needs to change the value according to the file size that we are getting after convertrecord processor Replacement Strategy Prepend //we are prepend the the content with the above header line Evaluation Mode Entire text Input content:- 247048764,181120060,15/03/2018 08:34:00 by LDL,15/03/2018,FUNDQ,FUNDQUEST,FR,CUS - assujetti in EU outside Lx,BNPAMFR,20173748,Fund Data Management,LIS,FDM,Credit Note,Validated,LU,EUR,"-7,543.23","-7,543.23",0,-7543.23 Output Content:- inv_idn,inv_number,usr_mdf,invIssDat,invCliCod,invCliNam,invCli_RegCountry,invoiceClass,corCliCod,ofoChiNmb,bsnCtgDsc,bsnCtgCod,bsnCtgDspCod,invTyp,invStu,invOrgCod,cryCod,amnWthVat,amnWthVatInEur,mediaFeesInEur,kcFeesInEur 247048764,181120060,15/03/2018 08:34:00 by LDL,15/03/2018,FUNDQ,FUNDQUEST,FR,CUS - assujetti in EU outside Lx,BNPAMFR,20173748,Fund Data Management,LIS,FDM,Credit Note,Validated,LU,EUR,"-7,543.23","-7,543.23",0,-7543.23 ... View more