About saikrishna_tara

saikrishna_tara · ‎10-26-2016

Hi , I have a 100 GB file while I will have to Split in to multiple (may be 1000) files depending on the value "Tagname" in the sample data below and write in to HDFS. Tagname,Timestamp,Value,Quality,QualityDetail,PercentGood ABC04.PI_B04_EX01_A_STPDATTRG.F_CV,2/18/2015 1:03:32 AM,627,Good,NonSpecific,100 ABC04.PI_B04_EX01_A_STPDATTRG.F_CV,2/18/2015 1:03:33 AM,628,Good,NonSpecific,100 ABC05.X4_WET_MX_DDR.F_CV,2/18/2015 12:18:00 AM,12,Good,NonSpecific,100 ABC05.X4_WET_MX_DDR.F_CV,2/18/2015 12:18:01 AM,4,Good,NonSpecific,100 ABC04.PI_B04_FDR_A_STPDATTRG.F_CV,2/18/2015 1:04:19 AM,3979,Good,NonSpecific,100 ABC04.PI_B04_FDR_A_STPDATTRG.F_CV,2/18/2015 9:35:23 PM,4018,Good,NonSpecific,100 ABC04.PI_B04_FDR_A_STPDATTRG.F_CV,2/18/2015 9:35:24 PM,4019,Good,NonSpecific,100 In reality the "Tagname" will be continues to be the same(may be 10K+) until its value changes. I need to create one file for each Tag. Do i have to split the file in to smaller files (may be 20 , 5GB files) using SplitFile.? If i do that will it split exactly at the end of lines.? Do I have to read line by line using ExtractText or any better approach.? Can i use ConvertCSVToAvro and then ConvertAVROToJson and then split the Json file by Tag using SplitJson..?? Can i use do i have to change any default NiFi settings for this.? Regards, Sai

saikrishna_tara · ‎10-20-2016

it worked when i changed my ReplaceText to the format below. i think this has to do with the mime-type urlencoded. remember we cannot send special chars like @ i had to send it as %40 This is how the ReplaceText processor looks.. grant_type=password&client_id=6e880286&client_secret=d12f0f6d41cfe81fcfc122e3fc17a833&username=Saikrishna.Tarapareddy%40purina.nestle.com&password=7heStuhuwa also i had mime.type = application/x-www-form-urlencoded in my updateattribute processor. Thanks you all.

saikrishna_tara · ‎10-20-2016

@mkalyanpur our NiFi is running as NiFi service account with NiFi principal user , Hive is running as Hive user. I tried with both and getting the same error. is that what you are asking.? sorry for the confusion.

saikrishna_tara · ‎10-20-2016

@mkalyanpur i tried that too , but getting the same error. i am still getting the warning ,even though my nifi properties file has the settings about kerberos. I am using HIVE user for kerberos credentials , i also tried with mine. both are resulting same error. 14:26:12 UTC ERROR d3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/default;principal=hive/MyDomain.COM;: GSS initiate failed). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/default;principal=hive/MyDomain.COM;: GSS initiate failed) 14:26:12 UTC WARNING d3b68593-0157-1000-2894-db6013bf04f3 HiveConnectionPool[id=d3b68593-0157-1000-2894-db6013bf04f3] Configuration does not have security enabled, Keytab and Principal will be ignored

saikrishna_tara · ‎10-19-2016

Hi @mkalyanpur I have verified that transport mode is set to "binary" my DB Connection URL is jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain and these are all set in NiFi.config file.. nifi.kerberos.krb5.file=/etc/krb5.conf nifi.kerberos.service.principal=nifi/Myserver@MyDomain.COM nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.keytab nifi.kerberos.spnego.principal=HTTP/nifi/Myserver@MyDomain.COM nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/spnego.service.keytab nifi.kerberos.spnego.authentication.expiration=12 hours getting the error below.. 19:47:43 UTC ERROR d3b68593-0157-1000-2894-db6013bf04f3 HiveConnectionPool[id=d3b68593-0157-1000-2894-db6013bf04f3] Error getting Hive connection 19:47:43 UTC ERROR d3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain.COM: GSS initiate failed). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain: GSS initiate failed)

saikrishna_tara · ‎10-19-2016

Hi @Matt Burgess, any idea on what i am doing wrong above..getting 403 forbidden error. Regards, Sai

saikrishna_tara · ‎10-18-2016

I am trying to connect to Hive from NiFi using Kerberos. Looks like there is no way I can do that until the newer version NiFi 1.1.0? I cannot use any other modes as our Hadoop cluster is Kerberised. Can I go back to previous version and use dbcp connection pool.? Will it work with Kerberos.? any other options I have (from NiFi).??

saikrishna_tara · ‎10-18-2016

Can someone help me or point me to an example on how i can get connected to kerberos enabled Hive from NiFi.?

saikrishna_tara · ‎10-18-2016

What should I do to get the fix.?? Also how do I switch to binary end point.??

saikrishna_tara · ‎10-17-2016

Hi, I am trying to connect to create a connection Hive using different formats DB Connection URLs. All of them are giving different errors. first DB Connection URL and Error: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain; error: 21:37:29 UTCERRORd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain: GSS initiate failed). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain: GSS initiate failed) Second DB Connection URL and Error: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice Error 21:47:09 UTCERRORd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL ' jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice'. No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL ' jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice' Third DB Connection and Error jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice;auth=KERBEROS Error 21:49:56 UTCERRORd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] failed to process session due to java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory: java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory 21:49:56 UTCWARNINGd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Processor Administratively Yielded for 1 sec due to processing failure for all of these above i am commonly getting a warning HiveConnectionPool[id=d3b68593-0157-1000-2894-db6013bf04f3] Configuration does not have security enabled, Keytab and Principal will be ignored Thanks in advance, Sai

Online	Offline
Last Visited	‎10-26-2018 04:07 PM

Member Since	‎07-08-2016 07:25 PM
Last Visited	‎10-26-2018 04:07 PM
Posts	260
Kudos received	44

Cloudera Community

Re: I am using unpack processor in NIFI to unzip a...

Re: Changing NiFi Templates

Re: Hive Connection Pool Kerberos Issue

Re: NiFi user getting Tez exception

Re: Help with Hive Regex extract.

What is a good approach for Spilitting 100GB file ...

Re: InvokeHTTP error.

Re: NiFi Hive Connection Pool Error.

Re: NiFi Hive Connection Pool Error.

Re: NiFi Hive Connection Pool Error.

Re: How to add request headers and request body to...

Re: NiFi Hive Connection Pool Error.

Re: NiFi Hive Connection Pool Error.

Re: NiFi Hive Connection Pool Error.

NiFi Hive Connection Pool Error.