About saikrishna_tara

saikrishna_tara · ‎03-09-2017

@Matt Clarke i will try that , can you edit your post and remove our server name..i had it by mistake in my original post and removed it.

saikrishna_tara · ‎03-09-2017

Hi, Since my "nifi" service account kerberos ticket is expiring each day (24hrs), my putHDFS process is failing and i had to start and stop the process as i mentioned in another post. to fix that issue i am planning to regenerate the ticket once in every 12 hours , so that it will never expire. i am planning to use ExecuteScript or ExecuteStreamCommand for this and i am getting errors. as this is my first time using these processes i feel like i am doing something wrong here and asking for your help. Here are the different ways i tried with..please let me know if this can be done. ExecuteProcess: only command is set kinit command like this kinit -k -t /etc/security/keytabs/nifi.keytab nifi/server@domain.COM 20:26:58 UTC ERROR aec3331b-015a-1000-c1e0-4a71379a015b ExecuteProcess[id=aec3331b-015a-1000-c1e0-4a71379a015b] Failed to create process due to java.io.IOException: Cannot run program "kinit -k -t /etc/security/keytabs/nifi.keytab nifi/server@domain.COM": error=2, No such file or directory: java.io.IOException: Cannot run program "kinit -k -t /etc/security/keytabs/nifi.keytab nifi/server@domain.COM": error=2, No such file or directory ExecuteStreamCommand: only command path is set kinit command like this kinit -k -t /etc/security/keytabs/nifi.keytab nifi/server@domain.COM 20:29:24 UTC ERROR aed90433-015a-1000-527f-016723eb1b0c ExecuteStreamCommand[id=aed90433-015a-1000-527f-016723eb1b0c] ExecuteStreamCommand[id=aed90433-015a-1000-527f-016723eb1b0c] failed to process due to org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "kinit -k -t /etc/security/keytabs/nifi.keytab nifi/server@domain.COM": error=2, No such file or directory; rolling back session: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "kinit -k -t /etc/security/keytabs/nifi.keytab nifi/server@domain.COM": error=2, No such file or directory

saikrishna_tara · ‎01-25-2017

Nevermind... PutHiveQL fixed it..but i need to use a ReplaceText to produce a file with the ALTER TABLE command.

saikrishna_tara · ‎01-25-2017

Hi, I created an external table in Hive with partitions pointing to a HDFS location. I am using NiFi to load csv files into that location by using putHDFS. and then i am using SelectHiveQL to execute the ALTER TABLE command. It is successfully able to load the partitions as i can query them from Hive, but its routing to failures as the ALTER TABLE is not producing any output. Here is the error.. SelectHiveQL[id=7c2b4a39-f772-1ea7-7e5b-48be29a34850] Unable to execute HiveQL select query ALTER TABLE hdf_moat.tbl_moat_display_sai ADD PARTITION (MOATDATE=20150412) LOCATION '/user/putarapasa/MOAT_Daily/20150412/' for StandardFlowFileRecord[uuid=9781e831-a494-4512-8121-7bed28908763,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1485365330956-237, container=default, section=237], offset=0, length=50994078],offset=0,name=GroupM_Nestle_Daily_20150412.csv,size=50994078] due to org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: The query did not generate a result set!; routing to failure: org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: The query did not generate a result set! how can we handle this.?? Regards, Sai

saikrishna_tara · ‎01-11-2017

Hi @aawasthi, suppose i have an external table "test_display" which points to '/user/test/display' which already have 30 files one for each day. and i can query the table. when i created the table i did not specify any partition. now like you mentioned above i tried to execute the command below ALTER TABLE test_display ADD PARTITION (moat_date='2017-01-05') LOCATION '/user/test/display' Error occurred executing hive query: Error while compiling statement: FAILED: ValidationFailureSemanticException table is not partitioned but partition spec exists: {moat_date=2017-01-05} do i have to manually create the folder 2017-01-05 under '/user/test/display'

saikrishna_tara · ‎01-11-2017

Hi, i created an external table in HIVE with 150 columns. i have a .csv file for each day , and eventually i will have to load data for 4 years. i just loaded one month worth of files which turned into 2mill rows. i now like to partition the table by date (which first column in the table and file). 1.how can i partion on existing table and 2. can i do it dynamically when i load other files. Regards,Sai

saikrishna_tara · ‎12-20-2016

@D'Andre McDonald Here is what I did , which is basically what is described above..i have 80 files going in to 17 folders on a different server.(hoping Nifi user has access to the remote folder). list and fetch will be your GetFile and then in my first UpdateAttribute I am extracting the filename (since my filename has other info which I am not interested in)..my destination folder on the remote system depends on my file name. if you can route your files based on your incoming file name as it is you can ignore this step . incoming file name to first UpdateAttribute is export-20150218000000-20150219000000-FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV Out going file name is FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV my second UpdateAttribute process is where I set the Folder based on file name..here I have used Advanced tab but you can use properties also. basically I am wrote all my rules here..setting the destination Folder based on file names. and finally in my in your put file you can use that property and create the folder dynamically.. Hope this helps... Sai

saikrishna_tara · ‎12-08-2016

Hi, I see in the documentation in the link below , there an option to append https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/ but in my PutHDFS i don't see that option .. i am Version HDF-2.0.0.0-579. any idea why.?

saikrishna_tara · ‎12-07-2016

@kkawamura, Thank you. i will try that. so with above example it zips all the files under source-files into one zip file.? also is there anyway that i can process this at subfolder level.? in my case , under a folder "January" i will have 31 subfolders 01012016 to 01312016 (a folder for each day with 8000 files in each folder). if i point the above command at folder "January" it will try to zip all 31 subfolders into one which may come nearly 80G and it may be difficult for me to transfer to HDFS thru site2site. i was looking at 1 zip for each subfolder. so in this case i will endup with 31 zip files. Thanks again, Sai

saikrishna_tara · ‎12-06-2016

Hi, we have a use case where we need to save the source files as it is. for each day we will have a folder with 1000 files. we will process 1000 files and send required 100 files to HDFS. but we also want to compress and save the whole folder for any future references.(we like to keep the source as it is at a folder level not at file level) is there anyway i can do this using NiFi.? Regards, Sai

Online	Offline
Last Visited	‎10-26-2018 04:07 PM

Member Since	‎07-08-2016 07:25 PM
Last Visited	‎10-26-2018 04:07 PM
Posts	260
Kudos received	44

Cloudera Community

Re: I am using unpack processor in NIFI to unzip a...

Re: Changing NiFi Templates

Re: Hive Connection Pool Kerberos Issue

Re: NiFi user getting Tez exception

Re: Help with Hive Regex extract.

Re: Error Executing KINIT command from ExecuteProc...

Error Executing KINIT command from ExecuteProcess ...

Re: SelectHiveQL routes to failure on The query di...

SelectHiveQL routes to failure on The query did no...

Re: How can we create dynamic partitions in Hive e...

How can we create dynamic partitions in Hive exter...

Re: How can we use a single set of Put/Get Process...

NiFI PutHDFS Append is not available.

Re: Can we read and compress folders using NiFi.?

Can we read and compress folders using NiFi.?