Member since
07-08-2016
260
Posts
44
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1969 | 05-02-2018 06:03 PM | |
3929 | 10-18-2017 04:02 PM | |
1225 | 08-25-2017 08:59 PM | |
1700 | 07-21-2017 08:13 PM | |
7037 | 04-06-2017 09:54 PM |
02-28-2018
07:48 PM
Hi, Our current nifi jvm settings are java.arg.2=-Xmx16g java.arg.3=-Xms16g i need to read a huge JSON file 22GB , mainly to replace white spaces from it. i am planning to use the list-->fetch-->splittext-->replacetext-->mergecontent approach which i used earlier for similar use cases. but since the file now is bigger than the JVM , i am thikning i will get outofmemory errors since NiFi needs to read the file before it splits it. am i correct.? i can change the jvm settings to use 32 or 48 gb , but just want to get expert opinion on this.?? Regards, Sai
... View more
Labels:
- Labels:
-
Apache NiFi
12-13-2017
09:39 PM
Hi, i am running a load process where it has to go thru 1 TB files and only extract 100 MB and loads to HDFS. but my content repo is 1 TB and getting error on it. i thought it would start deleting the files once it reaches 50% , which its not doing. how do i cleanup all the processed files?? nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% can i change the archive.enabled=false and restart nifi will clear up the space.?? Regards, Sai
... View more
Labels:
- Labels:
-
Apache NiFi
12-05-2017
05:00 PM
Hi, i am trying to connect to an AWS S3 bucket which i was given access to. I got uid,pwd and a key from the owners. But in ListS3 processor i dont see those fields. i see Access Key and Secret Key properties. Is Access key and password are same ? We dont have to put in a user name.? also is Region a must.? do i need to specify just the bucket name for ex vendor-company-bucket or do i have to fully specify the URL like https://vendor-company-bucket.s3-us-west-2.amazonaws.com? Regards, Sai
... View more
Labels:
- Labels:
-
Apache NiFi
11-29-2017
09:10 PM
@Matt Andruff , that's true , I already came up with a process. i did that using a couple of tables , one to write all successful files with names and dates and another for missing files and dates. i populate these on the file insertion NiFi flow. i join those to find if the missing files ever come back and landed success table. Thanks anyway for your time on this. Regards, Sai
... View more
11-29-2017
07:39 PM
@Matt Andruff both will not work. That will compare partitions. i guess there is no way to find what i am trying to do without writing a script. I create a partition when a file arrives and on some days if files didn't come thru those partitions wont exits. at the end of the month I want to find out on which dates I didn't get the files (so partitions wont exist) , I have to manually go and check HDFS or check show partitions command to findout missing. I was checking to see if that can be found from hive partitions.. Regards, Sai
... View more
11-29-2017
06:54 PM
@Matt Andruff sorry , if I confused both of you. But I am not looking a way to find missing partitions when HDFS folders exists. I am looking to find missing HDFS folders (which otherwise are Hive Partitions). I am trying to see if I can find out from a Hive command. Regards, Sai
... View more
11-22-2017
02:30 PM
@rtrivedi I was asking about finding missing HDFS Directories.?? If everything goes well I should have 7 HDFS folders and Hive Partitions for a week if I partition by day. at the end of the week I want to run a command\process to check if I got all the folders\partitions and if any missing...this can be a week or month etc.,
... View more
11-21-2017
10:08 PM
Hi, I have a Hive Table partitioned by process_dt . so if my data ingestion process is creating a partition per day. if i want to findout all the missing partitions at the end of the week or month etc..how can i find.? is there a SQL or command to find that.? Regards, Sai
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
11-16-2017
04:31 PM
@Matt Burgess Nevermind , I was able to do this using AvroSchemaRegistry. Thank you. Regards, Sai
... View more
11-16-2017
03:16 PM
@Matt Burgess, do we need to have Schema Registry(SR) to use Schemas or can we do this without SR.? Regards, Sai
... View more