Member since
12-23-2016
38
Posts
2
Kudos Received
0
Solutions
09-28-2017
12:18 PM
Hi Team, I am trying to create a database in Hive using spark sql in spark 2.0 version & encounter the below permission issue, command - spark.sql("create database test")
************************************************************************************************************************
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: java.security.AccessControlException: Permission denied:
************************************************************************************************************************ However when I try to do the same in Spark 1.6 version, I am able to do it without any issues.
command - sqlContext.sql("create database test") Do I need to set some configuration to execute from Spark 2.0? Thanks in Advance!
... View more
Labels:
- Labels:
-
Apache Spark
09-16-2017
04:46 PM
@Yash Thanks!
... View more
09-16-2017
01:06 PM
Hi, I am trying to upload my template created in NiFi version 1.1(HDF 2.0) to NiFi version 1.2(HDF 3.0). However I am facing some versioning error/conflicts. Could you please help me overcome this issue? Thanks, R.Rohit
... View more
Labels:
- Labels:
-
Apache NiFi
09-16-2017
07:08 AM
Hi, We are currently running NiFi as a single instance & we are planning to move to a clustered setup(3 node cluster).
Please consider the below sample flow,
ListFile -> updateAtrribute -> routeOnAttribute -> ExecuteStreamCommand(Executes a shell script) -> fetchFile -> updateAtrribute -> fetchFile -> putFile Since we are going to run in cluster setup, we need to use Remote Processor groups to balance the load. We going to place the RPG after ListFile processor,
ListFile(on Primary Node) -> RPG
Input port -> updateAtrribute -> routeOnAttribute -> ExecuteStreamCommand(Executes a shell script) -> fetchFile -> updateAtrribute -> fetchFile -> putFile My question is, if I want my ExecuteStreamCommand(which triggers a shell script) to execute only on the primary node & rest of the processors in all the nodes, can I go ahead and change the settings of processor to run 'On Primary Node'? Will it have any impact on the flow? Thanks,
R.Rohit
... View more
Labels:
- Labels:
-
Apache NiFi
08-29-2017
01:14 AM
@Wynner Is there a way to overcome it? Any alternate way to parameterize?
... View more
08-28-2017
05:44 PM
@Wynner Attached the sample data. The total length of the record is 40. We are trying to convert the fixed width data to delimited data using regex. sample-data.txt Thanks!
... View more
08-28-2017
05:15 PM
@Wynner ReplaceText Configuration - where search_pattern - ^(.{1})(.{2})(.{18})(.{19}) & replace_pattern - $1|$2|$3|$4 Thanks!
... View more
08-28-2017
04:44 PM
@Wynner I am passing them as flow file attributes only. But still I am not able to parse it properly. Could you please check and help me out? Thanks in Advance!
... View more
08-28-2017
02:20 PM
Hi Team, I am trying to parameterize 'Search Value' & 'Replacement Value' properties in ReplaceText processor. However, I am not getting the desired result. search_pattern - ^(.{1})(.{2})(.{18})(.{19}) replace_pattern - $1|$2|$3|$4 If I use the parameters 'search_pattern' & 'replace_pattern'(${search_pattern} & ${replace_pattern}), replace is not happening. However, if I hardcode the patterns, the search & replace is happening as expected. Could you please help me out? Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
08-07-2017
01:30 AM
Thanks @Hellmar Becker
... View more
08-06-2017
09:36 AM
Hi Team, I went through the NiFi documentation to get an idea about Controller Services. But still I am not clear when to go for Controller services. Could you please help me with answering the below queries? 1. What are Controller Services & its characteristics/advantages? Why do we go for it? 2. In what scenarioes we can/should use it? Thanks & Regards, R.Rohit
... View more
Labels:
- Labels:
-
Apache NiFi
07-09-2017
07:45 AM
Hi Team, I have a scenario where I need to read a file with older timestamp, after a file with latest timestamp has been processed by the ListFile processor. Below are the details of what I tried, I am setting the Min & Max age filter in the ListFile processor as below, Min Age – 300 sec [5 minutes]
Max Age – 864000 sec [10 days] Then, I touch a file in the file system with the latest timestamp as shown below, -rw-r--r-- 1 userA userB users 0 Jul 9 00:57 a.txt the file gets picked by the ListFile processor.Then, I touch a file in the file system with older timestamp as shown below, -rw-r--r-- 1 userA userB users 0 Jul 5 00:00 b.txt However this file is not getting picked by the processor.
My understaning was that files whose modified time is between 5 minutes to 10 days old, should get picked. Could you please let me know the actual behaviour of the Min/Max filter? Also could you please let know whether the scenario which I have explained above can be achieved? Thanks & Regards, R.Rohit
... View more
Labels:
- Labels:
-
Apache NiFi
04-21-2017
04:53 AM
Got it! Thanks for your answers @Wynner and @Matt Clarke.
... View more
04-20-2017
01:32 AM
Hi Team, I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow, ListFile -> FetchFile -> CustomProcessor -> PutHDFS The Custom Processor's concurrency is set to 4.
I encounter a single file whose data size is 31 GB. I wanted to know how
the data will get processed in this case? Will back Pressure get applied ?
Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
04-09-2017
12:55 PM
@Wynner I have only the hive-site.xml file mentioned in the 'Hive Configurations Resources' setting. I will check the contents of this file & let you know.
... View more
04-09-2017
12:53 PM
@Matt Clarke We are using NiFi 1.1.0. Actually we have NiFi installed in 2 environments, one in Dev & the other in SIT. The configurations/settings in Dev & SIT are the same. The NiFi installed in SIT is working fine, however the one installed in Dev was also working absolutely fine, however it suddenly started throwing this exception couple of days back.
... View more
04-07-2017
04:07 AM
Hi Team, I am using PutHiveQL to add a partition to the Hive Metastore. However the processor throws java.lang.NoClassDefFoundError:org/apache/http/config/Lookup exception as shown below, Please let me know if more information is required. Any help would be appreciated. Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
04-03-2017
12:22 AM
Thanks Meyer!!!
... View more
04-02-2017
09:32 AM
Hi Team, In my file I have some records as show below, Sample Input- 1|transaction|I|2017-03-28|Current 1|master|B|2017-03-28|Current 1|master|A|2017-03-28|Current Required Output- 1|transaction|I|2017-03-28|Current 1|master|A|2017-03-28|Current While processing the file using NiFi, I want to consider only records having "I" & "A" in the third position. I am trying to use "RouteText" processor using regular expression to filter the records. However I am not successful in getting the correct expression for filtering the records.
Could you please help me out? Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
04-02-2017
09:17 AM
Hi Bala, I am using ReplaceText processor to replace "|" delimiter to CTRL-A delimiter. However the pipe delimiters are not getting replaced by proper CTRL-A delimiter . Below is my sample input & required output, Input- 1|transaction|I|2017-03-28|Current
1|master|B|2017-03-28|Current
1|master|A|2017-03-28|Current Output- 1^Atransaction^AI^A2017-03-28^ACurrent
1^Amaster^AB^A2017-03-28^ACurrent
1^Amaster^AA^A2017-03-28^ACurrent
Note - If the CTRL-A delimited file is viewed from Linux system using Vi editor, CTRL-A characters will be displayed as ^A. Thanks!
... View more
04-02-2017
03:43 AM
Hi Team, I have a file in local file system(LINUX) with fields delimited with "|"(pipe). I am trying to read the file and replace the "|" delimiter with CTRL-A delimiter. However I am not able to do it. I tried using a ReplaceText processor with search pattern as "|" & replace pattern as "\u0001". Could you please help in resolving this query? Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
02-22-2017
04:49 PM
1 Kudo
Team, I am getting the below files in a directory in the same order,
companies.dat
data.xml
transaction.dat
accounts.dat
bankdetails.dat I use a List File processor to read the files from a directory. When we encounter ".xml" file, it should trigger a shell script & rest of the files (transaction.dat, accounts.dat & bankdetails.dat) should not be processed until script completes its execution. Once the script is complete, the rest of flow files should be processed.
If .dat file then process the file(Path A).
If .xml file, then trigger the script(Path B) & then process the rest of the .dat files(Path A). Could you please help us with an approach? Please let me know if more info is required. Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
02-13-2017
02:25 PM
Hi, When I create a template in one Nifi instance & upload the same template in another instance, sometimes the flow gets mis-aligned. Is it a common thing or I am making a mistake somewhere? Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
01-23-2017
01:11 AM
Hi @Sebastian Carroll Thanks for your reply! I understand NiFi doesn't really have a concept of a 'job' or a 'batch' & its like stream processing.
I am using the scheduler to start a NiFi process group. If incase any proessor fails due to some reason and my flow gets hanged(does not proceed further & even records/file do not move to the failure relationship), is there a failure code which NiFi return, which we can capture & alert the users?
... View more
01-22-2017
02:34 PM
Hi, We are planning to trigger NiFi using a scheduler(Control-M). I would like to know if incase of any failure, does NiFi return any return code(failure code/message), so that the scheduler can catch it & create an alert(ticket) ? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
01-09-2017
04:27 PM
Thanks Matt for the response!! I have updated my requirement above.Could you please take a look at it & help me out?
... View more
01-09-2017
04:26 PM
Hi Matt, Below is my requirement on a high level, As I mentioned in my previous post, the data file contains the actual data & control file has the details of the data file. If there are 100 records in the data file, the control has these details(name of the data file,file size and record count). My requirement is the read the control file, get the record count(store it in an attribute). Then read the data file & validate it against the record count derived from the control file to check whether the record count matches. If the record count matches, I will process the data file, if not I have to move both control file & data file to reject path in HDFS. While I read the files, I fetch the control file(CTRL_ABC.txt) first & then from the control file name I fetch the data file(ABC.txt),store the filenames in 2 separate attributes.To read the contents of the files I use ListFile Processor & 2 FetchFile processors(to read the files separately).do the validation. If the validation fails, I need to move both the files to reject path. Is there a way to move these files at once or do I need to move the files separately using 2 different FetchFile & PutHDFS processors? Sample Flow- ListFile->UpdateAttribute(to get data/ctrl file names)->FetchFile(read ctrl file)->UpdateAttribute(get record count)->FetchFile(to read data file)->Custom Processor(to do validation)
Thanks!
... View more
01-09-2017
02:14 PM
Hi,
I have a scenario where I get a data file & control file. The data file has the actual data & the control file has the details about the data file, say filename,size etc. Below are the file names, file_name - ABC.txt
ctrl_file_name - CTRL_ABC.txt I have to read the files from local filesystem & write into hdfs.
And say, I have the below flow in NiFi,
List File -> Update attribute(file_name - ABC.txt,ctrl_file_name - CTRL_ABC.txt)
Is there a way to combine these 2 attributes & read it using Fetch file processor & write it at once into HDFS? Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
12-25-2016
03:54 PM
Thanks Hellmar!!!
... View more
12-25-2016
03:54 PM
Thanks Devin!!!
... View more