Trouble creating databases in Hive using Spark 2.0

rohit_ravishank · ‎09-28-2017

Hi Team, I am trying to create a database in Hive using spark sql in spark 2.0 version & encounter the below permission issue, command - spark.sql("create database test") ************************************************************************************************************************ Caused by: org.apache.hadoop.hive.metastore.api.MetaException: java.security.AccessControlException: Permission denied: ************************************************************************************************************************ However when I try to do the same in Spark 1.6 version, I am able to do it without any issues. command - sqlContext.sql("create database test") Do I need to set some configuration to execute from Spark 2.0? Thanks in Advance!

rohit_ravishank · ‎09-16-2017

@Yash Thanks!

rohit_ravishank · ‎09-16-2017

Hi, I am trying to upload my template created in NiFi version 1.1(HDF 2.0) to NiFi version 1.2(HDF 3.0). However I am facing some versioning error/conflicts. Could you please help me overcome this issue? Thanks, R.Rohit

rohit_ravishank · ‎09-16-2017

Hi, We are currently running NiFi as a single instance & we are planning to move to a clustered setup(3 node cluster). Please consider the below sample flow, ListFile -> updateAtrribute -> routeOnAttribute -> ExecuteStreamCommand(Executes a shell script) -> fetchFile -> updateAtrribute -> fetchFile -> putFile Since we are going to run in cluster setup, we need to use Remote Processor groups to balance the load. We going to place the RPG after ListFile processor, ListFile(on Primary Node) -> RPG Input port -> updateAtrribute -> routeOnAttribute -> ExecuteStreamCommand(Executes a shell script) -> fetchFile -> updateAtrribute -> fetchFile -> putFile My question is, if I want my ExecuteStreamCommand(which triggers a shell script) to execute only on the primary node & rest of the processors in all the nodes, can I go ahead and change the settings of processor to run 'On Primary Node'? Will it have any impact on the flow? Thanks, R.Rohit

rohit_ravishank · ‎08-29-2017

@Wynner Is there a way to overcome it? Any alternate way to parameterize?

rohit_ravishank · ‎08-28-2017

@Wynner Attached the sample data. The total length of the record is 40. We are trying to convert the fixed width data to delimited data using regex. sample-data.txt Thanks!

rohit_ravishank · ‎08-28-2017

@Wynner ReplaceText Configuration - where search_pattern - ^(.{1})(.{2})(.{18})(.{19}) & replace_pattern - $1|$2|$3|$4 Thanks!

rohit_ravishank · ‎08-28-2017

@Wynner I am passing them as flow file attributes only. But still I am not able to parse it properly. Could you please check and help me out? Thanks in Advance!

rohit_ravishank · ‎08-28-2017

Hi Team, I am trying to parameterize 'Search Value' & 'Replacement Value' properties in ReplaceText processor. However, I am not getting the desired result. search_pattern - ^(.{1})(.{2})(.{18})(.{19}) replace_pattern - $1|$2|$3|$4 If I use the parameters 'search_pattern' & 'replace_pattern'(${search_pattern} & ${replace_pattern}), replace is not happening. However, if I hardcode the patterns, the search & replace is happening as expected. Could you please help me out? Thanks!

rohit_ravishank · ‎08-07-2017

Thanks @Hellmar Becker

rohit_ravishank · ‎08-06-2017

Hi Team, I went through the NiFi documentation to get an idea about Controller Services. But still I am not clear when to go for Controller services. Could you please help me with answering the below queries? 1. What are Controller Services & its characteristics/advantages? Why do we go for it? 2. In what scenarioes we can/should use it? Thanks & Regards, R.Rohit

rohit_ravishank · ‎07-09-2017

Hi Team, I have a scenario where I need to read a file with older timestamp, after a file with latest timestamp has been processed by the ListFile processor. Below are the details of what I tried, I am setting the Min & Max age filter in the ListFile processor as below, Min Age – 300 sec [5 minutes] Max Age – 864000 sec [10 days] Then, I touch a file in the file system with the latest timestamp as shown below, -rw-r--r-- 1 userA userB users 0 Jul 9 00:57 a.txt the file gets picked by the ListFile processor.Then, I touch a file in the file system with older timestamp as shown below, -rw-r--r-- 1 userA userB users 0 Jul 5 00:00 b.txt However this file is not getting picked by the processor. My understaning was that files whose modified time is between 5 minutes to 10 days old, should get picked. Could you please let me know the actual behaviour of the Min/Max filter? Also could you please let know whether the scenario which I have explained above can be achieved? Thanks & Regards, R.Rohit

rohit_ravishank · ‎04-21-2017

Got it! Thanks for your answers @Wynner and @Matt Clarke.

rohit_ravishank · ‎04-20-2017

Hi Team, I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow, ListFile -> FetchFile -> CustomProcessor -> PutHDFS The Custom Processor's concurrency is set to 4. I encounter a single file whose data size is 31 GB. I wanted to know how the data will get processed in this case? Will back Pressure get applied ? Thanks!

rohit_ravishank · ‎04-09-2017

@Wynner I have only the hive-site.xml file mentioned in the 'Hive Configurations Resources' setting. I will check the contents of this file & let you know.

rohit_ravishank · ‎04-09-2017

@Matt Clarke We are using NiFi 1.1.0. Actually we have NiFi installed in 2 environments, one in Dev & the other in SIT. The configurations/settings in Dev & SIT are the same. The NiFi installed in SIT is working fine, however the one installed in Dev was also working absolutely fine, however it suddenly started throwing this exception couple of days back.

rohit_ravishank · ‎04-07-2017

Hi Team, I am using PutHiveQL to add a partition to the Hive Metastore. However the processor throws java.lang.NoClassDefFoundError:org/apache/http/config/Lookup exception as shown below, Please let me know if more information is required. Any help would be appreciated. Thanks!

rohit_ravishank · ‎04-03-2017

Thanks Meyer!!!

rohit_ravishank · ‎04-02-2017

Hi Team, In my file I have some records as show below, Sample Input- 1|transaction|I|2017-03-28|Current 1|master|B|2017-03-28|Current 1|master|A|2017-03-28|Current Required Output- 1|transaction|I|2017-03-28|Current 1|master|A|2017-03-28|Current While processing the file using NiFi, I want to consider only records having "I" & "A" in the third position. I am trying to use "RouteText" processor using regular expression to filter the records. However I am not successful in getting the correct expression for filtering the records. Could you please help me out? Thanks!

rohit_ravishank · ‎04-02-2017

Hi Bala, I am using ReplaceText processor to replace "|" delimiter to CTRL-A delimiter. However the pipe delimiters are not getting replaced by proper CTRL-A delimiter . Below is my sample input & required output, Input- 1|transaction|I|2017-03-28|Current 1|master|B|2017-03-28|Current 1|master|A|2017-03-28|Current Output- 1^Atransaction^AI^A2017-03-28^ACurrent 1^Amaster^AB^A2017-03-28^ACurrent 1^Amaster^AA^A2017-03-28^ACurrent Note - If the CTRL-A delimited file is viewed from Linux system using Vi editor, CTRL-A characters will be displayed as ^A. Thanks!

rohit_ravishank · ‎04-02-2017

Hi Team, I have a file in local file system(LINUX) with fields delimited with "|"(pipe). I am trying to read the file and replace the "|" delimiter with CTRL-A delimiter. However I am not able to do it. I tried using a ReplaceText processor with search pattern as "|" & replace pattern as "\u0001". Could you please help in resolving this query? Thanks!

rohit_ravishank · ‎03-12-2017

Hi, I have the below files in the input directory in the below pattern, cntl_table_1.xml table.1.dat cntl_table_2.xml table_2.dat cntl_table_3.xml table_3.dat cntl_table_4.xml table_4_1.dat table_4_2.dat The files with (.xml) extension has the structure of the table & files with (.dat) extension has the actual data for the table. If a particular data file for a table is more than 500MB, the files will be split into 2 or more files. In our case it is table_4. In total we will be getting files for 15 different tables. Once we receive the files(*.xml) for 15 tables(we have to wait till all 15 files have been received), we have to trigger a shell script, which looks for these 15(.xml) files in the input directory & creates Hive tables. Once the shell script completes successfully, we have to start processing the data files from the input directory. I have to achieve the above logic using NiFi but still not successful. Could anyone please let me know, if this logic can be achieved through NiFi? Also, please let me know if more information is required. Thanks!

rohit_ravishank · ‎02-22-2017

Team, I am getting the below files in a directory in the same order, companies.dat data.xml transaction.dat accounts.dat bankdetails.dat I use a List File processor to read the files from a directory. When we encounter ".xml" file, it should trigger a shell script & rest of the files (transaction.dat, accounts.dat & bankdetails.dat) should not be processed until script completes its execution. Once the script is complete, the rest of flow files should be processed. If .dat file then process the file(Path A). If .xml file, then trigger the script(Path B) & then process the rest of the .dat files(Path A). Could you please help us with an approach? Please let me know if more info is required. Thanks!

rohit_ravishank · ‎02-13-2017

Hi, When I create a template in one Nifi instance & upload the same template in another instance, sometimes the flow gets mis-aligned. Is it a common thing or I am making a mistake somewhere? Thanks!

rohit_ravishank · ‎01-23-2017

Hi @Sebastian Carroll Thanks for your reply! I understand NiFi doesn't really have a concept of a 'job' or a 'batch' & its like stream processing. I am using the scheduler to start a NiFi process group. If incase any proessor fails due to some reason and my flow gets hanged(does not proceed further & even records/file do not move to the failure relationship), is there a failure code which NiFi return, which we can capture & alert the users?

rohit_ravishank · ‎01-22-2017

Hi, We are planning to trigger NiFi using a scheduler(Control-M). I would like to know if incase of any failure, does NiFi return any return code(failure code/message), so that the scheduler can catch it & create an alert(ticket) ? Thanks

rohit_ravishank · ‎01-09-2017

Thanks Matt for the response!! I have updated my requirement above.Could you please take a look at it & help me out?

rohit_ravishank · ‎01-09-2017

Hi Matt, Below is my requirement on a high level, As I mentioned in my previous post, the data file contains the actual data & control file has the details of the data file. If there are 100 records in the data file, the control has these details(name of the data file,file size and record count). My requirement is the read the control file, get the record count(store it in an attribute). Then read the data file & validate it against the record count derived from the control file to check whether the record count matches. If the record count matches, I will process the data file, if not I have to move both control file & data file to reject path in HDFS. While I read the files, I fetch the control file(CTRL_ABC.txt) first & then from the control file name I fetch the data file(ABC.txt),store the filenames in 2 separate attributes.To read the contents of the files I use ListFile Processor & 2 FetchFile processors(to read the files separately).do the validation. If the validation fails, I need to move both the files to reject path. Is there a way to move these files at once or do I need to move the files separately using 2 different FetchFile & PutHDFS processors? Sample Flow- ListFile->UpdateAttribute(to get data/ctrl file names)->FetchFile(read ctrl file)->UpdateAttribute(get record count)->FetchFile(to read data file)->Custom Processor(to do validation) Thanks!

rohit_ravishank · ‎01-09-2017

Hi, I have a scenario where I get a data file & control file. The data file has the actual data & the control file has the details about the data file, say filename,size etc. Below are the file names, file_name - ABC.txt ctrl_file_name - CTRL_ABC.txt I have to read the files from local filesystem & write into hdfs. And say, I have the below flow in NiFi, List File -> Update attribute(file_name - ABC.txt,ctrl_file_name - CTRL_ABC.txt) Is there a way to combine these 2 attributes & read it using Fetch file processor & write it at once into HDFS? Thanks!

rohit_ravishank · ‎01-01-2017

I am having the header & trailer details as separate attributes. I am trying to add the header & trailer records to the flowfile. I am currently using 2 ReplaceText attributes to prepend/append the header/trailer deatils. Is their a better to achieve it? Example:- Input - header - 0$20011227$20011228$1 trailer - 2$5$1$2$3 Flow file Content - 1$Suresh 1$Kaamesh 1$Vijay Output - 0$20011227$20011228$1 1$Suresh 1$Kaamesh 1$Vijay 2$5$1$2$3

Online	Offline
Last Visited	‎09-28-2017 12:18 PM

Date Registered	‎12-23-2016 04:22 AM
Last Visited	‎09-28-2017 12:18 PM
Total Messages Posted	38
Total Kudos Received	2

Trouble creating databases in Hive using Spark 2.0

Re: Query on executing NiFi in a clustered setup w...

NiFi template upload migration from 1.1 to 1.2 ver...

Query on executing NiFi in a clustered setup with ...

Re: Parameterizing ReplaceText Processor

Re: Parameterizing ReplaceText Processor

Re: Parameterizing ReplaceText Processor

Re: Parameterizing ReplaceText Processor

Parameterizing ReplaceText Processor

Re: Need for Controller Services in Apache Nifi

Need for Controller Services in Apache Nifi

Min & Max age filter in ListFile Processor.

Re: Query regarding Back Pressure Threshold limit

Query regarding Back Pressure Threshold limit

Re: PutHiveQL Exception due to missing class file.

Re: PutHiveQL Exception due to missing class file.

PutHiveQL Exception due to missing class file.

Re: Filtering records from a file using NiFi.

Filtering records from a file using NiFi.

Re: Help needed in replacing Pipe delimiter with C...

Help needed in replacing Pipe delimiter with CTRL-...

Need inputs on processing data using NiFi for a pa...

Need help on a logic using NiFi.

Nifi flows getting mis-aligned.

Re: Triggering NiFi from a Scheduler

Triggering NiFi from a Scheduler

Re: Combining Attributes in NiFi

Re: Combining Attributes in NiFi

Combining Attributes in NiFi

Adding header and trailer details to a flow file.