Member since
06-08-2017
1049
Posts
518
Kudos Received
312
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 11197 | 04-15-2020 05:01 PM | |
| 7097 | 10-15-2019 08:12 PM | |
| 3088 | 10-12-2019 08:29 PM | |
| 11424 | 09-21-2019 10:04 AM | |
| 4315 | 09-19-2019 07:11 AM |
12-28-2017
06:41 PM
3 Kudos
@Paresh Baldaniya We cannot change the existing filename in HDFS but we can do alternate solution in NiFi as below. You need to use UpdateAttribute processor before PutParquet processor. Update Attribute we are going to update the filename before putparquet processor so that everytime when file goes to putparquet processor will have same file name every time. Add new property to Update attribute processor filename
desired_parquet_filename.prq Configs:- PutParquet:- So we are going to have same filename everytime and in this processor we need to change Overwrite Files property to True //if the same filename exists in the directory processor will replace the existing file with new file Configs:- Flow:- 1.GetFile
2.UpdateAttribute //change the filename by adding filename property
3.PutParquet //change the Overwrite files property to true If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors.
... View more
12-24-2017
06:06 PM
1 Kudo
@Mohammed Syam If you want to compare the response time of select queries 1.By using Ambari Hive View won't display the execution time of the query. Hive View Execution:- Logs Tab Hive View:- If you click on Logs tab also there is no logs for the query. It is impossible compare the time taken for select query to execute, because for select queries there is no map reduce job is going to be initialized, if application id is not created then we cannot compare them programatically. 2.Another way to get at least the execution times for select query is running from hive from command line(not from ambari hive view) Open Hive from command line and execute the select query. Once select query finishes at bottom of the results hive shows how many rows has been selected with the response time will be displayed. Example:- hive# select * from text_table;
+----------------+----------------+------------------+----------------+------------------------+--+
| text_table.id | text_table.dt | text_table.name | text_table.ts | text_table.dtts |
+----------------+----------------+------------------+----------------+------------------------+--+
| 1 | 2017-10-10 | hcc | 12:00:00 | 2017-10-10 12:00:00.0 |
| 1 | 2017-10-11 | foo | 12:00:00 | 2017-10-11 12:00:00.0 |
| 1 | 2017-10-12 | foo | 12:00:00 | 2017-10-12 12:00:00.0 |
| 2 | 2017-11-10 | bar | 09:00:00 | 2017-12-23 09:00:00.0 |
+----------------+----------------+------------------+----------------+------------------------+--+
4 rows selected (0.105 seconds) Scroll down to the last line in the above snippet and you can view that 4 rows and execution time is 0.105 seconds. If you are running from Hive from command line hive displays how many rows are displayed and what is the execution time for the query as a last line in results. If you are using Ambari Hive View won't display these stats also. Only way to compare results of select query is Execute your select from Hive command lines and compare them manually.
... View more
06-12-2019
08:02 PM
Hi @Shu I am trying to create the directory with current time stamp in hdfs and as above I tried but did not work since it doesnt have extract_date what should I use to get the desire output ? Right now my flow is : ListHDFS -----> Updateattribute ----->PutHDFS. any suggestions please ?
... View more
12-25-2017
09:32 PM
1 Kudo
@Shu : There was a culprit record ending up with new line. Thanks for assisting. escapeJson() is working perfect now. thanks.
... View more
12-22-2017
03:16 PM
1 Kudo
@Shu that did the trick! Many thanks for your help and your patience as well.
... View more
12-19-2017
03:27 PM
@Lukas Müller Use EvaluateJSON path processor with below configs:- then we are adding longitude and latitude attributes to the flowfile as then use url in invoke http processor as http://api.openweathermap.org/data/2.5/weather?lat=${location.latitude}&lon=${location.longitude}&APPID=myapikey If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors.
... View more
12-18-2017
02:40 PM
2 Kudos
@Rajesh AJ Use Get File (or) List/FetchFile processors to fetch the file. then use 1.Split Text Processor(if you are having each url for a line) with line split count as 1 If your file having 4 lines then after split text processor will give seperate flowfiles for each line. Example:- Input 1 file(having 4 lines) and output will be 4 flow files(each line as seperate flowfile) 2.Extract Text processor to extract the url's to attributes for the flowfile. According to your file content size you need to change the Maximum Buffer size property and i'm extracting all the contents of the flowfile to url attribute by using regex .* url
(.*) //extract the whole content of the flowfile add the content to the flow file url attribute then use 3.invoke http processor with ${url} We are going to use the extracted url attribute from extract text processor in invoke http processor. The extracted url attribute will be changed dynamically according to the flowfile content. Flow:- 1.Get File
2.Split Text
3.Extract Text
4.Invoke HTTP If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors.
... View more
12-20-2017
08:23 AM
Thank you very much for your help, I didn't consider the expression support.
... View more
12-14-2017
03:58 PM
2 Kudos
@balalaika For that case you need to specify Demarcator property as Shift+enter Configs:- For merge content reference https://community.hortonworks.com/questions/149047/nifi-how-to-handle-with-mergecontent-processor.html
... View more
12-12-2017
01:40 PM
2 Kudos
@Ranith Ranawaka
After Attributes to Json processor use Replace Text processor with Search Value "clientid"\s+:\s+"(.*)" Replacement Value "clientid" : $1 Replacement Strategy Regex Replace Evaluation Mode Entire text
Configs:- Input:- {
"clientid" : "2",
"id":"1",
"name":"HCC"
} Output:- {
"clientid" : 2,
"id":"1",
"name":"HCC"
} So we are searching for the value of client id and replacing the value without quotes by using replace text processor.
... View more