Member since
10-24-2016
47
Posts
16
Kudos Received
0
Solutions
09-30-2017
04:24 AM
@Jorge moyano I tried the below code as you suggested but still facing the same problem File fileObj = new File(pathname);
FileBody fileBody = new FileBody(fileObj, ContentType.MULTIPART_FORM_DATA);
HttpEntity multiPart = MultipartEntityBuilder.create().addPart("template", fileBody).build();
HttpPost httpPost = new HttpPost("http://localhost:8080/nifi-api/process-groups/root/templates/upload");
httpPost.addHeader("Content-type", "multipart/form-data");
httpPost.setEntity(multiPart);
HttpClient httpClient = HttpClientBuilder.create().build();
HttpResponse response = httpClient.execute(httpPost); Response HttpResponseProxy{HTTP/1.1 500 Internal Server Error [Date: Sat, 30 Sep 2017 04:24:37 GMT, X-Frame-Options: SAMEORIGIN, Content-Type: text/plain, Transfer-Encoding: chunked, Server: Jetty(9.4.3.v20170317)] ResponseEntityProxy{[Content-Type: text/plain,Chunked: true]}}
... View more
09-29-2017
01:09 PM
@Andrew Grande can you please help on this?
... View more
09-28-2017
10:06 AM
I am trying to create a REST API call to import a template into my NiFi UI post which Instantiate the same.
Below is the code which I tried, String pathname = "D:\\Users\\bramasam\\Downloads\\BalaBackUp.xml";
String restString = "http://localhost:8080/nifi-api/process-groups/f80896d4-c71f-3395-d527-8c6bd69f44d0/templates/upload";
HttpPost httpPost = new HttpPost(restString);
File fileObj = new File(pathname);
httpPost.addHeader("Content-type", "multipart/form-data");
FileEntity fileEntity = new FileEntity(fileObj, ContentType.MULTIPART_FORM_DATA);
httpPost.setEntity(fileEntity);
HttpClient httpClient = HttpClientBuilder.create().build();
HttpResponse response = httpClient.execute(httpPost);
StatusLine status = response.getStatusLine();
System.out.println(status.getStatusCode()); I am getting a response code of 500 and with the below response HttpResponseProxy{HTTP/1.1 500 Internal Server Error [Date: Thu, 28 Sep 2017 09:43:28 GMT, X-Frame-Options: SAMEORIGIN, Content-Type: text/plain, Transfer-Encoding: chunked, Server: Jetty(9.4.3.v20170317)] ResponseEntityProxy{[Content-Type: text/plain,Chunked: true]}} Below is the {id} from the BalaBackUp.xml file which I am trying to import from <?xml version="1.0" ?>
<template encoding-version="1.1">
<description></description>
<groupId>bd5dba8b-015d-1000-1fd5-450ede38b7a5</groupId>
<name>BalaBackUp</name>
<snippet>
<processGroups>
<id>f80896d4-c71f-3395-0000-000000000000</id>
<parentGroupId>29a5776d-9728-3fee-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>0.0</y>
</position>
<comments></comments>
<contents>
<connections>
<id>c0d0e26d-5ee2-3d60-0000-000000000000</id>
<parentGroupId>f80896d4-c71f-3395-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>f80896d4-c71f-3395-0000-000000000000</groupId>
<id>1f9e926a-71fc-356f-0000-000000000000</id>
<type>PROCESSOR</type> can you please help me on what I am missing?
... View more
Labels:
- Labels:
-
Apache NiFi
09-15-2017
10:28 AM
I am trying to automate the Deployment of nifi flow. We searched and found a link (given below) https://github.com/aperepel/nifi-api-deploy It contains the Groovy code. But we are looking out for REST API in Java. The link also says that, REST APIs and concepts have changed significantly from NiFi 0.x to NiFi 1.x (It is explained for NiFi 0.x in the link) Since we are using NiFi 1.3.0, I thought it would be the best to ask for help.
... View more
Labels:
- Labels:
-
Apache NiFi
07-31-2017
12:37 PM
@sri chaturvedi Please try converting both the variables to String datatype before comparison. Sample: ${d_last_updt:toDate('yyyy-MM-dd'):format('yyyyMMdd'):toString():gt(constant value:toString())} Hope this helps
... View more
07-31-2017
12:29 PM
@Gayathri, I am not sure about your complete requirement. But, the MergeContent processor is used only when you have to merge the file contents which you have split up-steam using SplitContent processor. I can suggest you with some workarounds, 1. You may have a common filename for files which follow specific schema 2. Write the file into a local file system using PutFile or into Hadoop using PutHDFS processors 3. Select 'Conflict Resolution Strategy' property to 'Append' when you write the file to disk Hope this helps.
... View more
03-17-2017
06:23 AM
Hi @Rooster Raul. Your properties looks good. Can you please check if the folder you are trying to access have necesssary permissions for the user credentials. Becasue that is when you wont see any error during connection and everything stays idle thinking that there is no files to be listed.
... View more
03-16-2017
06:39 AM
1 Kudo
@Rooster Raul Can you please put in some screenshot of how your properties looks like. Against which property was the directory "/amazon/jungle/" configured? Is it against the 'Remote Path' property. Anyways here is the quick check list property. 1. Remote Path: /amazon/jungle/ 2. File Filter Regex: Here, you need to put in a RegEx that matches your file name. You can make use of this link to check if your RegEx exactly matches the file name (http://regexr.com/)
... View more
02-07-2017
12:56 PM
Can anyone please help me on this?
... View more
01-31-2017
11:28 AM
1 Kudo
@Raj B Please check for the 'Writes Attribute' in 'usage' of the processor. It describes the attribute each processor writes in case of failure. Below is the Usage of 'ExecuteStreamCommand' and it details 4 different error it writes during the execution.
... View more
01-27-2017
07:58 AM
1 Kudo
NIFI CSVtoAvroConverter Processor does not support decimal data type. We have replaced decimal with double instead.
When decimal is replaced with double in the outgoing schema option, we are facing the below issue.
Number which has precision=”25” , scale=”3” eg 1234567890123456789012345.123 is stored as 1.2345678901234568E24 in Hive.
If someother process in the future tries to read this value in hive as a string, the meaning of the field completely changes. Can you please provide a solution for this.
... View more
- Tags:
- Data Ingestion & Streaming
- NiFi
- nifi-processor
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
Labels:
- Labels:
-
Apache NiFi
01-25-2017
07:46 AM
@Raj B You can do this in many ways, 1. You can have an update attribute processor at the beginning of every Processor Group to specify the module name. But this cannot be replicated at processor level. Here, you keep updating the same attribute throughout the flow 2. You have have a 'Metadata' -> as a table or a flat file that will log all the attributes of the flow file. Please check AttributeToJson processor. This can log flow attributes both as an attribute or content. Also, note that every nifi processor adds additional 'error' attribute in case of any error which can be captured by AttributeToJson processor. Hope this helps
... View more
01-21-2017
09:34 AM
Can the password in Hiveconnectionpool be passed via a configuration or property file
... View more
01-20-2017
03:57 PM
Thanks @Matt. Also, we see that the hive connection pool service prompts for username and password. The password when keyeed-in is visible. How can this be made invisible while keying in the password.
... View more
01-20-2017
02:06 PM
Whenever we deploy a new template in nifi UI, the PutHiveQL processor seems to be in disabed state. Can you please let me know ways in which it can be enabled without manual intervenssion or how can I automate it.
... View more
Labels:
- Labels:
-
Apache NiFi
01-04-2017
09:27 AM
I see an option 'Concurrent Tasks' in Scheduling in Nifi version 2.0. Here are my questions regarding Concurrency. 1. I see this only at indivudual processor level. Can this be set at 'Processor Group' level. In short can I parameterize it. 2. How can I arrive at this number. What are all the fators that decide concurrency. E.g. Memory, average data load etc, Other application in the same box like spark.
... View more
Labels:
- Labels:
-
Apache NiFi
11-15-2016
10:26 AM
I have a CSV file with 9 columns. How can I remove duplicates among columns 4 through 9? What we tried: 1. Split 1-4 columns in a file 2. Split 4-9 columns -> Deduplicate records Now, i tried using 'ReplaceTextWithMapping' to merge the files with 4th column (Common on both files). But I am not sure if my approach is right. Is there any other way to achieve this.
... View more
Labels:
- Labels:
-
Apache NiFi
11-15-2016
07:16 AM
I am using the CSVToAVRO converter processor. Is there a way I can mention the field as "not null" in the record schema. Or, do we have any other method to do this
... View more
Labels:
- Labels:
-
Apache NiFi
11-14-2016
02:03 PM
I am using the 'ConvertCSVToAVRO' processor. My requirement is, I should further process the file only if all the records matches the schema. But, now, the processor generates flowfile as soon as the 'record' matches the schema. How can i continue with the flow only with 'entire file' matches the schema
... View more
Labels:
- Labels:
-
Apache NiFi
11-10-2016
01:19 PM
1 Kudo
I have a requirement where I will have to decide the flow based on the record header. Since the file is huge, I don't want to waste time reading the entire file where only one record is enough. Is there a way I can achieve this.
... View more
Labels:
- Labels:
-
Apache NiFi
11-10-2016
10:33 AM
3 Kudos
I have a 'Control A' separated file. I am trying to append each line with a date value with the regular expression, ${srcdelim}${now():format('mm-dd-yy')} where ${srcdelim} contains the value '\u0001' from a schema file. Apparently, the output file contains the string \u0001 rather than a control A delimiter. I have attached my sample input and output. Character Set: UTF-8
... View more
Labels:
- Labels:
-
Apache NiFi
11-09-2016
01:33 AM
I am trying to Append each line with a time-stamp or a date like ${srcdelimiter}${now():format('mm-dd-yy')}
... View more
11-08-2016
03:23 PM
@Matt Burgess we decided to do a routeOnAttribute for files which are separated by Control A or CSV. We now face a new challenge while using Control A separated file. What we do is, we get this delimiter as a flowFile attribute and we use this attribute say ${srcdelim} (whose value is - \u0001) in ReplaceText Processor. The resultant file is not Control A separated file. I just see a character \u0001 instead. can you please help me replace a control A character
... View more
11-08-2016
09:34 AM
2 Kudos
Can someone please provide me with some link that explains the basic concepts of Apache Nifi, Flowfile, how efficiently it handles memory, master slave relations, etc?
... View more
Labels:
- Labels:
-
Apache NiFi
11-03-2016
05:45 AM
Thanks @Matt Burgess and @Andy LoPresto. My requirement here is, I am keeping watch on a folder where my input files come in. These files can have different delimiters \u0001, \u0002 or a CSV. So, I was thinking if we can convert these files into AVRO by passing the delimiter as an attribute in the flowfile. Is there an efficient way to handle this other than using replace text.
... View more
11-02-2016
10:27 AM
I am in need to convert a CTRL A separated file into AVRO. How can I do this? I tried using ConvertCSVToAvro Processor and tried to pass an expression to 'CSV delimiter' property. It does not accept regular expression. Is there any other way to achieve this in Apache Nifi?
... View more
Labels:
- Labels:
-
Apache NiFi
11-01-2016
07:39 AM
This is my flow SplitText -> De-Deuplicator (here, I use the processor to De-Dupe the records within a File) -> MergeContent Here, my question is regarding the working of MergeContent. Say suppose I keep watch on a folder where 6 files of 2GB each fall. Will the output of mergecontent maintain the input files as is? I ask this question because I see various optional properties in MergeContent which says "if not specified there is no maximum". What exactly does this mean? Is there any default maximum which cut off the files?
... View more
Labels:
- Labels:
-
Apache NiFi
10-24-2016
01:26 PM
1 Kudo
Hi, I have a requirement, where I am in need to Log all the flowFlile attributes in a separate text file. What I read is that Apache Ni-Fi by design have Provenance which gives a detailed Log information. But, however, since we are new to Ni-Fi, we are not sure about the access levels that to the Production support teams would have, because of which we would like to Log the Attributes in a Text file for which we can give them all access.
I also tried using the existing processor "LogAttribute" where I don't see an option to write out into a physical file location.
... View more
Labels:
- Labels:
-
Apache NiFi