Member since
12-21-2015
57
Posts
7
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3963 | 08-25-2016 09:31 AM |
08-04-2017
02:09 AM
Thanks for your answer. We have NiFi below v1.2.0. Is there a workaround while we are not yet upgrading our cluster?
... View more
08-03-2017
08:40 AM
I am working on a workflow inside a Process Group. The flow inside the Process Group ends with an update on SalesForce API, but what this Process Group outputs is the original input. I will only send to the Output Port once I get a successful Response from the processor calling the SalesForce API. What should I do within the Process Group to to have this result?
... View more
Labels:
- Labels:
-
Apache NiFi
07-05-2017
06:13 AM
Is it possible to place a NiFi processors name or id to a flowfile that it outputs? For example a Processor InvokeHTTP I named as CallRESTEndpoint, the flowfile that would come out of it will have attribute, like origin:CallRESTEndpoint? Or if possible to create a chain of origin that will append the processor's name or id to this attribute. I need this in error / failure handling. If you can share a better idea in tracing errors / failure, that'd be a good idea. (But please answer the original question also, if there is one.)
... View more
Labels:
- Labels:
-
Apache NiFi
06-21-2017
07:48 AM
We have a custom query that we wish to put inside the ExecuteSQL processor, but this one proves a bit tricky: SELECT * FROM mytable AS OF TIMESTAMP TO_TIMESTAMP('${now():format('yyyy-MM-dd HH:mm:ss')}', 'YYYY-MM-DD HH24:MI:SS'); The processor is CRON driven where we run it every 15 minutes. The expression language inside the first single quotes is NiFi's expression language to get the current time in the format specified. Note that the expression language: ${now():format('yyyy-MM-dd HH:mm:ss')} itself uses single quotes. I tried substituting double quotes for the Oracle query, but Oracle won't accept the syntax as valid. So how do we substitute the intended date-time string inside the first single-quotes?
... View more
Labels:
- Labels:
-
Apache NiFi
06-19-2017
11:30 AM
In NiFi, is there an existing FlowFile attribute that can tell when it exited (or when it was outputted by) a certain processor? I am trying to measure end-to-end processing time of a workflow.
... View more
Labels:
- Labels:
-
Apache NiFi
06-07-2017
03:09 AM
1 Kudo
We have an SFDC REST endpoint (let's call this UpdateSFDC) where we stream and post database changes, in JSON format, to reflect it to SalesForce object store. Before we get to that endpoint, we need to Login to SalesForce first to this endpoint: https://test.salesforce.com/services/oauth2/token So we use InvokeHTTP (let's call this Login2SFDC) and appends the parameters to the Remote URL field in the processor: https://test.salesforce.com/services/oauth2/token?grant_type=password&client_id=<client_id>&client_secret=<client_secret>&username=<username>&password=<password + key> This will respond with a token which we need to extract and add as an attribute in every JSON format flowfile (let's call this CreateSignedRequest). Here's a NiFi schematic: The Login2SFDC is configure this way: However, this is quite inefficient because for every JSON flowfile, we need to login to SalesForce everytime, which will add overhead, not to mention the risk that Login2SFDC REST endpoint will eventually throw some sort of a max limit request reached. Rather, we want to login once and as long as the token is valid (not yet expired), then we will just add the token as an attribute to the JSON flowfile and go to UpdateSFDC REST endpoint straight (via CreateSignedRequest). So I will convert the Login2SFDC processor from InvokeHTTP to ExecuteScript. I have 3 questions for this problem: 1. How do I this in ECMAScript and/or Python? When I use their respective native HTTP client functionalities, they do not seem to work. Should I be importing and using Java libraries (like HttpClient) instead in which case, the scripting language is simply a wrapper to Java libraries? 2. I am successful in getting a response from Login2SFDC REST endpoint when in NiFi. I tried writing a corresponding Java code (using same exact endpoint and credentials) using HttpClient library: CloseableHttpClient client = HttpClients.createDefault();
String urlStr = "https://test.salesforce.com/services/oauth2/token?grant_type=password&client_id=<client_id>&client_secret=<client_secret>&username=<username>&password=<password + key>";
HttpPost httpPost = new HttpPost(urlStr);
String jsonPayload = "{\"qty\":100,\"name\":\"iPad 4\"}";
StringEntity entity = new StringEntity(jsonPayload); httpPost.setEntity(entity);
httpPost.setHeader("Accept", "application/json");
httpPost.setHeader("Content-type", "application/json");
CloseableHttpResponse response = client.execute(httpPost);
System.out.println(response.getStatusLine().getStatusCode());
InputStream is = response.getEntity().getContent();
String responseStr = IOUtils.toString(is, "UTF-8");
System.out.println(responseStr);
client.close(); but I get this response: Status Code 400 {"error":"invalid_grant","error_description":"authentication failure"} Am I missing something in the Java code? 3. Is there a better NiFi flow design for this kind of problem? Hi @Matt Burgess, can you help?
... View more
Labels:
- Labels:
-
Apache NiFi
05-26-2017
05:50 AM
We have a VM with JDK 7 and a number of things depend on it. We are installing NiFi v1.2 but this one is compatible with JDK 8 only. We cannot remove JDK 7 because we cannot risk to break the dependency. Is it possible to install JDK 8 and point NiFi to use this, without changing the JAVA_HOME variable pointing to JDK 7?
... View more
Labels:
- Labels:
-
Apache NiFi
05-25-2017
09:46 AM
We have an Oracle DB and we run nightly batch job but when that job would end varies. Now we have a NiFi workflow getting data from this Oracle DB but should not be during the batch job run. How do we turn off/on the entry point processor (say ExecuteSQL) whenever the Oracle Batch job is running/has stopped? We thought of using Oracle DB trigger and then invoke a HTTP request within that trigger when the job would end, but is there a processor in NiFi where you can put up a simple web service / REST endpoint and then use that to trigger or turn on the ExecuteSQL processor?
... View more
Labels:
- Labels:
-
Apache NiFi
04-25-2017
08:37 AM
We have a system composed of many databases and tables and we want to use NiFi to query these tables based on our requirements. Since NiFi's QueryDatabaseTable processor is statically linked to a single table, what we intend to do is to dynamically generate many processors of this kind to match the number of our tables in our different systems. Is this possible using ExecuteScript processor (or anything similar)?
... View more
Labels:
- Labels:
-
Apache NiFi
03-21-2017
01:21 PM
Thanks for the tip Matt. I am now in the part where I have two InvokeHTTP processors, the first for login or getting the access token (let's call this SFDCLogin), and the second is where the SFDC REST API is configured for posting changes to SalesForce's ObjectStore (let's call this SFDCUpdate). I having a hard time doing the plumbing on this. The idea is, the FlowFiles (wherever they come from, assuming its GetFile processor) would move to SFDCUpdate and then if rebuffed due to missing or expired access token, will go to SFDCLogin processor and do whatever it needs to do and then go to SFDCUpdate. Note that I only need to pass through SFDCLogin once, once authenticated; not multiple times. Any custom logic, attribute, property, or expression language to make this work? I also tried using the ExecuteScript processor using ECMAScript or Python, and do the REST API calls there on the login part. However, I'm hitting a roadblock on XMLHttpRequest (in ECMAScript) or requests (in Python) as it seems I need to point to their libraries. Should I be using a Java library (jar of HttpComponents perhaps) instead as these scripting languages are meant to wrappers only? Or can I point to their respective libraries natively?
... View more