Member since
01-11-2016
355
Posts
230
Kudos Received
74
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
8285 | 06-19-2018 08:52 AM | |
3212 | 06-13-2018 07:54 AM | |
3665 | 06-02-2018 06:27 PM | |
3962 | 05-01-2018 12:28 PM | |
5502 | 04-24-2018 11:38 AM |
10-20-2017
10:17 AM
Hi @pratik vagyani What's your HDP version? Have you followed official documentation on the topic? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_cloud-data-access/content/s3-get-started.html
... View more
10-20-2017
10:14 AM
Hi @Balakrishna Dhanekula What you need first is to add this node your Ambari managed cluster. This can be done by installing an Ambari agent on this node then adding it to Ambari. Later, you can install components that you want on this new node. There's a good article on this topic here : https://community.hortonworks.com/articles/80284/hdf-2x-adding-a-new-nifi-node-to-an-existing-secur.html
... View more
10-20-2017
05:28 AM
@Kiem Nguyen you don't need to do all this. I think it's a bad idea to change your JSON data just to be able to parse it. You can have surprise when your application evolve. So as a summary: Use my solution above to extract your key Use the below solution to extract the content With Key and Value in hand, use put MongoDB to store your data in the right collection To extract your JSON content, use evaluatejsonpath with the following configuration. Add a dynamic key my_content and use this JSON path $.* You get all the data that you are looking for: Can you try this an let me know if it works ? Thanks
... View more
10-20-2017
04:51 AM
Hi @Adrian Oprea There's no specific processor for H2 DB but you can use it with standard JDBC processor in NiFi. You can use a Database Connection Pooling Service with JDBC driver forH2 DB and use processor like PutDatabaseRecord or any other JDBC processor to ingest data. There are lot of questions/article on SQL processor in NiFi. Have you tried this?
... View more
10-19-2017
08:52 PM
Hi @senthilkumar kalimuthu For 1 look at UpdateCounter processor https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.UpdateCounter/index.html For 2 look to wait notify processors. They are similar to P and V functions if you are used to Semaphores https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.Wait/index.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.Notify/index.html
... View more
10-19-2017
07:48 PM
Hi @Alvin Jin Happy that this information was helpful. Yes, you need to add your input port at the root canvas. If you need to work with process group, you still need to have an input port at root, then add an input port in your process group, then connect your root input port with process group: If you have recursive process groups you should do the same thing to connect your external input port until your internal process group. Regarding data distribution, you should know that S2S do batching of data to have better performance. So it tries to send maximum data to one node before moving to the next one. This is very efficient. But if you have few flows (when testing for example) all flow files will go to a single node. Mainly S2S open a connection to a node and try to send data for x miliseconds. If this time is not enough it moves to the next node. Other parameters are flow file count and size. These parameters can be defined globally. In the last versions of NiFi you can define these parameters at each input port level which is very convenient. I show you how to configure this below (red instructions) You can small values to test load balancing. Please close the thread if it answers your question. If you have further questions on load balancing ask new question. This will be helpful for other users. Thanks
... View more
10-18-2017
10:58 AM
@Kiem Nguyen Since you have the key with this regex, you can use it to get value with EvaluateJsonPath.
... View more
10-18-2017
10:23 AM
1 Kudo
@Kiem Nguyen I am not a Regex expert but you can test this and work on it to have a better implementation. ExtractText with following Regex (\w+) You can see that there's an attribute projectname which contains project_1
... View more
10-18-2017
09:59 AM
Exactly. This is what I am looking for.
... View more
10-18-2017
09:58 AM
Hi @Kiem Nguyen In this case you can use ExtractText to scan your flow content and extract the name of the project. With the right regular expression you can extract your key and store in an attribute projectname. Once you have this, use it in the PutMongodb to update the required collection using ${projectname} for the Mongo Collection Name. Can you try this and let me know if this works ? Thanks
... View more