As far as I know, there is no Kotlin in the software stack ... but you should wait for an answer from somebody with more experience. Nevertheless, in the meantime, you can have a look for yourself, as the code is public: https://github.com/apache/nifi ... View more

Now, based on your Flow Logic, where do you encounter problems and what are those problems? In addition, how do you retrieve dynamically the destination and the size at destination? As far as I know, PutFile does not write those attributes so you will have to manually define them in your Flow and I do not see any UpdateAttribute or UpdateContent on your canvas. The size, the file type, the filename and the source are written by default when the file is extracted and your can further use them with the help of the NiFi Expression Language. ... View more

hi @NafeesShaikh93, What logs are you trying to push into Graylog? The logs generated by NiFi when transferring the files from Source to Target, or are you talking about other logs?   What have you tried and why did it not succeed? Let's start with that, as your Flow Logic was built specific for your use case and somebody else's logic might not suite your needs. ... View more

Hi @ManishR, 1. How to find the processor or processor group by Integration Name/Folder/Event Name? What do you mean by integration name, event name and folder? Folder I assume you refer to Processor Group (external or normal) but integration name and event name are unknown to me. If you are trying it identify Processors, Processor Group, Queues or any other objects from within your Canvas Board, you mostly have two options: 1-you either use the NiFi REST Api (https://nifi.apache.org/docs/nifi-docs/rest-api/index.html ) or 2-you use the search bar in the top right of the screen. 2. How to find the source and destination file details in any of the processor or processor group? What do you mean with this question? In NiFi there are a series of processors which can be used to retrieve SOURCE files (ListSFTP/FetchSFTP/GetSFTP,GenerateFlowFile,InvokeHTTP,ConsumeKafka,ExecuteSQLRecord,etc) and there is a series of processors used to save your data in a TARGET location (like PutFile,PublishKafka, PutGCSObject, PutHDFS, PutDatabaseRecord,PutS3Object, etc). If you are to speak about a certain processor, the source data is within the queue linked to the specific processor, whereas the target data is located within the queue linked from your specific processor to another processor. Yet again, your question is quite general and ambiguous to get a specific answer.   Sorry but I do not understand what exactly you are trying to achieve so if you require a more technical and more elaborate answer, you will have to come with a more detailed questions, based on a specific use case - maybe even your NiFi Flow. ... View more

@ManishRcan you please describe your problem a little bit better? What did you try to do? Why did it not work? What should have been the expected outcome? What are you actually trying to achieve?   What you have provided so far is quite general like " which is the best car in the world " / " how can I find the best milk " / "who came first? the chicken or the egg". As you described your question, the answer is: your existing processors are present on the canvas. The files are in the source, the target or the queues between the processors, whereas the integrations are the integrations which you have constructed or used so far. ... View more

@ManishRcan you please describe your problem a little bit better? What did you try to do? Why did it not work? What should have been the expected outcome? What are you actually trying to achieve?   What you have provided so far is quite general like " which is the best car in the world " / " how can I find the best milk " / "who came first? the chicken or the egg". As you described your question, the answer is: you can find the source files in the source and the target files in the target, where the source is what you have defined your source to be and the target is the target you have set. ... View more

hi @Satya1,   The inactive sessions are getting terminated after 25 mins after the extraction is complete. For example, if you extraction takes 25 minutes to load the data from your database and into the flowfile (no matter the file type), you will keep on seeing those inactive sessions for another 25 more minutes, meaning that they will be completely gone after 50 mins, since you started your flow.   If you want, you can reduce the time to a lower value, as 25 is a bit big for most cases. You can modify it into 5 minutes and test it for yourself. I did it and my flow took 2 minutes, the sessions got evicted after 5 and in 7 minutes all the sessions were gone. ... View more

@hackerwayhow does your state-management.xml look like? Are you using the embedded zookeeper by any chance? ... View more

hi @wffger2, I just tested on my local machine and if I download the NiFi Flow definition for a Processor Group, I can find each processor ID in that Processor Group. Are you sure you are not looking in a different export from a different environment? If the same exact flow is imported on a different environment (meaning other servers) and you download the Flow definition, you will not have the same IDs. For example, in your Process Group on Dev you have an Processor PutFile which has the id 1d29eb87-d66f-31ba-d653-93b704036191. If you exported everything on the UAT Environment that Processor Group and the Processor will not have the same exact IDs as on DEV because NiFi generates new UUIDs for those elements --> maybe the ID 1d29eb87-d66f-31ba-d653-93b704036191 is already in use on the UAT environment by another processor. I might be wrong in my post and you could wait for another answer, as somebody might know otherwise. ... View more

Assuming that there would be a way to check for flow files which went through failure, how would you extract them? Manual search within the Data Provenance Menu from within the GUI? Or using the REST API? Now, what I do know is that you cannot identify those files that easy, because they do not write any specific lines within the Data Provenance. If you have a processor right after the failure queue (or if you terminate the failure queue in the processing processor) you can query using that Component ID and identify the type = DROP, meaning that those files have been "processed" (Indicates a provenance event for the conclusion of an object's life for some reason other than object expiration). More about the types can be found here: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.2/bk_user-guide/content/provenance_events.html ... View more

Well if I understand your flow logic correctly, what you are trying to achieve is not really possible, so please correct me if I am wrong. In your QueryRecord you receive input from the InputPort and from the QueryDatabaseTable Flow as well. Based on the FlowFile coming from that InputPort, you are trying to filter the data, using a SQL which is built on the input coming from the QueryDatabaseTable Flow, right? Basically you want to filter the InputPort Data (FlowFile1) based on the ${value} from QueryDatabaseTable (FlowFile2). Based on my current experience with NiFi, such a thing is not possible because the processors work on a single FlowFile and not with 2+ FlowFiles (unless merged). But maybe somebody with more experience can tell you otherwise. You would need to rethink your flow and build something else --> hence the idea with LookupRecord/LookupAttribute. ... View more

Have you thought on using LookupRecord instead of QueryRecord? In LookupRecord you can define your RecordReader, to be able to read your JSON input data, as well as a RecordWriter, to write your data in whatever format you want. In addition to this, you have the Lookup Service, where you can use the DatabaseRecordLookupService (as Controller Service) and connect to your database and looking up a Key-Value Combination based on a table name. Or you can try with LookupAttribute if you want to add an attribute in your JSON File with the value you want to use to perform the routing of the file. You can use in LookupAttribute the SimpleDatabaseLookupService where you connect to your DB and define the table in which you query the Key-Value combination. As I did not fully understand your question (nor do I see one), the provided answer is a generic one. ... View more

hi @Sivagopal , What NiFi Version are you running and what property did you set in nifi.sensitive.props.algorithm? As far as I know, since version 1.14, the algorithm PBEWITHMD5AND256BITAES-CBC-OPENSSL might have been deprecated and you need to use a newly supported/updated version, which can be found here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm --> Property Encryption Algorithms Section. ... View more

Hi @Ray82 , I am shooting here in the blind, but have you tried setting the command Path = /usr/bin only and the Command arguments = rm -f and_everytyhing_else? I am asking because I am using the ExecuteStreamCommand similar as you are but with python and within the command path i set python and the entire thing I have to execute under the command arguments and it works like a charm. It might not be the recommended way of using this processor but at least I got my things running. ... View more

well if you want to modify the port, you have to stop nifi, modify nifi.properties and restart nifi. But without knowing how you configured NiFi ( content of nifi.properties) nobody can tell you how to solve your issue. Maybe you can add the content of nifi.properties here and afterwards we can guide you further. Nevertheless, how Matt stated below, you need to check your application logs and see if NiFi started correctly. If NiFi starts correctly, you will see a log line saying that NiFi UI (User Interface) is available at a specific URL. You need to take that URL and paste it in your browser. If such a line is not available, than you need to check the logs to see what the error is. ... View more

Without knowing what you have configured, I will try and assume some of your configurations: - based on your installation, I assume that you are using HTTP and not HTTPS, right? If I take a look in your screenshot, I can see that you are running on port 8443, which is the default port configured within nifi.web.https.port. I assume that you wrote the host and the port in the nifi.web.http.host and the nifi.web.http.port properties and not the HTTPS properties, right? If not, this is what might be causing your issue. - besides that, have you check the logs for any errors and so on? I am talking here about the bootstrap logs (nifi-bootstrap.log) and the application logs (nifi-app.log). If everything starts normally, within the application logs you should see the link where NiFi is available. Otherwise, there is an error and you have to take a look. PS: it would help if you could attach the nifi.properties here, with all the confidential information written as standard placeholders (like <hostname>, <password> , etc) ... View more

can you maybe post the statement you are running in mongo + the result and afterwards the query you execute in NiFi and the entire error message you receive? Right now it seems to be an error regarding the format of the date/query. I am not that familiar with mongoDB but maybe in the meantime somebody with more experience will intervene. ... View more

Hi @judywatson , the answer pretty much depends on the actions you are going to do in NiFi. The easy answer is yes, you can run NiFi on that gaming laptop, but don't expect to be able to run complex end-to-end flows :). For example I have used a small linux server, with 8GB of RAM and a CPU with 4 cores to handle a couple of my flows (like extracting data out of a couple of DBs, transferring files from a SFTP and into AWS, converting some data CSV-AVRO, PARQUET-AVRO). I assigned 4GB to HEAP and left it to run and I had no issues with it. But again, it mostly depends on what you are trying to achieve and the size of the data you are trying to ExtractTransformLoad 🙂 ... View more

Why does it not work in NiFi? What sort of an error message do you receive? NiFi basically executes what you set in the query property. If you can execute it in mongo, it should work from NiFi as well. ... View more

maybe you can try with something like? I have no access to any MongoDBs right now, to test it myself 😞 { "TransactionHistory" : { " $gt " : "<your_value_here>" }}    ... View more

I see that you are using something which is not default nor belonging to NiFi. I would suggest you have a look in your JAR Files from PROD and see if you can find something which might point to something like batchiq. Most likely the JAR file is missing from your dev environment. ... View more

@anoop89I never used file-identity-provider so I am not really experienced with that 😞 would it be possible to provide a short snippet from conf/login-credentials/xml? You can remove all the PII data and replace them with something dummy, but I would really like to see how the file is structured and try to reproduce the behavior on my local device. Was this file generated automatically or have you created manually and kept on using on your prod/dev instances? PS: are you using the NiFi Cloudera version? ... View more

Hi @deepak123 , what do you mean by Nifi is performing slow? Based on your question, it is not very clear in which point you encountering performance issue. When waiting for an answer from your InvokeHTTP endpoint or when doing some actions on the result of the API call? ... View more

hi @anoop89 I can confirm you that version 1.19.1, 1.20.1 work very well without ldap or kerberos. I have installed two clusters, one in which there is no security active (cluster not secure) and one in which I have only activated the login with a single user and password. But here I think it mostly depends on the version (the open-source, the Cloudera version, etc) you are ussing. What I can tell from your logs is that you might have defined a false class for your login identify provider. By default, when I have unzipped the NiFi ZIP File, the nifi.properties file contained the following lines:     nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml nifi.security.user.login.identity.provider=single-user-provider     The login-identify-providers.xml is defined as seen below, but you have two other options which are commented: LDAP(<identifier>ldap-provider</identifier>) and KERBEROS (<identifier>kerberos-provider</identifier>)     <provider> <identifier>single-user-provider</identifier> <class>org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider</class> <property name="Username"/> <property name="Password"/> </provider>     Maybe you are trying to use the option file-provider from within the authorizers.xml file, which comes by default as commented and it is not recognized when starting NiFi? I think that your best solution here would be to compare the configuration files from your Dev Environment with the configuration files from your PROD Environment. By doing that you will identify where you defined the wrong property and you can correct it straight away. ... View more

hi @moahmedhassaan, it would really help if would provide more details about your flow, even how you query your data. Having and testing a MongoDB is not easy for everybody because there are not lots of people who have it available. Nevertheless, there might a solution, not very efficient but it might do your thing: Add a GenerateFlowFile Processor, where you configure a property like : ${now():format("yyyy-MM-dd HH:mm")}:${now():format("ss"):minus(1)}. Set this processor to run only on the primary node so you won't have to many generated files.   Send the success queue to your GetMongo Processor. Within the GetMongo Processor, in the query property you write your query with the condition on transactiondate > The_property_defined_in_Generate_Flow_File.   Again this is not a very bright solution, but it could suite your needs until somebody with more experience can provide you with a solution 😊 ... View more