Thank you for your response cotopaul. In MQTT 5 there is a feature for Request-Response, in that feature there is a field setting to determine the response topic (different from the actual topic). The response topic field represents the topics on which the responses from the receivers of the message are expected. I was wondering if maybe NiFi could get the response topic field value from the MQTT as an attribute ... View more

@SAMSAL  - it worked ! Thanks ... View more

@cotopaul Please go through my latest response before concluding its a hardware or setup issue. FYI, Nifi is working fine for more than a week for me since I removed the ExecuteScript Processor. That's the only change I did and I replicated the issue several times before posting here. Nifi has restarted several times since then without any issues.  could you please care to explain what sort of hardware issue it could be that it affects only ExecuteScript processor running python code?      ... View more

Thank you very much! This gives me confidence that my attempts are not going in the wrong direction.I already use ExecuteStreamCommand Processors excessively and it is a pleasure to use them here as well. ... View more

Thank you for replying.  I have tried so many different ports and I am even trying external zookeeper.  None of my nifi's can connect to any port I provide them.  It's almost like something is wrong with the code.  I have installed ZK on one of my actual nifi servers and that starts up immediately on port 2181.  That's what is leading me to think it's something in the code.  The other odd thing I keep seeing is, apache.zookeeper.ClientCnxnSocketNetty future isn't success. ... View more

@sarithe, What is the format of the record you are trying to convert? What is the data type of the field you are trying to convert? Are you trying to modify the value of a record or the value of an attribute? If you are trying to update the values from within each record, you should try using an UpdateRecord processor, in which you define a Record Writer and a Record Reader. Next, you can add another property in your processor and define it like: Property: /your_column --> pay attention to the slash in front as it is very important. Value: ${field.value:multiply(1000):toNumber():toDate("yyyyMMdd", "GMT"):format("yyyyMMdd")} --> this is just an example, as I do not know how your data looks like and how you want it displayed. You can use NiFi's EL to define the exact format you required, but make sure you use field.value if you want to modify the value from within that specific column. ... View more

@ushasri, If you are reading an Excel File, you can use an ConvertRecord processor, in which you define an Record Reader (CSVReader most likely) where you tell NiFi how to read the value (value separator). Next, with the help of a Record Writer (I assume CSV again or maybe Avro) you define how your data should look like, including the header. For both the Reader and the Writer you will have to define a schema, where you can do the replace of blank space with underline. ... View more

@acasta, Don't get me wrong, but I highly doubt that NiFi is ignoring/deleting somehow the files, without your intervention or configurations. What I would suggest you are the following two actions: - First of all, add an LogMessage/LogAttribute after your have unzipped all those files. Basically, double your success queue from your processors where you unzip your tar file and log each file which was extracted. In this way, you get a list with all the files extracted out of your zip file. Make sure to set the Queue as single node to check nifi-app.logs on a single node. - Next, add another LogMessage/LogAttribute after your processor with which you save the data into your Bucket. Send the name of the unzipped files into the logs to get a list with all the files which have saved into your bucket.Make sure to set the Queue as single node to check nifi-app.logs on a single node. Afterwards, you can compare the lists and see if you have extracted and saved all your files. If the lists are 1:1, it means that the problem is not related to NiFi itself, but to something else: like another system doing something in your bucket, having files with the same name which get over written, etc. Another option would be to use DEBUG on all your processors and use RUN ONCE until you process everything you have to process and analyze in real time what is happening. ... View more

Hi @cotopaul  Thanks for your reply. I am familiar with Jolt (a json to json transformation library). I have been thinking to add the required padding using the funmctions in Jolt and then use the FreeFormTextRecordSetWriter controller service. This service take the name of the key in Json and prepare the file containing only the value. It also keeps the padding added in the previous Jolt Transform. I think using 10 UpdateAttribute will be tough and I have multiple fields that need the required padding/empty spaces. Thank you for your answers! ... View more

@SandyClouds, I do not have a template because I no longer have access to that project but I have provided you will all the info you need to develop your own system 🙂   And it mostly depends on your use case. You can use any API you require for your use case. But in terms of processors your require the following: InvokeHTTP to perform the API Call, EvaluateJSONPath + SplitJSON to extract the relevant lines out of your bulletin boards and RouteOnAttribute to identify the errors you need. Other than that, your imagination is your best friend. ... View more