@georg ,   There's not much info in your description for me to give you a precise answer. But, given you mentioned "container", I'm assuming that the other system is running on a Docker container in the same host as NiFi is running.   If that's the case, you can use "docker inspect" to find out what the exposed port of the container is and then your URL will be something like either "http://localhost:<port>" or "http://host.example.com:<port>". If TLS is being used, replace "http" with "https".   I hope this helps. If not, please provide more details information about the components you're talking about.   Cheers, André   ... View more

Thanks for the link to the video, @MartinTerreni .   In a traditional NiFi flow that read from Kafka and writes to Kafka, the offsets of the source topic are indeed stored by the consumer in Kafka itself. With this, if NiFi crashes or restarts, the flow will continue to read the source topic from the last offset stored in Kafka.   The problem, which @mpayne explained in the video, is that there is a decoupling between the consumer and the producer in a traditional flow. And this can cause data loss or data duplication, depending on the scenario.   For example, the ConsumeKafka processor commits offsets to the source Kafka cluster in batches at regular intervals. It is possible that some messages read by the consumer are written by the producer to the destination topic before the offsets of those records are committed to the source Kafka cluster. If the flow stops abruptly before the commit happens, when it starts again it will start reading from the previously committed offset and it will write duplicate messages to the target topic.   On the other hand, since there's no tight coupling between consumer and producer, the consumer could read some messages and commit their offsets to the source cluster before the ProduceKafka processor is able to deliver those messages to the target cluster. If there's a crash before those messages are sent and some of the flowfiles are lost (e.g. on node of the cluster burned down) that data will never be read again by the consumer and there will be a data gap at the destination.   The new feature explained in the video address all of these issues to guarantee Exactly Once Delivery semantics from Kafka to Kafka.   I hope this helps clarifying it a bit more 🙂 Cheers, André   ... View more

@wichovalde ,   The error below is a server-side error that should (hopefully) be logged in the Atlas server log. Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: https://<atlas-server>:31443/api/atlas/v2/entity/uniqueAttribute/type/hive_db?attr%3AqualifiedName=inventarios%40cm&ignoreRelationships=true&minExtInfo=true&user.name=superuser, status: 500, message: Server Error   Try to find the entries in the Atlas server log that match this call and it could tell you a bit more about the problem.  If the Atlas log doesn't seem to have the corresponding information, try setting its log threshold to DEBUG in Cloudera Manager and restarting the service and repeating the test.   André  ... View more

Hi @drgenious ,   When you open the Oozie Editor, notice that the default line of icons shown at the top of the editor are "Documents" rather than "Actions". Documents enable to embed content you created previously in your Oozie workflows (saved Hive queries, Spark programs, Java programs, etc.). If you click on the "DOCUMENTS" word, you will see a drop-down with the "Actions" selection. Clicking on it will switch the line of Document icons with the Action icons you're looking for: HTH, André ... View more

The configuration of the ElasticSearchLookupService Controler Service is pretty straight forward as you have noticed. It uses another controller service, a ElasticSearchClientServiceImpl service, to connect to the Elastic service.   Most of the connectivity details need to be specified on the ElasticSearchClientServiceImpl service configuration.   The best place for reference is the documentation: ElasticSearchLookupService ElasticSearchClientServiceImpl   Cheers, André ... View more

@ABBI @yamaga ,   The lookup service require that the lookup key be unique for it to work correctly. If there are duplicates you can chose to ignore them, but only one value will ever be returned.   One thing you can do is to consolidate records with the same key under the same line, so all the values will be returned and you can then deal with it (e.g. split the values)  in the NiFi flow.   In your example, we could change the lookup files like below:   col2,col5,col6,col7,col8 2,abc,China,123,col8 4,def,USA,118,col8 8,"qwe,zyx","Canada,England","118,118","col8,col8" Regards, André ... View more

Can you share your code? ... View more