Ask the Community

Hello,I have got a JXM exporter configured for Kafka brokers to send metrics to prometheus which allows me to monitor Kafka via Grafana Dashboards. All is working okay, however, stuck with a weird issue.Backaground:JMX exporter is configured using below property for Kafka brokers:in Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) have added a key KAFKA_JMX_OPTS and its value as -javaagent:/var/lib/prometheus_jmx_config/jmx_prometheus_javaagent-0.20.0.jar=9091:/var/lib/prometheus_jmx_config/kafka_jmx_exporter.yml.The jar and yml files provided in the configuration are in place and all works okay. We are able to get required metrics to Grafana via Prometheus.Issue:Whenever we attempt to restart Kafka brokers, it does happily, however, when attempted a rolling restart it fails with below error in stderr log file: + [[ healthy partitions stay healthy == *\p\a\r\t\i\t\i\o\n\s\ \s\t\a\y\ \h\e\a\l\t\h\y* ]]+ call_kafka_topics --at-min-isr-partit

Hello,I have setup NiFI 2.1.0, and am trying to figure out how to allow users in a NiFi Microsoft AD Group to authenticate to the GUI.  I have it setup so users can log in if I add their LDAP account.  But, I'd rather just have users in the group I created in AD be able to login instead of having to go in and add new users to NiFi each time.Here are the configs I have setup for my authorizers.xml: <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Initial User Identity 1">ldapuser</property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authenticat

I'm currently trying to get Nifi running in docker in the insecure mode without tls enabled. The reason for this is I want to run Nifi with ssl termination upstream and not have to deal with the complications of having to manage truststores and keystores.By default it seems that even configuring the environment variables for `nifi_web_http_port` and `nifi_web_http_host` will not remove the default configuration for the https port and host.I've modified the start.sh script for the container to set the http_port and http_host to be the expected values and removing the default values for https_port and https_host. I've also set the configuration `nifi.remote.input.secure` to be false. Now on startup Nifi failing to load a Key Store at ./conf/keystore.p12. Even with tls seemingly disabled by the means I've found necessary from all other guides on the matter Nifi is still expecting a keystore?Has anyone had success with running nifi 2.0 insecure in docker?

Hello Team,We are planning to upgrade our use of NiFi from 1.x to 2.x, and we are looking very much forward to try the new version.In NiFi 1.x we are using the "file-manager.sh" script in the toolkit for making backup and restore. It appears that this file is no longer present in NiFi 2.x.1) Has this file been removed from NiFi 2.x permanently, or do you know of any plans for adding it to NiFi 2.x in a future release? 2) If the file is not coming to NiFi 2.x will there be any replacement file for making backup/restore, or should we collect files for backup ourselves such as flow.json.gz?Best Regards

Hi TeamWe recently upgraded our AKS cluster from 1.28.5 to 1.29.4 . The apache nifi - version 1.16.3 was already installed in it and running successfully.  After the upgrade to 1.29.4 , one of the Pod - Nifi was continuously restarting and goes to Crashloopbackoff. When we investigate, the server container keeps restarting and there were no logs showing any error as it is changing status to Running and then to Not ready  very quickly and pod reports Crashloopbackoff. We are also unable to login inside the pod due to this. At times when the status is running , it is allowing to connect for few seconds and we noticed that the KeyStore.jks and truststore,jks from certmanager is not present within tls folder. We are not sure what is the cause of the continuous restart of the server container after upgrade of aks cluster to 1.29.4when we compared the logs of the nifi-0 and nifi-1 pod, the nifi -0 pod showing below log and running successfully. Whereas nifi-1 and nifi-2 do not

Hi,We want to stop Cloudera environment. After we stopped it via Management Console, we discovered there are liftie-***-liftie-infra-NodeGroup instances still running on AWS. Liftie Nodes are part of Scaling Group.Is it safe to stop those instances by hand via AWS console?We would be able to start them also by hand and later start Cloudera env.

Hello! I'm trying to convert float values on my flow files into integers. My Jolt Specification looks like this:[{"operation": "modify-default-beta","spec": {"progress": "=toInteger","setup_value": "=toDouble","proposed_setup_cost": "=toDouble","real_setup_cost": "=toDouble","invoiced_value": "=toDouble","last_recognized_value": "=toDouble","starting_date": ["=toDateTime", "yyyy-MM-dd HH:mm:ss", "coerceOrNull"],"last_progress_change": ["=toDateTime", "yyyy-MM-dd HH:mm:ss", "coerceOrNull"],"end_date": ["=toDateTime", "yyyy-MM-dd HH:mm:ss", "coerceOrNull"]}}] But somehow my "progress" value is still being outputted like a float. If I have the value 100.0, it is not transformed to 100, for example. I also tried using "=toInteger(@(1,value))", but that also didn't worked.

I am trying to integrate NIFI with Solace event broker and have created JMSConnectionFactoryProvidor service, which is enabled and running. However my JMSConsume processor gives this error - ConsumeJMS failed to process session due to Host not specified;Below are the configs for my FactoryService and processor:What am I missing in the configuration?Thanks.

Hi, all,To improve the throughput of nifi, I enlarged my standalone nifi to a three nodes cluster. And, configured my data flow accordingly, now it works but the load balance doesn't work as expected.The load balancing is configured as round-robin, and it seems that all flow files are dispatched to three nodes at the beginning. However, later I noticed that the cluster slowed down dramatically, with only one node being processing and the other two running nothing. However, I think the unfinished flow files in the queue should be dispatched to other nodes.192.168.1.200 is the master node and the slowest node in the cluster. My nifi version is 1.12.1,  is it the expected behavior of round robin on this version?I want to maximize throughput, please tell me how I can do that.

Hi, someone have passed certification exam of CDP 4001?I do the exam once but i didn't pass (58% of score 😞 ). Someone can suggest me a good material for preparation?Thank you

Hello , i am trying to use sftp with nifi . this is my docker image :  volumes:nifi-conf: services:nifi:image: apache/nifi:latestcontainer_name: nifiports:- "8089:8443"- "5656:5656"volumes:- nifi-conf:/opt/nifi/nifi-current/confenvironment:NIFI_WEB_PROXY_HOST: localhost:8089SINGLE_USER_CREDENTIALS_USERNAME: adminSINGLE_USER_CREDENTIALS_PASSWORD:  sftp:image: atmoz/sftpvolumes:- ./sftp/upload:/home/foo/ports:- "2222:22"command: foo:pass:1001 at first the found the sftplist running with no output and when i stop it i get 2 active threads and canot stop or start it just terminal   when the docker image load and debug it i founds that the owner of /home/foo not foo it is root and when i change it manually i get bad ownership error of chroot directory  

Hello, Im currently setting new cluster.  When I try to start some role I get notifications that: Role is missing Kerberos keytab. Go to the Kerberos Credentials page and click the Generate Missing Credentials button.So I try to Generate Missing credentials and I got: failed script:/opt/cloudera/cm/bin/merge_credentials.sh due to:/opt/cloudera/cm/bin/merge_credentials.sh failed with exit code 1 and output of <<+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf-merged-out15742810764060677617.keytab+ shift+ ktutil+ chmod 600 /var/run/cloudera-scm-server/cmf-merged-out15742810764060677617.keytabchmod: cannot access '/var/run/cloudera-scm-server/cmf-merged-out15742810764060

Currently I am facing issue in data stored in HBase where the old data got updated with recent timestamp and majorly columns being null which is partial record even though there was no update transaction received or any update operation performed in Hadoop.Need help to understand in which all scenarios this is can happen or anyone who faced this kind of issue?

Hello,I want to upgrade the NiFi version for my 3 nodes from 1.22.0 to 1.28.1.my question is: can i do the upgrade directly without going through intermediate versions? are there any recommendations?thank you for your feedback in advance

Hello:How to use HAProxy to connect for Kafka with Kerberos authentication?I have three kafka brokers, and i try to use haproxy in front of kafka, but kerberos authenticated failedMy haproxy.conflisten kafka bind *:6677 mode tcp balance roundrobin server kafka1 kafka-1.kafka.net:6668 check server kafka2 kafka-2.kafka.net:6669 check server kafka3 kafka-3.kafka.net:6666 checkI also modifiedkafka1 server.propertiesadvertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668listeners=INTERNAL://:6667,LB://:6668listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXTinter.broker.listener.name=INTERNALlistener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET"kafka2 server.propertiesadvertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6669listeners=INTERNAL:/

We have been using nifi with anonymous enabled with initial development.Now the requirement is to disable anonymous and have user based authorization / authentication.To start with, we want to disable anonymous and have single-user-authorizer.Manage to create Administrator user with the set user script. Able to login with Administrator. But the anonymous is still enabled. Code created user Administrator is still visible with anonymous user. Update nifi.properties with false flag as shared below. But still we are able to access without credentails. nifi.security.user.authorizer=single-user-authorizernifi.security.allow.anonymous.authentication=falsenifi.security.user.login.identity.provider=single-user-providernifi.security.user.jws.key.rotation.period=PT1Hnifi.security.ocsp.responder.url=nifi.security.ocsp.responder.certificate=

I am trying to execute a PowerShell script in java windows minifi workflow using the ExecuteStreamCommand Processor (https://community.cloudera.com/t5/Support-Questions/How-to-execute-a-shell-script-while-using-NiFi-in-Windows/td-p/153638)The script is working when executed manually in the PC but does not work in the workflow. The powershell script checks the current user and if the workflow user is an admin and writes the output to a file.The properties of the ExecuteStreamCommand Processor are as follows :Command Path : C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exeCommand Arguments : -File C:\\scripts\\AdminCheck.ps1 -OutputFile C:\\Path\\adminOutput.txtThe command that works when executed on the PC :-C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe File C:\\scripts\\AdminCheck.ps1 -OutputFile C:\\Path\\adminOutput.txtThis is the powershell script :-param ([string]$OutputFile = "C:\Path\adminOutput.txt" )# Get the current user$C

I use CDP Private Cloud Base 7.1.7 and just enabled Kerberos security. I followed the setup documentation but could not proceed further than this step <https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-kerberos-authentication/topics/cm-security-kerberos-enabling-step7-prepare-cluster.html>.In short, I lost "supergroup" access to hdfs. Here are details.* I created an AD account mysuperuser@example.com and an AD group mysupergroup@example.com.* After Kerberos is enabled, I changed dfs.permissions.superusergroup=mysupergroup, and restarted the cluster. Certainly,  "mysupergroup" and "mysuperuser" do not exist anywhere in Hdfs POSIX permission settings.* I kinited mysuperuser@example.com, but got hdfs permission denied error. It looks like that Kerberos could not understand AD groups associated with the kinited account.* Then I changed dfs.permissions.superusergroup=mysuperuser, restarted all services, but still got permission denied error.I intended to use Ranger

Hi All, I want to find out what are the roles / groups assigned to a DB / Schema in Hive, how I can get those details ? please suggest. Thank you. Regards,PK

Hi, I've been trying to get this to work, but am not sure how to complete it.From a previous SplitJson I have a queue of Flowfiles in these two formats.{"s_id": 1000,"parent": 0,"items": {       "2020": {                "s_id": 2020,                 "parent": 1000        },        "2021": {                "s_id": 2021,                "parent": 1000        }  }}and{"s_id": 1001,"parent": 0}I would like to un-nest them further to the Flowfiles.{"s_id": 1000,"parent": 0}{"s_id": 2020,"parent": 1000}{"s_id": 2021,"parent": 1000}{"s_id": 1001,"parent": 0}I've tried another SplitJson with a JsonPath expression of $.items[*], however this throws a failure for the json where "items" doesn't exist.Can someone please guide me on the steps and process

In Apache Atlas, I am trying to model the data flow of different processes. The issue I am having is that some of these processes share common DataSets but I don't necessarily want the different processes I am modeling to appear to be connected to each other.For example, in this lineage model, I want to show that there is an input of an XML Data source file into a process that outputs and transferred to another computer.{"entity": {"typeName": "datasystem_datatransfer","attributes": {"id":"b75af137-9279-4c73-be9f-0e37b686dde5","qualifiedName": "b75af137-9279-4c73-be9f-0e37b686dde5@datasystem_datatransfer","displayName": "Data Transfer Use Case 1","inputs": [{"uniqueAttributes":{"qualifiedName": "25b60fe5-891c-4c94-87ab-b075d838ec30@datasystem_datasource"},"typeName": "datasystem_datasource"}],"outputs": [{"uniqueAttributes":{"qualifiedName": "21781e1b-4b94-435b-be0a-141776267c4e@datasystem_computer"},"typeName": "datasystem_computer"}],"description": "Data transfer from Data Source to

안녕하세요 CML에서 nexus 로컬 레지스터리를 통해 docker image업로드시   아래와 같이 에러나가서daemon.json 을 수정하여 처리하였으나, cml서버에서 nexus로 image push할때는 에러가 나지 않으나, cml에서 add runtime할때에는 에러가 발생합니다조언 부탁드립니다감사합니다 #에러내용 cml could not fetch the image metadata http:server gave resopnse to HTTPS client

Hi,can you tell me if it is possible to change the owner of a project in cdsw? And how?  Many thanks

Good morning,Is there a way to automate the creation of users, groups, and/or policies in NiFi? For example, I have configured LDAP authentication in NiFi -- is there a way to manage not only authentication but also authorization? It is not necessary for this to be LDAP-based.I was considering the possibility of using a script with the API, but I wanted to know if there is an "out-of-the-box" solution available.Thank you.

I'm working in with nifi to grab parquet files from a S3 bucket. But when I read in the parquet files the arrays in the data end up with the following format: [ { "id": 1, "name": "John", "address": { "street": "Main St", "city": "New York" }, "hobbies": [ { "element": "coding" }, { "element": "music" } ], "greetings": [ { "element": { "intro": "hello", "end": "bye" } }, { "element": { "intro": "hola", "end": "adios" } } ], "gender": [ { "element": "M" } ], "record_id": [ { "element": "2a2c6c86947719eacc1742adf1d6f2c7" } ] }] Instead of the des