Q&A

Hi Team, Am using self signed certificates, for API calling but while import the configuration passwords are not stick with the properties, can someone please suggest a method, How we can add truststore and keystore password using NiFi CLi or via API call to SSL ContextService.   

Dear all,I am trying to load a PySpark Dataframe into an Impala table, using the jdbc connector. However, the df.write statement fails, because the "Create table" - statement that is generated contains quotation marks for the column names:Do you have any idea how to get rid of these quotation marks? If not, what would be a different approach to load a dataframe into an Impala table?I also tried spark.sql('select identifier_id as identifier from tempView').write.jdbc(...), but here I am getting the error "File /tmp/hive does not exist". Thanks a lot in advance for any help! 

Hbase is failing with below error . error Connection failed: [Errno 111] Connection refused to master1:16000

We have configured both NiFi Settings Maximum Timer Driven to 80 & Maximum Event DrivenThread Count to 120. We have a 4 node cluster with 64 cores. On diagnostics we see only average usage of 5 cores. Additionally, the data is queuing up and processing slowly.Issue is The CPU utilization does not go beyond some percentage. What further configurations can help utilize the CPUs and increase the computation power.

Hi everybody,I need to install CDH at least 5.8 ( Virtual Box) and always face with this error: 403 Forbidden (varnish) the provided credentials were incorrectKnow any how to fix it? Many thanks,Mehdi

I have registered AD in Ranger Usersync and want to sync deleted user in ad. so, I am following the guide on the link (https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-ranger-authentication-unix-ldap-ad/topics/security-ranger-authentication-usersync-deleted-users.html) However, even if I delete the account from AD, the account does not change to the hidden state in Ranger. Is there anything else I need to configure besides what's in the document?

I have a stream in which I generate a json file with GenerateFlowFile, then I use Base64EncodeContent to encode the json file. I need to take the base64 encode result and concatenate it with a sentence, to finally move to invokehttp. I tried MergeContent for this and was not successful.I'm not able to find a way to get the result of the Base64EncodeContent encode and concatenate it with the sentence. I would like suggestions on how to carry out this flow.Below is the flow I drew:  

Hello there, I have flowfiles with json content, which I transform before using them to send a post request with InvokeHTTP.I have an attribute "url_value" which I can use to determine my "api_value". For now I use EvaluateJsonPath to add those values to the flowfile attributes, then I use UpdateAttribute where I have set some rules to get the correct api_value.Now my question: how do I get the value back into my content? If I use AttributesToJSON, it overrides all previous content, therefore I need to have extracted all necessary attributes with EvaluateJsonPath before. This is not a very elegant solution since there can be many entries and I actually only need to update one of them!Is there no processor similar to UpdateAttribute but for flowfile content? How could I use JoltTransformJSON to change a value depending on another attribute? Edit:I would need something like if url_value:contains("webpage1") set api_value to "wepage1"elseif url_value:contains("webpage2") set api

Most of the same Impala queries respond within 20 seconds. But sometimes it takes more than 60 seconds.I see a hotspot in 'SCAN HDFS' of 'ExecSummary' in slow query profile. (avgTime 6s462ms, maxTime 54s766ms ) low response time Query Info: - Duration : 13.1s - HDFS Scanner Average Read Throughput: 382.2MiB/s - Threads: Network Receive Wait Time: 10.0m high response time Query Info: - Duration : 1m, 4s - HDFS Scanner Average Read Throughput : 988.1 MiB/s - Threads: Network Receive Wait Time: 59.5m If HDFS SCAN is the cause of the slow response time, is the 'HDFS Scanner Average Read Throughput' value higher in the slow query? And one more question.I compared the fast and slow operators of the HDFS_SCAN_NODE stage in the slow query profile.RowBatchQueueGetWaitTime(54.76s) and ScannerThreadsTotalWallClockTime(54.77s) I wonder why the two metric values are high. HDFS_SCAN_NODE (id=0)Hdfs split stats (<volume id>:<# splits>

Good Morning I'm trying to access an address of another Linux server through Nifi that is also installed on a Linux server, to search for files. I'm trying to use the FetchFile Processor for this but I'm not getting the error "Routing to not found". Could you help me how to make this connection between one server and another?

Hi, We're experiencing a peculiar issue these past few days - our python scripts started failing when run with spark-submit. On first glance our logs show that the scripts are encountering syntax errors whenever we are using code related to modules, but further troubleshooting showed that in actuality the modules are the issue. When we comment out pieces of code that throw syntax errors, we instead receive import errors ("No module named..."). We have two versions of python on our cluster but it appears that spark-submit still is using the proper python version with all our modules installed on it. Our scripts run just fine through pyspark, for some reason however, spark-submit does not recognize the imported modules when we run scripts through it. What is more, YARN doesn't seem to recognize these jobs as failed, they are not logged at all, probably because they crash as soon as we start importing modules. So basically we do not have access to YARN logs for these jobs.&

How can I use the output of the Base64EncodeContent (or the Base64Encode function) as input for an XML template, where the flow file content (not an attribute) is the XML with the encoded value included?  There doesn't seem to be a NiFi reader that reads the encoded flow file content, so I'm not sure how best to use this processor aside from using a couple of ReplaceText processors to bracket XML substrings around it.I appreciate any input.  Thanks for reading.example:Initial Flow File: "The quick brown fox jumped over the lazy dog"Encoded Flow File: VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZWQgb3ZlciB0aGUgbGF6eSBkb2c= <?xml version="1.0" encoding="UTF-8" ?><a xmlns="some/name/space/goes/here"> <b>This is a template</b> <c>ENCODED DATA GOES HERE</c></a>  

Hello,on our system CDP 7.1.7, we have use konfiguration parameterkafka.properties_role_safety_valveadd set attributesasl.kerberos.principal.to.local.rulesto map ActiveDirectory principals to entities created in ranger.In our system, the AD user have a prefix e.g. xjohndoe@SAMPLE.COM maps to a ranger entity "johndoe"During a spark-submit (over yarn), we also need to pass a principal, however as there is no such mapping, we obtain an error saying the unix user "xjohndoe" does not exist. This is true indeed, we eed to map it to "johndoe". Ist there any possibility to map principals during spark-spark-submit possibly similarly to sasl.kerberos.principal.to.local.rules in kafka or any other possibility?Best regardsJaro

Hi peopleI am working with HttpInvoke processor with Post and I try to catch the response body using the propertie "Put Response Body in attribute", the processor execute well and it generate the response on the queue Response but it dosent generate the attribute. Someone has some idea where is my error?

Good afternoon, Is there a way to use the FetchFile 1.16.2 processor to access files from Windows when the Nifi application is on Linux?

I am trying to set partition retention times on existing Hive manged tables using the following:  ALTER TABLE <table name> SET TBLPROPERTIES ('discover.partitions'='true');ALTER TABLE <table name> SET TBLPROPERTIES ('partition.retention.period'='1d'); as stated on this page below, however I am still able to search partitions older than a day so it appears to not be working? It does mention on the page that this is for 'external' tables, can anyone let me know if this an 'age off' retention period is possible on managed tables? am I missing any commands etc? https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/using-hiveql/content/hive-set-partition-retention.html Thanks in advance

Hello,I have setup the Hadoop cluster on single server ( on my laptop). I have uploaded the simple text file data into Hive.I am running the select query :Select * from ratings;but this simple query is not giving me any result even after 10 minutes.All services of Hive are running.Status of Ambari :   my query : What should I do ? Thanks 

Hi everyone,I've trying to monitoring nifi api. In our nifi we have configure oidc login and it works fine.As monitoring, we using zabbix.The problem in zabbix is when try to check api page (https://nificluster.info/nifi-api/flow/cluster/summary) and I receive http 401 error. In the web scenario monitoring, I have configured 2 steps: connect to https://nificluster.info/nifi/  with login and password. This steps it's ok, the required string - "log out" - and required status code 200 works. In this step I've configured viariables username, password for logging and follow redirects flag enable.connect to api https://nificluster.info/nifi-api/flow/cluster/summary without any parameters except requird string with number of cluster nodes and required status code 200.Who could help me resolve this 401 error on the api nifi? 

Hello Everyone, We are triggering our Nifi processors (version 1.16.2 , LDAP integrated ) through Curl within cron job. Curl is invoking the processor using access token (as we moved from http call to https) Generating access token : curl https://$hostip:$port/nifi-api/access/token -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data 'username="${ldapusername}"&password="${ldappassword}"' '--compressed --insecure   Curl invoking Nifi processor using acces token stored in variable $nifi_token : curl -i -H 'Content-Type: application/json' -H 'Authorization:Bearer '$nifi_token -XPUT -d '{"id":"'${processorid }'","state":"STOPPED"}' https://$hostip:$port/nifi-api/flow/process-groups/$processorid '--insecure' But it is failing for token expiration : WWW-Authenticate: Bearer error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Expired JWT", error_uri="https://tools.ietf.org/html/rfc67

Is there a size limit to the tar file generated by MergeContent?ERRORMergeContent[id=8a06a034-0183-1000-b8b5-c045959a56e9] Failed to process bundle of 2 files due to entry size '14743306287' is too big ( > 8589934591 ).; rolling back sessions: java.lang.RuntimeException: entry size '14743306287' is too big ( > 8589934591 ).   

Hi, I want to save the out put of the file after replace the text.when i put xml file for consumed NIFI replace the text and move xml file into processed folder,NIFI didn't change into original xml file,i just want to know during process what is the code which NIFI use.

hello cloudera community,   we are having problems using livy with job spark to read hive by jupyter notebook   when we run a simple query, for example:   "spark.sql("show show databases").show()"   returns the error below"org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient"   Could you help us with this setup?   ps: we are using cdh 5.16.x

Hello, please help me.env  jdk8 + jce  HDP 3.1.5, ambari 3.7.5Kerberos is installed successfully. When enabling Ambari Kerberos, an exception is thrown:---------------------- ambari-server.log --------------------2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Containers per node - cluster[containers]: 42022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Ram per containers before normalization - cluster[ramPerContainer]: 33282022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Min container size - cluster[yarnMinContainerSize]: 10242022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for map - cluster[mapMemory]: 30722022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for reduce - cluster[reduceMemory]: 30722022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationCluster

I used nifi 1.13.2 cluster of 2 nodes.Node1 is rhel7, node2 is redos, embedded Zookeeper.The cluster works correctly (nodes see each other, the coordinator is selected.)Certificates on nodes are self-signed.Faced with the problem of authorization in the UI.LDAP authorization does not work (without cluster the LDAP authorization worked on both nodes).I am trying to connect to node1.When I enter my username and password, I return back to the authorization page.The log on the node1:2022-09-19 05:36:14,892 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://node1.domain.ru:8080/nifi-api/flow/current-user (source ip: <IP node1>)2022-09-19 05:36:14,895 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <user name>2022-09-19 05:36:14,895 DEBUG [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>2022-09-19 05:36:14,895 DEBUG [Ni

I would like to know how is the best way to create surrogate key or SK in Impala. If there is no UDF function SURROGATE_KEY(). How could I do?