Q&A

Hi everyone, I have a problem because I dont know how I can access are all variables (see image) from a script groovy.Maybe someone want to share some idea in order to solution my problemRegards. 

Hello,I am trying to extract value from a key/value pair json record. I am trying to extract the value of "image" key.  I used an ExtractText processor but there was no match. Although, there was a match when i used an online regex extractor(https://onlinetexttools.com/extract-regex-matches-from-text). Regex i used in the extracttext processor: (?<=\"image\"\s:\s")[A-Z-a-z-0-9\-\:\/\.\_]+ My json record:{ "id" : "03ee73b8-a553-4575-8dfa-d0da4e7939e9", "url" : "https://ll.thespacedevs.com/2.0.0/launch/03ee73b8-a553-4575-8dfa-d0da4e7939e9/", "launch_library_id" : null, "slug" : "falcon-9-block-5-galaxy-33-34", "name" : "Falcon 9 Block 5 | Galaxy 33 & 34", "status" : { "id" : 2, "name" : "TBD" }, "net" : "2022-10-05T23:07:00Z", "window_end" : "2022-10-06T00:14:00Z", "window_start" : "2022-10-05T23:07:00Z", "inhold" : false, "tbdtime" : false, "tbddate" : false, "probability" : null, "holdreason" : "", "failreason" : "", "hashtag" : null,

If I have a flow file with the following JSON how can I1) Evaluate "addresses" to determine if it is of type Array or type Map2) If type Map then convert "addresses" into an array using native NiFi capabilities (i.e. no string parsing)?  { "name": "John Doe", "addresses": { "work": { "number": "123", "street": "5th Avenue", "city": "New York", "state": "NY", "zip": "10020" } }}   This is what I need it to look like:  { "name": "John Doe", "addresses": [{ "work": { "number": "123", "street": "5th Avenue", "city": "New York", "state": "NY", "zip": "10020" } }]}   I appreciate the input and support!  Thank you.

Hi there, does Cloudera support Iceberg in on-prem deployments? If not, by when it's expected to be available?

Hi Team, We are running some Hive, and Impala queries and getting the below error for the job. It was running fine earlier, but now it's failing. We tried multiple ways, like re-created the tables and then tested with the REFRESH command for the table. Whenever the job executes then automatically permission is changed with the user. It was like impala: hiveCould you please suggest how we can w take it forward? Note: We recently upgraded the cluster to CDP.

Hello I am trying to make a data flow through Nifi which are listFile -> fetchFile -> data transformation -> putKudu However, I want to somehow pause the fetchFile processor or hold on to the flowfiles received from the local server until a previous flowfile completely goes through the downstream, after putKudu processor, then release the next flowfile one by one.I looked at Wait and Notify processors but could not find a way to notify the next flowfile in the queue to be released. Is it possible to apply this logic in the Nifi without using custom processor? Any help is appreciated.

Hello, a new CVE appears on Apache Spark. Does it impact every versions of Spark ?https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-33891Thanks in advance for your help.

how to know why sparkcontext was shut down ??ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.spark.SparkException: Job 1 cancelled because SparkContext was shut down

Hi, I have lots of generated hprof files in /tmp. 1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid98619.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid97929.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid96759.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid92257.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid9035.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid87683.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid87640.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid86040.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid82228.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid77347.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid77317.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid75484.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid72638.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid71811.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid7086.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid66069.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid65902.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid65186.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid61886.hprof1.3G /tmp/m

How to uninstall NiFi from Linux 7.9? The process org.apache.nifi.registry.NiFiRegistry is up and running on the server. We need to remove the product completely. I am not sure how it was installed.We do not use d2iq dcos framework.  

Hello everyoneI am using PutSql processor and I need to execute a Procedure from the Oracle database, I has used the follow code in the SQL Statement:declareinputvar varchar2(20) := "1";outputvar varchar2(20) ;beginMyPKG.MyStProcedure( inputvar, outputvar);end;My questions are: 1) How can I recieve the input parameter and pass to it my code? and 2) How can I send the output parameter to next queue o some attribute ?

I need to send an attachment by email through Nifi, but I can't get the PDF file because there are several files from several people within the same directory, I have a BASE64_ENCODE field in the Oracle table that is the same as the generated PDF file. How can I pass this information from this field through Nifi's PutEmail processor?    

Hi All, I have been trying to using group by with having clause in the ExecuteSQL processor but it doesn't seem to be returning the count. The SQL query looks like : select id_num, count(*) from xyz where id_num<>0group by id_numhaving count(*)>1 I put this in the SQL select query. I just get an empty array. How do I use group by and having in the select query in Nifi? A normal query without these clauses works perfectly but I need a count to check for duplicates in the database. Thank you in Advance

Hi,    I am trying to secure NiFi authentication using LDAP configuration. ). I am able to get to the login screen, but then I am getting an error on logging in ( The supplied username and password are invalid) even though the username and password are correct.Can someone help me in the direction, on what might be the problem?  

hello cloudera community, we are having a problem accessing the hive cli of a certain user and the spark-shell too when executing the query "show databases" in hive cli returns the error: "FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset " when executing the "show databases" query in spark-shell returns the error: "org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException;" "WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.org.apache.thrift.transport.TTransportException" when we use beeline and run the query "show databases" it works without problem could you help us with this problem? we are using cl

Hi there, In Cloudera manager, after installing the parcels of CDH, and CFM.added the Nifi service by installing nifi csd files in csd repo.and zookeeper is also added as dependency. After starting the NIFI service, NIFI node status is Down all the time.And couldn't find any folders like log, conf in the parcel-repo to check the error logs. Nifi node status is unknown health. couldn't start the service. We have hosted the services on a single host.Please, suggest a way to identify the issue. Below is the directory structure of CFM parcel and nifi files. 

Hi I am using the cloudera hortonworks sandbox docker image, and have followed this tutorial to run  Jupyter notebooks: https://community.cloudera.com/t5/Support-Questions/Installing-Jupyter-on-sandbox/td-p/201683 This works. The notebook is started using the python kernal. The error is encountered when attempting to create the spark context: FileNotFoundError: [Errno 2] No such file or directory: '/usr/hdp/current/spark-client/./bin/spark-submit'  FileNotFoundError Traceback (most recent call last)<ipython-input-4-fbb9eeb69493> in <module>----> 1 spark = SparkSession.builder.master("local").appName("myApp").getOrCreate()/usr/local/lib/python3.6/site-packages/pyspark/sql/session.py in getOrCreate(self) 226 sparkConf.set(key, value) 227 # This SparkContext may be an existing one.--> 228 sc = SparkContext.getOrCreate(sparkConf) 229

Hi, I want to ask a few questions about having multiple Hive Mestastore instances and how to correctly configure the compaction initiator and cleaner threads on these. The point of having multiple Hive Metastore instances is because we have many concurrent connections and, as we have seen on the official documentation (https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-hive.html), it is not recommended to have more than 16-24GB of Java Heap due to the impact of Java garbage collection on active processing by the service. 16-24GB of Java Heap is the recommended amount of memory to manage 41-80 concurrent connections (we have around 50 on each Metastore). So, what i want to ask is what is the correct configuration for the compactor initiator and cleaner threads (hive.compactor.initiator.on) if we have multiple Hive Metastore instances. Can it be enabled on every instance? Should it be only enabled on 1 instance? Our environment is ru

Hi folks,I have a problem on my CDP Datalake causing failure on master node.Apparently some services fail due to lack of free memory space.   Everything seems to work correctly but I don't know what can happen if the free space reduces further.I have tried these solutions, however, with no effect if not momentary:- lowering some alarm thresholds (from 10% to 7% of free space)- removing some log files from the machine in /var/log and /tmp folders @shehbazk Can you give me some advice?And I would also like to understand if it is possible and how to increase the capacity higher than the current 236 GB. @pandav  

Hello all,   I am a student of Purdue Global University.  As part of several classes, we have been instructed to utilize Oracle VM VirtualBox Manager, alongside an installation of Cloudera-Quickstart-VM-5.13.0-0-Virtualbox. Last semester this installation worked throughout my class just fine.  Then, this semester a class requires it as well, but when I went to boot the original installation I was surprised to find it won't boot.  I am running into what I think is a Kernel Panic error on boot.  I have tried removing the Cloudera instance and reinstalling it using the PDF guide provided to us by the school.  However, on reinstall, the Kernel boot error remained.  I have also tried shutting down and restarting my computer, as well as changing the chipset setting from PIIX3 to ICH9, as suggested in another thread.  I have a video showing a screen capture of what I receive on boot, as well as the logs, and the PDF we use for installation.   <L

Hi All, I need to download my old certofocation from cloudera that was completed in 2020 but I cant find any resource on how to. Can someone please help with this

Hi,  I am trying to secure NiFi using LDAP configuration. I followed blog by mintops and pvillard articles for reference. I am running NiFi on windows (not in cluster configuration). I am able to get to the login screen, but then I am getting an error on logging in ( The supplied username and password are invalid)            I followed most instructions as mentioned in the blog by mintops. Can someone help me in the direction, on what might be the problem?PS: NiFi version and toolkit versions -1.8.0.I haven't created certificates, just the configs needed to update in Keystore and truststore passwords by the toolkit.The LDAP configurations were already being used in another program, so that is also not the issue. login-identity-provider.xmlauthorizers.xml:        nifi.properties              &nb

I am working on a proof of concept situation with a Nifi Cluster on Openshift. As part of the PoC I want to enable authentication (currently have a working cluster with no auth), so I am wondering if it possible to do it with the Single User method as it is built in and should be the easiest to setup. I have been unable to find any documentation outlining this process or possibility, so if there is any outlining this setup, assuming it is possible, please direct me towards it.

I have a Handle HTTP Request Processor to receive incoming requests , which have various workflows connected to it, I want to apply authentication on this processor for incoming requests, I have figured out that  I need to set "Need Authentication" and configure a secure socket layer  Context service, But I have no idea how secure socket layer works,what are key stores and trust stores ? how to generate them and how to use them, Do the client needs to send certificates ? how do I verify it ? I intend to test it by sending request to this processor using Postman.Any help is appreciated, Thank You.

"message":"Could not login: the Client is being asked for a password, but the ZooKeeper Client code does not currently support obtaining a password from the user. Make sure that the Client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the Client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper Client using the command 'kinit <princ>' (where <princ> is the name of the Client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this Client. If you continue to see this message after manually ref