Q&A

Hello Cloudera Community, We have a use case where we cannot afford the restart of catalogd, as the reload of metadata takes many hours and queries are impacted during this long reload. Is it possible to have multiple catalogd instances with an active / passive setup? In this case the active catalogd would act as the main catalogd and in case of failure, one of the passive catalogds would take the lead becoming active. This would eliminate the huge metadata reload time in case of failure in the catalogd node.  It would be really interesting to have the feedback of the community about such setup, and see if someone has already tested it. Thanks a lot for your replies 

Cluster manager looks fineUbuntu 20.04, bare metal host. No SElinux, no firewallThe agent log saysBEGIN sudo dpkg -l openjdk8 | grep -E '^ii[[:space:]]*openjdk8[[:space:]]*'dpkg-query: no packages found matching openjdk8END (1)BEGIN sudo apt-cache show openjdk8E: No packages found___________If you skip JDK installation in the wizard, it fails withBEGIN sudo dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*'dpkg-query: no packages found matching cloudera-manager-agent Any clues?

I need to encrypt some content with PGP, but i can't use the processor Encrypt Content because we require that the "Keyring file route" and the "passphrase" must be entered through attributes (We must use an old version of NiFi). I make this script using the repo of NiFi and modifying the things that i need and using a single script file considering how the processor works. My problem is that the result can't be decrypted backwards.I'm using Nifi 1.13.2With Groovy as languageand Bouncy Castle (bcpg-jdk15on-1.70 and bcprov-jdk15on-1.70)And my code is: import org.bouncycastle.bcpg.ArmoredOutputStreamimport org.bouncycastle.openpgp.PGPCompressedDataimport org.bouncycastle.openpgp.PGPCompressedDataGeneratorimport org.bouncycastle.openpgp.PGPEncryptedDataimport org.bouncycastle.openpgp.PGPEncryptedDataGeneratorimport org.bouncycastle.openpgp.PGPExceptionimport org.bouncycastle.openpgp.PGPLiteralDataimport org.bouncycastle.openpgp.PGPPublicKeyimport org.bouncycastle.openp

Hi,I need to call a soap service that is secured with WS-Security username token authentication and not basic auth. I also need to rely on the Parameter Context value for the id and password. I am using replace text to crate the soap request with the http header. When I try to reference the password it says I cannot due to the fact its sensitive and Sensitive Parameters may only be referenced by Sensitive Properties. I understand basic auth and WS-Security are different types of security. I do not see how to build/populate the security header using my value from the parameter context. Any help would be appreciated. Thank you!

Hi,  I need to install an OS package (yum install ...) -- in order to install packages needed for a (CML) R session. How can I do it on CML ? apparently I cannot do it from the session terminal (on /home/cdsw)  thanks

Hello all,I would like to know what would be the best approach, to change a schema of flowfile. I have got the following scenario : I ingest a csv file, and I would like then to rename column name inferred from the header and put them in a parquet file. Additionally, I want to cast some columns (String) to Integer or other types.I think it may be possible to do this job using the convertRecord component, or may be with the QueryRecord component but I am not sure if it is the best approach.Do you have any idea? If you have got few example it would be nice too.Thanks

Hello Team,   We are experiencing an issue with tables for Hive on HBase where we get the following issue, Note: The pure Hive tables and query on HBase are working fine.       Failed after attempts=11, exceptions:2022-02-28T12:12:54.719Z, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60115: Call to rs.host.500/rs.host.ip.500:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=5,methodName=Scan], waitTime=60002, rpcTimeout=59991 row '' on table 'a-hbase-table' at region=a-hbase-table,,1598840250675.94c031e70c63dbb0f4726251987eb4ec., hostname=rs.host.500,16020,1645349772604, seqNum=550418org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=11, exceptions:2022-02-28T12:12:54.719Z, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60115: Call to rs.host.500/rs.host.ip.500:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=5,methodN

Hi Team, We are planning for the log4j vulnerability in Ambari 2.7.5.0 ofHDP 3.1.5.0. Since we are in EOF. Kindly suggest then patch details 

Hi Team, I have configured my CDP and Keycloak. First I took my Keycloak XML file from "SAML 2.0 Identity Provider Metadata" and with this I have created one Identity provider. And I got one JSON from cloudera community and with this JSON created one client. Json file pasted atlast. With the client I created I pasted my CDP Identity provider ID in "Assertion Consumer Service POST Binding URL" in Keycloak client. After doing these steps with the provided "Target IDP initiated SSO URL" I tried to login all I get is "404 Misconfigured account. IdentityProviderConnector for ID '{39XXX08}' not found" . But if I copy my cloudera Manager URL and paste that in incognito this will redirect me to my keycloak server with this error "We are sorry... Invalid Request". JSON FILE:{"clientId": "cdp-client","surrogateAuthRequired": false,"enabled": true,"clientAuthenticatorType": "client-secret","redirectUris": ["https://consoleauth.altus.cloudera.com/saml"],"webOrigins": ["https://conso

Hadoop components are using default timestamp as UTC. Is there any way to change the default time zone from UTC.Now only seeing option to do the shifting using  

INSERT into `edl_dev_sathyam_raw_mpi_fnxnegotiat`.`negotiation` VALUES (1,'sat', 0)when running above query in impala it was executing successfully but not executing in hivethouing error like below could guide on this how to fix """Permission denied: user=hive, access=WRITE, inode="/user/sathyam.kunda/dev/data/raw/multiplan/mpi_fnxnegotiator.db/negotiationteam":impala:edlpocdev:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:553)"""

I am Getting the following warning on my cluster:This role's Heap Dump Directory is on a filesystem with less than 10.0 GiB of its space free. /tmp the following is df -h command on the partitionFilesystem Size Used Avail Use% Mounted on/dev/mapper/rhel-root 40G 32G 5.8G 85% / how to resolve this issue as i cant find any log or temp files on this filesystem.also there most of it is being consumed by this folder21G /var/lib/solr-infra/ranger_audits_shard1_replica_n1/data Any suggestions.   

Hi,When I setup a MiNiFi flow with a RPG I can set the various batch settings for the input ports (count, duration & size) and see them in the exported xml:<inputPorts><batchSettings><count>5</count><duration>15s</duration><size>1 MB</size></batchSettings> However, when I use the toolkit to convert it to yml these properties no longer appear.Are these settings supported on MiNiFi and, if not, is there any plan to support them? Thanks for your help. 

I want to transform this JSON:{  "_id" : "6218e53465793fa20ea11524",  "patientorderitems" : [ {    "poi_id" : "6218e53465793fa20ea1152a",    "patientorderlogs" : [ {      "pol_id" : "6218e53465793fa20ea1152e",      "useruid" : "61ee4995f16eebb6b7e1c644",      "modifiedat" : "2022-02-25T17:18:28Z"    } ]  }, {    "poi_id" : "6218e53465793fa20ea11525",    "patientorderlogs" : [ {      "pol_id" : "6218e53465793fa20ea11529",      "useruid" : "61ee4995f16eebb6b7e1c644",      "modifiedat" : "2022-02-25T17:18:28Z"    } ]  } ]}To this JSON: {{ "_id" : "6218e53465793fa20ea11524", "poi_id" : "6218e53465793fa20ea1152a", "pol_id" : "6218e53465793fa20ea1152e", "useruid" : "61ee4995f16eebb6b7e1c644", "modifiedat" : "2022-02-25T17:18:28Z"},{ "poi_id" : "6218e53465793fa20ea11524", "poi_id" : "62

Hello,         We are trying to migrate the data from Cloudera solr 4.10.3  to plain apache Solr cloud.         We didn't find any proper document to migrate seamlessly.         Anyone can provide clear documentation on the migration or suggest us the process to follow for the migration.  Thanks & Regards,J. Ganesh Kumar. 

I have a nested json with few first level keys. I want to compare two keys before and after in that json to see if the states are matching or not. If the states match then route to update api else insert api.What I have tried so far : getfile-->evaluate json path(to get the two keys before and after)-->route on attributesNow my problem is that route on attributes always evaluates to unmatched even if the values in eval json path for before and after are the same.Expression : Eval json path 1 : a_before = $.['a']['before'][*]['ip'] Eval json path 2 : a_after = $.['a']['after'][*]['ip']Above results in the same ip - ["1.0.0.0"]RouteOnAttribute : ${a_after:equals(${a_before})}This always results in an unmatched even if the records match.Any help will be appreciated.

Hi,how can I create, as Account Administrator, a CDP user using the default cloudera-sso IdP? I've already set an external IdP with SAML, but I would set a new local account and use it to sign-in with Cloudera SSO.Thank you, best regardsAndrea 

We are trying to create self-signed certificateswhen applying the settings in the cloudera manager, it returns the error below in the log "/var/log/cloudera-scm-server/cloudera-scm-server.log": 2022-02-28 15:55:29,205 WARN 81046291@scm-web-48:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown2022-02-28 15:55:29,205 WARN 1719302079@scm-web-42:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown2022-02-28 15:55:29,205 WARN 1655534592@scm-web-47:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown2022-02-28 15:55:29,234 WARN 1719302079@scm-web-42:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown2022-02-28 15:55:29,419 WARN 1719302079@scm-web-42:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown2022-02-28 15:55:30,167 WARN 1719302079@scm-web-42:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert:

Hello all,I am running into an issue with authentication when trying to connect my Flink (version 1.14.3) server with NiFi. NiFi 1.15.3 is running in secure mode in cluster with 3 nodes. When my Flink flow try to connect to NiFi, it throws: Flink Log: org.apache.flink.util.SerializedThrowable: Tried all cluster URLs but none of those was accessible. Last Exception was org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with explanation: null at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:372) at org.apache.nifi.remote.client.SiteInfoProvider.refreshRemoteInfo(SiteInfoProvider.java:69) at org.apache.nifi.remote.client.SiteInfoProvider.getPortIdentifier(SiteInfoProvider.java:220) at org.apache.nifi.remote.client.SiteInfoProvider.getOutputPortIdentifier(SiteInfoProvider.java:204) at org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:79) at

I'm using self signed certificates for mTLS communication between minifi agents and a stand-alone EFM server. Do I need to put the public keys of each into the others' truststores or is it ok to simply have the root/intermediate CA that was used to sign the public certs in the truststores? So far I haven't been able to establish communication over mTLS for either case, but would like to know so that I can go on and plan on developing infrastructure to automate importing into the EFM truststore all the minifi agent certificates if that is the case. I'm using Cloudera EFM 1.3.1 and Apache minif 1.3.0 Thanks,minh

Hi, In my process I'm triggering some emails at certain points. My problem is that the PutEmail processor just keep dying frequently (even two or three times per day)!!! Nifi Version 1.14.0  When processor dyes I have to terminate and restart it. This is the dump of the last crash:"Timer-Driven Process Thread-1" Id=68 RUNNABLE (in native code)at java.base@11.0.2/java.net.SocketInputStream.socketRead0(Native Method)at java.base@11.0.2/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)at java.base@11.0.2/java.net.SocketInputStream.read(SocketInputStream.java:168)at java.base@11.0.2/java.net.SocketInputStream.read(SocketInputStream.java:140)at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:102)at java.base@11.0.2/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)at java.base@11.0.2/java.io.BufferedInputStream.read(BufferedInputStream.java:271)- waiting on java.io.BufferedInputStream@40399b64at com.sun.mail.util.LineInputStream.

Hi,  I see the in Cloudera Manager in Kafka Status the diagram for "Total Bytes Received Across Kafka Brokers". What does it mean exactly? Is the pure input without replication, or is replication counted on it and the pure input would be only e.g. 1/3 for replication factor 3   Best regards Jaro

I am trying to display a column in HIVE as 'NULL' whenever the source data has a blank space. I tried using the SET TBLPROPERTIES ('serialization.null.format'='\\N') and SET TBLPROPERTIES ('serialization.null.format'='') which did not work. Does anyone know how to resolve this? 

I have the Flow File which has Duplicate Column i want to pick the Column threw index Number,is it possible to do with Query Record or any Processor Note: Column Will change with every new Flow File coming 

Hello, I have my NiFi application running on 1 server. And I need to connect to users SFTP server and need to execute few commands like cd, rm, rename etc on remote server. And to connect to that server I need to fulfill 2 step authentication (username/password and key authentication) So if i ssh and provide key from shell script which will be mounted in docker. For that i will have to add the key needs to be installed in all nodes and then needs to be added to users sftp server. It does not seem to be best solution I there any other way where  I can use NiFi processor to execute these commands to remote sftp server which needs to be connected through 2 step authentication Any advise will be very helpful Thank you.Shalaka