Q&A

Hi All,I am using  Postgress for Hive metastoreHow can we configure postgress HA with Hive with true HA on the Hive Metastore without changing the connection string manually?

Hi Community ! I am planning on taking the CDP-0011 certification exam soon. Can you recommend any book or online resource for preparation ?Regards,

1:08:30.213 PM ERROR RetryingHMSHandler[pool-9-thread-200]: HMSHandler Fatal error: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastoreat org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:171)at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:728)at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)at com.sun.proxy.$Proxy27.commitTransaction(Unknown Source)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1821)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1993)at sun.re

Hi everybody,i'm new to Nifi, and i try to understand how the in/out streams works in groovy. I've created one processor to generate flowfile, one to run a groovy script and a third one to putfile. I've tried many many things but for now my groovy script (not working) look like this:    import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.usermodel.*import org.apache.poi.xssf.usermodel.*import org.apache.poi.ss.util.*import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.extractor.*import java.nio.charset.*import java.io.*def flowFile = session.get()if(!flowFile) returnflowFile = session.write(flowFile, {inputStream, outputStream -> try {FileOutputStream out = new FileOutputStream("workbook.xls");outputStream.write(out);out.close();//is not working.// def record = "toto"// outputStream.write(record.getBytes(StandardCharsets.UTF_8))//is working and giving me a txt file with toto in it. } catch(e) { log.error("Error during proc

I have a scenario where am triggering API in batches (1000 records in each). from the first API I get the value of total records in attribute list X-Total-Count, i wants to merge them all. Please let me know how i can leverage the wait processor in it to wait until we get the whole data till value of X-Total-Count. Wait Processor properties, i have configured. 

We are having a CDP 7.1.7 private cluster with phoenix setup.We had created some phoenix tables long back with certain primary keys, however now when we are trying to check those primary keys on a table via phoenix shell, we are getting empty results.I have tried below commands and no information.!primarykeys <tablename>!column <tablename>!describe <tablename>select column_name from system.catalog where table_name='tablename' Phoenix version is 5.0.0 Please help. 

I wanted to wait until 'ExecuteSQL' processor finishes inserting big file into SQL database (table: bu_service_template), then move to other section. 'ExecuteSQL' processor inserts row by row content of a file. I wanted to wait until file is fully inserted and then move to a next file.My current flow:waitflow

We are looking to invoke any https API urls from Nifi using InvokeHTTP processor.Configuration Done: Invokehttp Processor: SSL Context Service:Tried configuring the Java certs file for the Truststore filename.  Exception: Processor throws exception saying, InvokeHTTP Routing to Failure due to exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ↳ causes: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ↳ causes: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCer

Hi, I'm using CDP Public cloud(7.2.15) with CML (2.0.34-b116) and have a Team created. However, we now want to rename the team name but dont have any option in the UI. Is there any option available through API??

Hello All,  I have encountered a rather weird bug with NiFi's SplitJson processor. I am using NiFi 1.18 and it's a three node cluster. I think the best way of explaining it will be to use as example, so here is how to recreate this bug:  I created this flow with the following sequence of steps:Create a GenerateFlowFile and set the custom text to: { "test": [ { "A": 1 } ], "test2": [ { "B": 1 } ] }Create the top SplitJson and set the path to: $.test.*Create all connections for the top SplitJson (original is terminated), start it and run GenerateFlowFile once -> The output queue should show a single flow file with this content: { "A": 1 }Now select the top SplitJson, Crtl+C, Crtl+V, move the copy down create its connections and set the path to: $.test2.*Start the bottom SplitJson, and run GenerateFlowFile once againBoth output queues have a new flow file and the contents of both flow files are: { "A": 1 }, which is correct for the top SplitJson,

I created my ML workspace , but when I try to create a project and click on the "Workspace" , I get a blank page. Appreciate any inputs.The underlying infra on AWS seems to be fine.Thanks for your feedback.  

If a cml app is not used for days, it is getting timeout and application goes from "Running" status to "Failed" status. Upon checking logs it shows - Engine exited with unknown engine status: -1How we can remove timeout option for cml apps deployed ? or any other option to correct this issue.

Give a scenario that data is written to HFile in Hbase. Now a read occurs and result is saved in blockcache. For the data that is there is in block cache an update occurs, which is saved in Memstore.   Now if the same data is read it will look for data first in Block Cache and if  cache is not yet expired the result is found there. If that data is returned to client then it would be an inconsistent read. Is it possible for an above scenario to occur or is my understanding wrong?

i try to start hive llap but when i start always failed and this logs from task log  WARN conf.HiveConf: HiveConf of name hive.heapsize does not existERROR cli.LlapStatusServiceDriver: FAILED: nulljava.lang.NullPointerException: null at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:199) [hive-llap-server-3.1.0.jar:3.1.0] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:712) [hive-llap-server-3.1.0.jar:3.1.0]FAILED: nullCommand failed after 1 triesanybody can help ? thanks 

I am trying to run an impala shell and I receive the below error:Error connecting: TypeError, __init__() got an unexpected keyword argument 'ssl_version'This happens after the upgrade to 3.4 impala version.

Hi All, When we try to install the Cloudera Management Service we got the below error "login request was rejected" in PostgresSQL. Please advise. PostgresSQL  - 10.19Cloudera manager - 7.6.5, andOS - RHEL 8.4. There is no Auto-TLS or Kerberos as it is a POC cluster we trying to install.   error message in Cloudera-scm-server.log: 2022-11-23 19:07:36,287 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546334432(RepMgrTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Logon denied for user/password. Able to find the database server and database, but the login request was rejected.2022-11-23 19:07:36,287 INFO CommandPusher-1:com.cloudera.cmf.command.components.CommandStorage: Invoked delete temp files for command:DbCommand{id=1546334432, name=RepMgrTestDatabaseConnection, host=hostname01.company.com} at dir:/var/lib/cloudera-scm-server/temp/commands/1546334432 Kind Regards,Sri

Metadata fetch for Impala on CDP fails with below error:INFO: Connected to metastore.alation.shims.HiveShimsBase$RetryingMetaStoreClient invokeWARNING: MetaStoreClient lost connection. Attempting to reconnect (3 of 3) after 1s for getTableObjectsByName method with arguments: [test_sampling_and_profiling02, [stpsalewithmt1000drows, stpsalewithmt1000drows_view, stpsale_view, stpsalesample_view]]org.apache.thrift.TApplicationException: Internal error processing get_table_objects_by_name at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name(ThriftHiveMetastore.java:1496) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name(ThriftHiveMetastore.java:1482) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:143

Hello all, I have a problem with connection from hue to hive and hdfs. For hive i have this error :   HTTPSConnectionPool(host='fqdn', port=10001): Max retries exceeded with url: /cliservice/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))  for hdfs :   Accès impossible : /user/svc_data.HTTPSConnectionPool(host='fqdn', port=14000): Max retries exceeded with url: /webhdfs/v1/user/svc_dataiku?op=GETFILESTATUS&user.name=hue&doas=svc_dataiku (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))   Need help please. Regards

Hi Team, I'm trying to merge two files(.pdf format) and combine it to one file using execute streamcommand using jar file. at java side the merge is working using the same jar. but nifi side the command is not working  , instead of merging files it pushing both individual files to outputstream and throwing the below error at  execute streamcommand processor."Failed to  write flow file to stdin due to broken pipe:java.io.IOException:Broken pipe"Below is flow which i'm using   and the command Arguements which i'm using is-jar;pdfmerge-0.0.1-SNAPSHOT.jar;/opt/nifi/nifi-current/temppdf/${filename:startsWith('invoice_')};/opt/nifi/nifi-current/temppdf/${filename:startsWith('QRBill_')};/opt/nifi/nifi-current/temp2/Output.pdf and here is my configuration of executeStreamCommand:Could you please help me on this? 

Hello All, I have just added a Ranger KMS service to a newly built CDP Cluster. When attempted to create key, got error stating "user not allowed to do create key". I thought it must be related to providing privs to the user in Ranger policies for KMS service and hence, logged in to the Ranger Admin UI where I can't see Ranger KMS service itself.I can see the plugin is enabled and responding in the plugins tab, cm_kms policy is syncing as per plugin status tab. However, service is not there.  Any suggestions please. Thankssnm1523

After reboot clutser hadoop, a few services like yarn don't start anymore with the following message : resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://hostname:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=401. <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/><title>Error 401 Authentication required</title></head><body><h2>HTTP ERROR 401</h2><p>Problem accessing /webhdfs/v1/ats/done. Reason:<pre> Authentication required</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> <br/> Any idea ?Thanks

Hi , NIFI was installed before with root user in our enviorment, now in order to switch it to non-root user, i'm wondering if you can help in the steps that i need to follow. Should i clean and re-install it again? The version is 2.4.2.0 Thanks SJ

Hello, I downloaded the latest Nifi version. i was able to connect nifi to an oracle database and it works fine. However, i need to connect nifi to hive, my problem is i cant find any processors related to hive. There is no SelectHiveQL processor and no PutHiveQL processor or any processor related to hive at all. not even the controller services for hive is found.    Am I doing anything wrong? I just need to connect Nifi to hive and i can't find the processors responsible for this. Please help. Thanks in advance

I am trying to write the below query in getMongo processor {"updatedAt" : {$gt : 2022-12-10}}But it is showing an error stating that query is not a valid JSON representation. I tried adding double quotes as well but when I run it, it doesn't show any error nor any results in success, failure or original nodes. My end goal is to fetch incremental data i.e. only the new data added daily (day -1) using "updatedAt" field. Kindly help me with this 

Hi folks, I'm new to CDP and I'm trying to figure out what roles I require to be able to view Data Hub Clusters as well as the associated Environment and Data Lake pages in CDP Public Cloud? I've been granted the following Account/Resource Roles:"IamViewer""EnvironmentUser" This allows me to see specific DataHub Clusters but when I click the links for either Environments or Data Lakes there's no information appearing. The EnvironmentUser Role has the following Policy attached: [ { "crn": "crn:altus:iam:us-west-1:altus:policy:EnvironmentUserPolicy", "policyStatements": [ { "rights": [ "datahub/read", "datahub/write", "datalake/read", "environments/read" ], "resources": [ "*" ] } ] }]Given the Policy has environments/read & datalake/read what am I missing here? Thanks.