Q&A

Stuck in 'upgrade control plane' step while upgrading ECS to 1.5.3 from 1.5.2.Casuses: cdp-release-post-upgrade-hook-job1 failed.container logs are =========================================****************************************************https://console-cdp.apps.cdppvc1.hadoop.com/etc/cdp/cdp_keys.jsonWhile list environments: Something is wrong with output, Output JSON:____ERROR__WHILE__CALLING__LIST__ENVIRONMENTS__COMMAND____ /opt/app-root/lib64/python3.9/site-packages/urllib3/connectionpool.py:1103: InsecureRequestWarning: Unverified HTTPS request is being made to host 'console-cdp.apps.cdppvc1.hadoop.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warningswarnings.warn(An error occurred: 5 NOT_FOUND: No access key found with name: 1b098f81-ce2c-4342-ba02-f0e607b0d83c (Status Code: 401; Error Code: ; Service: environments; Operation: listEnvironments; Request ID: 80bb6513-3fe8-41b9-bf2f-e0a6

Hi everyone, I would like some clarification. Thank you. During the installation of ECS, we encountered some errors that prevented successful installation. One of the issues was as follows: the Certificate Signing Request (CSR) remained in an "approved" state instead of being "certificated," causing it to remain stuck in a loop.   # /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get csrNAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITIONecs-tolerations-webhook-csr 55m kubernetes.io/kubelet-serving system:admin <none> Approved   How should we address this? Should we consider a reinstallation? Or is there an underlying cause behind this issue? Thank you. Related issue: Failure to Install Embedded Container Service (ECS), encountering "Failed to reconcile with temporary etcd" after manual uninstallation 

Dear all, I want to ask the difference between `insert entire record` and `insert record fields` for the record result contents.When i use `insert record fields`, i got error:but returned ID is what I want:  When i use `insert entire record`, i did not get any errors, but in stead i have this output How can I get the correct output that I want without any errors? 

Hey everyone, I was surfing the web since this morning looking for a way/guide/tutorial on how to secure Nifi so I can connect via https and not http but couldn't find anything that was compatible with the 2.0.0-M2 version.I am new to Nifi and i just want to secure my access to it.most of the guides i've stumbled upon use the nifi-toolkit with the command  ./bin/tls-toolkit.sh standalone -C "CN=my_username, OU=NiFi" to generate and sign the client certificate for the access, alongside some tweaks in the nifi.properties file.but in the toolkit version of nifi-2.0.0-M2, there is no file called tls-toolkit.sh so im kinda lost here as there is only :and according to the documentations with these scripts i didnt know how to secure nifi with the standalone method.could someone please help provide some insights on how to secure nifi with https? thanks 

Hello,I have a Controller Service for DistributedMapCacheServer, when i tried to disable it so i can change it's properties, it shows an error of a missing referencing component. Why does my controller service still detects a component that is already deleted and how do i solve this so i can change my controller service properties?Thanks,Regards,Adhitya

 Good morning Community,i'm suffering of DNS resolution getting the following fatalistic view every time I access to CM:I'm reading a lot but not getting any robust conclussion of what is the test that concludes in "DNS resolution error". it checks "host's hostname and canonical name" but, what does it means? is it the hostname? the name in the /etc/hosts? In my case, having[root@master1 ~]# hostnamemaster1[root@master1 ~]# python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"master1.domain.net192.168.0.90does it means the discrepancy between hostname and socket.getfqdn()?I can see how many people is getting this error but not a consistent response, could anyone give the solution here? for trying to understand this common issue?Many thanks in advance. 

Hi,I have set up Nifi(CFM 2.1.6.0-323) on our existing clouder private cloud base cluster.I have set up Nifi with LDAP and Ranger.Unfortunately, during the initial installation an error that the Nifi Group is missing occurs.As soon as I create the Nifi group manually in Ranger, the installation works without errors.Have any of you ever had the same error?  <authorizers><userGroupProvider><identifier>cm-user-group-provider</identifier><class>org.apache.nifi.authorization.CMUserGroupProvider</class><property name="Knox Nodes Properties Location">/var/run/cloudera-scm-agent/process/33235-nifi-NIFI_NODE/knox-conf/knox-gateway.properties</property><property name="NiFi Registry Nodes Properties Location">/var/run/cloudera-scm-agent/process/33235-nifi-NIFI_NODE/nifiregistry-conf/peer.properties</property><property name="NiFi Group">nifi</property><property name="Infer Unqualified Hostnames">false</prope

I'm running a HBase-Hadoop cluster with HBase 2.0.1 and Hadoop version 2.7.7. I need to remove a table which has regions in transition (3001 other regions) and I'd like to follow solution described in https://community.cloudera.com/t5/Support-Questions/HBase-Regions-In-Transition/m-p/310513. Unfortunately I cannot run `hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar` since I get following error:```Unrecognized option:-j-----------------------------------------------------------------------NOTE: As of HBase version 2.0, the hbck tool is significantly changed.In general, all Read-Only options are supported and can be be usedsafely. Most -fix/ -repair options are NOT supported. Please see usagebelow for details on which options are not supported.-----------------------------------------------------------------------```I built hbck2 following instructions in https://community.cloudera.com/t5/Community-Articles/HBase-HBCK2-tool-for-HDP-3-x/ta-p/244386.What could I do in order to ru

I want to convert json data with both Object and Array into Object at once.I used jolt, but I didn't reach the desired result.The json data is as follows.Input Data:{ "response": { "header": { "resultCode": "00", "resultMsg": "NORMAL_SERVICE" }, "body": { "dataType": "XML", "items": { "item": { "icaoCode": null, "airportName": null, "metarMsg": { "METAR": { "automatedStation": false, "id": "metar-RKSS-20240404000000Z", "permissibleUsage": "OPERATIONAL", "status": "NORMAL", "schemaLocation": "link data link data link data link data link data", "extension": { "msgText": "METAR RKSS 040000Z 09007KT 030V120 9999 SCT035 BKN150 15/05
\n Q1015 NOSIG=" }, "observation": { "OM_Observation": { "id": "obs-RKSS-20240404000000Z", "type": {

I want to use the Atlas API to add/edit user-defined properties to a particular entity. I've previously had success using the API to edit properties that are already defined, such as an entity description:```curl --location --request PUT 'https://[host]/api/atlas/v2/entity/guid/[guid]?name=description' \--header 'Content-Type: application/json' \--header 'Authorization: Basic' \--data '"a test table"'```However, it's not clear how to use the API to create and edit user-defined properties. When I try changing the url query to a user-defined property name, it complains that that property is not defined for the entity type.For reference, in the UI, this is done fairly easily. 

Hi for all!Does anybody know why terminated processor thread info (up right corner) in red does not disappear after it is terminated, the processor works fine but this red info is not disappearing for 2 weeks. How it is critical and how can I fix this as this red appearance is causing issue and everyone is questioning?Thanks!Screenshot in an attached. 

How to configure user authentication using LDAP with DAS service, I refer to https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/using-das/topics/das-user-authentication-ldap.html,  but it doesn't work. Could you share some examples? I want to confirm which configs are necessary.And How to debug this? Data Analytics Studio Webapp Server log doesn't have any useful info.  

How can I establish connection with MSSQL using DBCPconnectionpool if I have windows authentication. I tried adding integratedSecurity = true in the connection URL but it didn't work. 

We have upgraded nifi to 2.0M1 recently.   It was clean install and process groups were imported from Nifi-registry.   All InvokeHttp processors remained invalid and the reason seems to be that the SSL context service attribute was not recognized.  Please refer the attached screenshot.When I manually selected right SSL context controller service from drop down list, InvokeHttp processor becomes valid and and another observation here is: the action of setting right SSL context service is not treated as code change and process group still shows green.Please advise if there is any work around to fix this problem.  Currently we are manually setting controller service after automated deployment pipeline is complete.   To isolate the issue, we have tried to replicate by creating simple process group with InvokeHttp processor.  Downloaded definition and re-imported onto Nifi canvas.  Issue was reproduced.  This might be a bug in Nifi 2.0M

Good day -- I am having trouble getting NiFi Registry to work with OIDC.  I have NiFi itself, configured successfully -- working with Keycloak as the OIDC provider.  Everything is containerized -- I have a Docker stack running Traefik, Bind9, and Keycloak.I have another stack running NiFi (X2), Zookeeper, and NiFi Registry.As I stated, I have NiFi set up and working just fine with Keycloak. However, NiFi Registry is not prompting me for any login. It is not routing the request back to Keycloak for login, and when you see the NiFi Registry page, there is no identity listed and no menu options available.  It just looks like this:I have my authorizers.xml file set up this way: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class> <property nam

How to Evaluate attribute (output of Invoke Http which is in JSON format)This attribute value contain a Json in the following format  {    location : { FileSource :  "X" ,  URL : "Y"}} From this attribute value (Json), I need to extract File Source and URL and put it into another attribute with the same name.Note : Flow file Content should remain as it is.Any Pointer ..??

Hi everyone, I would like some clarification. Thank you.We are encountering a blockage during the installation of the Embedded Container Service (ECS). Encountering the following error:  2024/04/03 17:29:48 kubectl get secret cm.creds -n cdp2024/04/03 17:39:38 Error while getting account id. Get "https://console-cdp.apps.ecsm0.am.ocp.poc/authenticate/login": dial tcp: lookup console-cdp.apps.ecsm0.am.ocp.poc on 192.168.30.206:53: no such host2024/04/03 17:39:38 Retry GetAccountId after 10 seconds...2024/04/03 17:39:48 There is an error while retrieving the account id.panic: There is an error while retrieving the account id.goroutine 1 [running]:log.Panicf({0x176705e?, 0xc00074c4c0?}, {0x0?, 0xc00074c4c0?, 0xc00074c4ea?}) /grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/thirdparty/go/src/log/log.go:391 +0x67main.GetAccountId({0xc00074c4c0, 0x3c}) /grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/src/go/create-cm-cred.go:129 +0x1efmain.(*CdpInstaller).C

Hello Community,Installation is failing because of allkeys.asc not found...This is not mentioned in the installation proccess and not additional config required, why is this happening and how to fix it?why is this not mentioned in the documentation?I can see the file there, is that a bug?Thanks in advance 

We have a requirement to transfer files from an HDFS directory to a remote server.I've noticed options to copy files from HDFS to the local filesystem first (using copyToLocal) and then transfer files from the local filesystem to the remote server (using scp). But is there any direct method to copy files from HDFS to a remote server, such as using Sqoop functionality or any other methods like without copying to the local file system

hello cloudera community, we are having problem in impala after enabling kerberos on cdp cluster only the Impala StateStore role is started healthy, the other roles are in bad status checking the log of the Impala Catalog Server role, the following appears: -------------11:07:50.584 AM INFO cc:170SASL message (Kerberos (internal)): GSSAPI client step 1 11:07:50.587 AM INFO cc:78Couldn't open transport for hostname:24000 (No more data to read.) 11:07:50.587 AM INFO cc:94Unable to connect to hostname:24000 11:07:50.587 AM INFO cc:274statestore registration unsuccessful: Couldn't open transport for hostname:24000 (No more data to read.) 11:07:50.587 AM FATAL cc:87Couldn't open transport for hostname:24000 (No more data to read.). Impalad exiting.Wrote minidump to /var/log/impala-minidumps/catalogd/7ae4848b-cd34-4d4c-96cfeaa3-bd4a584f.dmp------------- how can we solve this problem? this problem is urgent!

We have used consumeJMS in NiFi ( CFM 2.1.5.0-215) in the past with traditional monolithic jars from IBM that had all the classes:dhbcore-7.5.jarESBSSLReader.jarjboss-common-4.0.0.jarjboss-j2ee-4.0.0.jarjms.jarmq-8.0.0.4.jarmq.allclient-8.0.0.4.jarmq.commonservices-8.0.0.4.jarmq.headers-8.0.0.4.jarmq.jmqi-8.0.0.4.jarmqjms-8.0.0.4.jarThe new client does not use these jars.From the IBM documentation for 9.3.5 client, it seems NiFi is using a "modular" configuration if I read things correctly. It has the warning:Note: If you have applications that use the available JAR files in a modular context by relying on the default module names, you must update your applications to require the new module names. The default module names are derived from the JAR file names.Followed by:You can configure your modular application to use IBM MQ classes for JMS (com.ibm.mq.allclient.jar) by completing the following steps:Configure the application to require the com.ibm.mq.java

I'm just curious to know. When I start a processor in the NIFI cluster, where does NIFI store the processor state "RUNNING" in this case?

Hi everyone, I would like some clarification. Thank you.We are using CDP Private Cloud Data Services 1.5.2 on OpenShift 4.12.46 (Kubernetes 1.25).Currently, we are unable to log in to the OpenShift Container Platform (OCP) using 'oc login'. Therefore, we are using the 'localhost-recovery.kubeconfig' file to execute OCP commands.Source: https://access.redhat.com/solutions/4845381To use the 'localhost-recovery.kubeconfig' file, run the following commands: export KUBECONFIG=/root/localhost-recovery.kubeconfigoc get ns We deleted the CDP Private Cloud Data Services on OpenShift from the Cluster Manager (CM) UI. As a result, the UI no longer displays the Private Cloud Data Services, although the actual Private Cloud Data Services still exist on OpenShift. However, we cannot perform operations through the CM UI (as it is not visible in the UI).In this situation, we have a few questions:How can we remove the Private Cloud Data Services from OCP?Is there a possibility to see the exis

Hi everyone, I would like some clarification. Thank you.During the installation of ECS, we encountered some errors that prevented successful installation. One of the issues was as follows: the Certificate Signing Request (CSR) remained in an "approved" state instead of being "certificated," causing it to remain stuck in a loop.Following the steps outlined in the "Manually uninstalling ECS from a cluster" and "Configuring a containerized cluster with SELinux" documents helped us overcome this issue. # /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get csrNo resources foundHowever, we encountered another problem: time="2024-03-27T02:37:24-04:00" level=info msg="etcd temporary data store connection OK"time="2024-03-27T02:37:24-04:00" level=info msg="Reconciling bootstrap data between datastore and disk"time="2024-03-27T02:37:24-04:00" level=fatal msg="Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token"&

I am trying to run a script in oozie and every time I receive the below error regarding impala.dbapi. The module is inserted correctly in the script.  Stdoutput Traceback (most recent call last):Stdoutput File "/tmp/sorting_table.py", line 8, in <module>Stdoutput from impala.dbapi import connectStdoutput ImportError: No module named impala.dbapiExit code of the Shell command 1<<< Invocation of Shell command completed <<<java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.ja