Solved

 Good morning Community,i'm suffering of DNS resolution getting the following fatalistic view every time I access to CM:I'm reading a lot but not getting any robust conclussion of what is the test that concludes in "DNS resolution error". it checks "host's hostname and canonical name" but, what does it means? is it the hostname? the name in the /etc/hosts? In my case, having[root@master1 ~]# hostnamemaster1[root@master1 ~]# python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"master1.domain.net192.168.0.90does it means the discrepancy between hostname and socket.getfqdn()?I can see how many people is getting this error but not a consistent response, could anyone give the solution here? for trying to understand this common issue?Many thanks in advance. 

Hi,I have set up Nifi(CFM 2.1.6.0-323) on our existing clouder private cloud base cluster.I have set up Nifi with LDAP and Ranger.Unfortunately, during the initial installation an error that the Nifi Group is missing occurs.As soon as I create the Nifi group manually in Ranger, the installation works without errors.Have any of you ever had the same error?  <authorizers><userGroupProvider><identifier>cm-user-group-provider</identifier><class>org.apache.nifi.authorization.CMUserGroupProvider</class><property name="Knox Nodes Properties Location">/var/run/cloudera-scm-agent/process/33235-nifi-NIFI_NODE/knox-conf/knox-gateway.properties</property><property name="NiFi Registry Nodes Properties Location">/var/run/cloudera-scm-agent/process/33235-nifi-NIFI_NODE/nifiregistry-conf/peer.properties</property><property name="NiFi Group">nifi</property><property name="Infer Unqualified Hostnames">false</prope

I'm running a HBase-Hadoop cluster with HBase 2.0.1 and Hadoop version 2.7.7. I need to remove a table which has regions in transition (3001 other regions) and I'd like to follow solution described in https://community.cloudera.com/t5/Support-Questions/HBase-Regions-In-Transition/m-p/310513. Unfortunately I cannot run `hbase hbck -j /tmp/target/hbase-hbck2-1.3.0-SNAPSHOT.jar` since I get following error:```Unrecognized option:-j-----------------------------------------------------------------------NOTE: As of HBase version 2.0, the hbck tool is significantly changed.In general, all Read-Only options are supported and can be be usedsafely. Most -fix/ -repair options are NOT supported. Please see usagebelow for details on which options are not supported.-----------------------------------------------------------------------```I built hbck2 following instructions in https://community.cloudera.com/t5/Community-Articles/HBase-HBCK2-tool-for-HDP-3-x/ta-p/244386.What could I do in order to ru

I want to convert json data with both Object and Array into Object at once.I used jolt, but I didn't reach the desired result.The json data is as follows.Input Data:{ "response": { "header": { "resultCode": "00", "resultMsg": "NORMAL_SERVICE" }, "body": { "dataType": "XML", "items": { "item": { "icaoCode": null, "airportName": null, "metarMsg": { "METAR": { "automatedStation": false, "id": "metar-RKSS-20240404000000Z", "permissibleUsage": "OPERATIONAL", "status": "NORMAL", "schemaLocation": "link data link data link data link data link data", "extension": { "msgText": "METAR RKSS 040000Z 09007KT 030V120 9999 SCT035 BKN150 15/05
\n Q1015 NOSIG=" }, "observation": { "OM_Observation": { "id": "obs-RKSS-20240404000000Z", "type": {

I want to use the Atlas API to add/edit user-defined properties to a particular entity. I've previously had success using the API to edit properties that are already defined, such as an entity description:```curl --location --request PUT 'https://[host]/api/atlas/v2/entity/guid/[guid]?name=description' \--header 'Content-Type: application/json' \--header 'Authorization: Basic' \--data '"a test table"'```However, it's not clear how to use the API to create and edit user-defined properties. When I try changing the url query to a user-defined property name, it complains that that property is not defined for the entity type.For reference, in the UI, this is done fairly easily. 

Hi for all!Does anybody know why terminated processor thread info (up right corner) in red does not disappear after it is terminated, the processor works fine but this red info is not disappearing for 2 weeks. How it is critical and how can I fix this as this red appearance is causing issue and everyone is questioning?Thanks!Screenshot in an attached. 

How to configure user authentication using LDAP with DAS service, I refer to https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/using-das/topics/das-user-authentication-ldap.html,  but it doesn't work. Could you share some examples? I want to confirm which configs are necessary.And How to debug this? Data Analytics Studio Webapp Server log doesn't have any useful info.  

We have upgraded nifi to 2.0M1 recently.   It was clean install and process groups were imported from Nifi-registry.   All InvokeHttp processors remained invalid and the reason seems to be that the SSL context service attribute was not recognized.  Please refer the attached screenshot.When I manually selected right SSL context controller service from drop down list, InvokeHttp processor becomes valid and and another observation here is: the action of setting right SSL context service is not treated as code change and process group still shows green.Please advise if there is any work around to fix this problem.  Currently we are manually setting controller service after automated deployment pipeline is complete.   To isolate the issue, we have tried to replicate by creating simple process group with InvokeHttp processor.  Downloaded definition and re-imported onto Nifi canvas.  Issue was reproduced.  This might be a bug in Nifi 2.0M

How to Evaluate attribute (output of Invoke Http which is in JSON format)This attribute value contain a Json in the following format  {    location : { FileSource :  "X" ,  URL : "Y"}} From this attribute value (Json), I need to extract File Source and URL and put it into another attribute with the same name.Note : Flow file Content should remain as it is.Any Pointer ..??

Hi everyone, I would like some clarification. Thank you.We are encountering a blockage during the installation of the Embedded Container Service (ECS). Encountering the following error:  2024/04/03 17:29:48 kubectl get secret cm.creds -n cdp2024/04/03 17:39:38 Error while getting account id. Get "https://console-cdp.apps.ecsm0.am.ocp.poc/authenticate/login": dial tcp: lookup console-cdp.apps.ecsm0.am.ocp.poc on 192.168.30.206:53: no such host2024/04/03 17:39:38 Retry GetAccountId after 10 seconds...2024/04/03 17:39:48 There is an error while retrieving the account id.panic: There is an error while retrieving the account id.goroutine 1 [running]:log.Panicf({0x176705e?, 0xc00074c4c0?}, {0x0?, 0xc00074c4c0?, 0xc00074c4ea?}) /grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/thirdparty/go/src/log/log.go:391 +0x67main.GetAccountId({0xc00074c4c0, 0x3c}) /grid/0/jenkins/workspace/workspace/App_builds/SOURCES/cdp-private/src/go/create-cm-cred.go:129 +0x1efmain.(*CdpInstaller).C

Hello Community,Installation is failing because of allkeys.asc not found...This is not mentioned in the installation proccess and not additional config required, why is this happening and how to fix it?why is this not mentioned in the documentation?I can see the file there, is that a bug?Thanks in advance 

hello cloudera community, we are having problem in impala after enabling kerberos on cdp cluster only the Impala StateStore role is started healthy, the other roles are in bad status checking the log of the Impala Catalog Server role, the following appears: -------------11:07:50.584 AM INFO cc:170SASL message (Kerberos (internal)): GSSAPI client step 1 11:07:50.587 AM INFO cc:78Couldn't open transport for hostname:24000 (No more data to read.) 11:07:50.587 AM INFO cc:94Unable to connect to hostname:24000 11:07:50.587 AM INFO cc:274statestore registration unsuccessful: Couldn't open transport for hostname:24000 (No more data to read.) 11:07:50.587 AM FATAL cc:87Couldn't open transport for hostname:24000 (No more data to read.). Impalad exiting.Wrote minidump to /var/log/impala-minidumps/catalogd/7ae4848b-cd34-4d4c-96cfeaa3-bd4a584f.dmp------------- how can we solve this problem? this problem is urgent!

We have used consumeJMS in NiFi ( CFM 2.1.5.0-215) in the past with traditional monolithic jars from IBM that had all the classes:dhbcore-7.5.jarESBSSLReader.jarjboss-common-4.0.0.jarjboss-j2ee-4.0.0.jarjms.jarmq-8.0.0.4.jarmq.allclient-8.0.0.4.jarmq.commonservices-8.0.0.4.jarmq.headers-8.0.0.4.jarmq.jmqi-8.0.0.4.jarmqjms-8.0.0.4.jarThe new client does not use these jars.From the IBM documentation for 9.3.5 client, it seems NiFi is using a "modular" configuration if I read things correctly. It has the warning:Note: If you have applications that use the available JAR files in a modular context by relying on the default module names, you must update your applications to require the new module names. The default module names are derived from the JAR file names.Followed by:You can configure your modular application to use IBM MQ classes for JMS (com.ibm.mq.allclient.jar) by completing the following steps:Configure the application to require the com.ibm.mq.java

I'm just curious to know. When I start a processor in the NIFI cluster, where does NIFI store the processor state "RUNNING" in this case?

Hi everyone, I would like some clarification. Thank you.We are using CDP Private Cloud Data Services 1.5.2 on OpenShift 4.12.46 (Kubernetes 1.25).Currently, we are unable to log in to the OpenShift Container Platform (OCP) using 'oc login'. Therefore, we are using the 'localhost-recovery.kubeconfig' file to execute OCP commands.Source: https://access.redhat.com/solutions/4845381To use the 'localhost-recovery.kubeconfig' file, run the following commands: export KUBECONFIG=/root/localhost-recovery.kubeconfigoc get ns We deleted the CDP Private Cloud Data Services on OpenShift from the Cluster Manager (CM) UI. As a result, the UI no longer displays the Private Cloud Data Services, although the actual Private Cloud Data Services still exist on OpenShift. However, we cannot perform operations through the CM UI (as it is not visible in the UI).In this situation, we have a few questions:How can we remove the Private Cloud Data Services from OCP?Is there a possibility to see the exis

Hi everyone, I would like some clarification. Thank you.During the installation of ECS, we encountered some errors that prevented successful installation. One of the issues was as follows: the Certificate Signing Request (CSR) remained in an "approved" state instead of being "certificated," causing it to remain stuck in a loop.Following the steps outlined in the "Manually uninstalling ECS from a cluster" and "Configuring a containerized cluster with SELinux" documents helped us overcome this issue. # /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get csrNo resources foundHowever, we encountered another problem: time="2024-03-27T02:37:24-04:00" level=info msg="etcd temporary data store connection OK"time="2024-03-27T02:37:24-04:00" level=info msg="Reconciling bootstrap data between datastore and disk"time="2024-03-27T02:37:24-04:00" level=fatal msg="Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token"&

I need your help in figuring out why this Hive Server 2 are not working in ambari, Please find the log information Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_server.py", line 143, in <module> HiveServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_server.py", line 53, in start hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_service.py", line 64, in hive_service check_fs_root(params.hive_server_conf_dir, params.execute_path) File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_service.py", line 173, in check_fs_root environment = {'PATH': execution_path} File "/usr/lib/ambari-

Hi everyone, our Hadoop cluster has recently encountered a strange issue. After starting the cluster, Spark jobs run normally, but after running for more than a week, job submission timeouts occur. Specifically, the jobs are all in the ACCEPTED state, but fail due to timeout after 2 minutes. The error log shows:Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/134.1.1.3:45866 remote=hdp144019.bigdata.com/134.1.1.19:45454]I found an app id, and then went to check the corresponding NodeManager log for attempting to start the AM. I found that the AM was never started at all. At the time when the job submission timed out, the NodeManager log reported the following log:2024-03-15 00:04:17,668 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:handle(1607)) - couldn't find application application_1709910180593_38018 while processing FINISH_APPS event. The R

I am trying to copy table rows from one oracle database to another oracle table. PutDatabaseRecord cannot find the second table. I checked many times that table  exists. But it is still giving error. Does anybody know how to solve it?  PutDatabaseRecord[id=cb461cc1-018d-1000-21fe-ec8db1afeca1] Failed to put Records to database for FlowFile[filename=040d505a-1d6c-4f36-8a2b-38bcf577a059]. Routing to failure.: org.apache.nifi.processors.standard.db.TableNotFoundException: Table fsc_bank_dim_zz not found, ensure the Catalog, Schema, and/or Table Names match those in the database exactly 

When try to use ExecuteSQL 1.25.0  to query  data from oracle 11g  , we received the below errors  can you guide me please , we have Apache NiFi in docker .Caused by: java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1ORA-01882: timezone region not found  

We have a requirement to merge files in a specific HDFS path. The merged file should also be created in the same HDFS path but with a different name. Could you please suggest how we can achieve this using the Nifi Processor?

Hi everyone,I need to create HUE table separating "|" for insert like following values into table with removing all not required """94112605011"|"DEFAULT"|"FTTH_WEB FAMILY PLUS!PDL&Ec5@1706725800;94112605011;1704047400"|"COUNTER"|"u"|"0E-12"|"2023-12-31 18:30:00.000000"|"2024-01-31 18:30:00.000000"|"0"|"{""data"":{""o"":{""-t"":""ECS_BalanceContextInfo"",""m"":[{""-c"":""57a2"",""-v"":""1"",""a"":[{""-n"":""unitType"",""v"":{""-s"":""VOLUME""}},{""-n"":""offerId"",""v"":{""-s"":""15""}},{""-n"":""offerName"",""v"":{""-s"":""FTTH_WEB FAMILY PLUS""}},{""-n"":""offerQuota"",""v"":{""-s"":""42949672960""}},{""-n"":""quotaThreshold"",""v"":{""-s"":""0""}}]}],""o"":[{""-t"":""ECS_BalanceContextInfoBase"",""m"":[{""-c"":""5136"",""-v"":""1""},{""-c"":""8d18"",""-v"":""2"",""a"":[{""-n"":""subscriptionId"",""v"":{""-s"":""9411260501111152164097540010""}}]}]},{""-t"":""ECS""},{""-t"":""AVS"",""m"":[{""-c"":""e3a0"",""-v"":""1""}]}]}}}"|"{""a"":[""0""]}"|"BYTES"|"0E-12"i HAVE USED FOLLOWING

Hello guys,I currently work on a company that does provide hive 3.1 servers by using knox or zookeeper (kerberos) authentication methods. I am able to connect and query data by using a odbc connection on my personal computer. Now I need to set the connection on a virtual jupiter notebook server with pyodbc , so, I am not able to install the ODBC (and probably the server is based on Linux anyway).I´ve seen many options as pyhive, implya and others, but everything I find is regarding Hive 2 servers.So, what could I use to set the python connection to the Hive 3 servers?

I have a kerberized hadoop cluster. Installed knox in kubernetes container with its keytab. Configured knox topology with shiro provider and ldap authentication. I'm able to access webhdfs fine with ldap credentials but Hive is failing with HTTP 401 error.Knox is configured with self-signed cert and hive is not configured with SSL.topology.xml -- Configured with shiro provider and ldap info<service><role>WEBHDFS</role><url>http://name1_hostname:50070/webhdfs</url><url>http://name12_hostname:50070/webhdfs</url></service><service><role>HIVE</role><url>http://<host_name>:10004/cliservice</url><param><name>replayBufferSize</name><value>8</value></param></service>Configured hive-site.xml with below configs in HS2 hosthive.server2.thrift.http.port = 10004hive.server2.thrift.http.path = cliservicehive.server2.transport.mode = httphive.server2.allow.user.substitution =

Hello,I'm currently running Nifi version 1.8.0, starting from last week I noticed that I'm missing PutSysLog message from my syslog system.  Some logs would appear and some wouldn't write to the syslog system.  I checked my syslog system and was unable to find any issue.  No other systems are experiencing issue writing to the syslog system.  I enabled DEBUG on PutSysLog and it shows it was successful in sending the message to the syslog system but no entry exists. At the same time other PutSysLog does record the entries in syslog system and the file flows through other processors.      Any suggestions on how to troubleshoot this issue?Thanks!