Q&A

How does the ZKFC running in the namenode server maintain a open session in zookeeper?

Which configuration should I change for set specific NIC for HDFS data-node data replication?I want to separate traffic for Service between Replication. 

1000 requests are queued in NIFI , invoke HTTP processor is showing socket timeout exception , i have added a self loop in invoke http processor so that it keep on retrying . These requests are stuck permanently.My issue is endpoint which invoke http processor is trying to invoke is up and running .  i checked from postman i am able to get success response. 2023-11-08 12:17:34,783 ERROR [Timer-Driven Process Thread-138] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=b48ff3b8-befb-32c2-b670-83445e1873f2] Routing to Failure due to exception: java.net.SocketTimeoutException: connect timed out: java.net.SocketTimeoutException: connect timed outjava.net.SocketTimeoutException: connect timed outat java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketI

Hello,Is there a way to check the replication factor of a particular folder in HDFS?While we have default replication set to 3 in CM for some reason files being uploaded in a particular folder shows up with replication factor of 1. RegardsWert

i face the mentuioned error while logging to nifi althogh in log files no error found and according to logs my ldap user logged successfully as below:2024-07-10 11:12:19,489 INFO [NiFi Web Server-33] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for AMOHAMED2792024-07-10 11:20:07,893 INFO [NiFi Web Server-71] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://node_name:9443/nifi-api/flow/current-user (source ip: 10.230.237.150)2024-07-10 11:20:07,895 INFO [NiFi Web Server-71] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for AMOHAMED279 

HI Team,          we have jar executable which is running fine in Nifi 1.15 version  after we upgraded to 1.17 , when we tried to execute that jar its showing as invalid or corrupt jar file.when we checked java version both nifi and jar executable are on same version i.e. java 8 however we noticed that jdk versions are different from previous version to latest versionnifi 1.15 having openjdk version 1.8.0_312Nifi 1.17 having openjdk version jdk1.8.0_332suspecting this caused issue, however we are not able to update the jar. Can anyone help here what might be the issue here and how we can solve this issue. thanks

Hi ,We are in the process of migrating from YARN to Kubernetes for its benefits and upgrading our Spark version from 2.4.4 to 3.5.1. As part of this transition, we have decided to use Scala version 2.12.18 and have upgraded Java from version 8 to 11. Currently, I am encountering three main issues:I am experiencing an ArithmeticException due to long overflow. Could the switch from Java 8 to 11 be causing this issue?The deployment mode specified as cluster in the spark-submit command is being overridden to client.I am unable to use AWS Hadoop package classes in spark-submit, despite including the jars in the container.$SPARK_HOME/bin/spark-submit \--master k8s://$K8S_SERVER \ \--deploy-mode cluster \--name testing \--class dt.cerebrum.iotengine.sparkjobs.streaming \--conf spark.kubernetes.file.upload.path=s3a://cb-spark/path \--conf spark.hadoop.fs.s3a.endpoint="http://xxxxxxx.xxx" \--conf spark.hadoop.fs.s3a.access.key="xxxx" \--conf spark.hadoop.fs.s3a.secret.key="xxxxxxxxx" \--conf sp

I've got an instance of hue server on my host configured to use infra-solr, and it's tell me that it's got stale configuration on solr_url because the server is changed. If I restart the hue service for the first minute everything is good, but just some minute later it tells me again that's got stale configuration. The solr_url value isn't changed, there's the same url that was before i restart the service. Seems like it cannot update hue.ini config fileAny help?

Hi all, I have a CDP 7.1.8 cluster running in my environment. Recently I would like to migrate the existing CM server to another host. I followed the instruction in the link https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-moving-the-cm-server-new-host2.html.  After that, the CM server is successfully migrated to the target host and I can see the cluster node recognized by the new CM server. However, I cannot see the cluster appear on the new CM server console. Finally, I have to manually restore the cluster using the link https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/configuring-clusters/topics/cm-api-import-configuration.html. My question is, am I suppose to restore the cluster manually? What I expected is I can see the cluster immediately without extra work done after switching CM server. Could someone clarify for me please? Thank you.

Hi TeamI wish to know the Hardware Specification of the D8 V3 specified in the Operation Database category described in the below estimation cost link D8 v3 $0.3200/hrhttps://www.cloudera.com/products/pricing/pricing-calculator.htmlLooking forward to your reply ! RegardsNaveen

Hello,is it possible to create seperate individual list from one list based on some conditions. e.g i have a list that have category & product information so i have to create individual list based on category/item id with excludeInd flag 0/1,List A - CategoryIds List -> Exclude List B - CategoryIds List -> Include List C - ProductIds List -> Exclude List D -ProductIds List -> Include Input{ "correlationId": "3242a7b4-7929-43b3-9fb6-0d7e173373af", "sourceSystemId": "PCS", "messageTimestamp": "2024-05-03T00:00:00", "id": 1234, "promotionId": 2303, "offerId": 2303, "description": "Members Get Tees $15", "customerDescription": "Members Get Tees $15", "levelCode": 0, "typeCode": 3, "templateId": 4, "couponCode": null, "isCouponCodeRequired": 0, "startDatetime": "2024-06-20T00:00:00", "endDatetime": "2024-06-24T23:59:00", "cancelationDatetime": null, "comments": null, "distributionRuleCode": null, "exclusiveDiscountInd": 0, "tiers"

Hello,We are on CDSW 1.10.5. We are getting a weird error as below in our jobs:xEngine exited with unknown engine status: -1xPost "https://100.52.0.1:443/api/v1/namespaces?timeout=1m0s": dial tcp 100.52.0.1:443: connect: connection refusedAlso, the application service goes down. What may be the reason here?

Hi to all,let's say i have a flow file with some given attributes.What i want to achieve is to add flow attribute with a new one those value is retrieved from SQL Server query, where 2 existing attributes is passed in where clause(of course will return only one result) Just for exampleCurrent Flow file:year = 2018brand = audiI would add a third attribute called "model" with value retrieved from a following query:"SELECT model FROM cars WHERE year=${year} AND brand=${brand}" I've searched around but not still found a good answer....Tried evaluating lookup services but still not usable in this scenario. Can anyone can point me on the right direction? Many Thanks

Hello,Could you assist me in retrieving the single branchName value from priceRuleAttributes list based on id, When i am doing without id match its working fine and giving me single branchName value for each index but with id match its combining all values at 1st index.Input  { "Data": { "ID": "09878666", "DATE": "2022-01-01", "ARRAY_ONE": [ { "NAME": "test_1", "priceRuleAttributes": [ { "type": "CFA", "id": "PR_BRANCH_NAME", "values": [ "MEMBERS" ] }, { "type": "CFA", "id": "PR_PRICE_RULE_PROMO", "valueBoolean": true }, { "type": "CFA", "id": "PR_TARGET_SEGMENT", "values": [ "1" ] } ] }, { "NAME": "test_2", "priceRuleAttributes": [ { "type": "CFA", "id": "PR_BRANCH_NAME",

Hi Team, I have been trying to generate a scenario like three separate modules to join through Fork Enrichment with Join Enrichment process.For example:  There are three processor groups A , B and CMy existing code designed like join the A and B modules with Fork Enrichment with Join Enrichment process by adding strategy SQL, and getting the expected result.Now, latest requirement to join the Module C into SQL query with existing flow( SELECT a. id, b.name, c. city FROM A a JOIN B b ON a.id = b.id JOIN C c ON c.id = a.id)Is it possible to implement one original and two enrichment flows with Fork Enrichment with Join Enrichment, or any alternative approachPlease advice.  

Hi everyone!I have json from kafka: [ {"before" : {"old_key1" : old_value1,"old_key2" : old_value2,"old_key3" : old_value3,"old_key4" : old_value4},"after" : {"new_key1" : new_value1,"new_key2" : new_value2,"new_key3" : new_value3,"new_key4" : new_value4},"table" : "table_name","op" : "u", -- operation ['u-update', 'c-insert', 'd-delete']"tm" : 1720539441000000000 - timestamp of changes} ]I need execute on database sql like:update table_nameset new_key1=new_value1, new_key2=new_value2, new_key3=new_value3, new_key4=new_value4where old_key1=old_value1 and old_key2=old_value2 and old_key3=old_value3 and old_key4=old_value4; What's the best way to update rows in a database with before and after values by NiFi. I can`t use a groovy script because our team know sql only and if I produce it on groovy, I must support this alone. I have tried create sql by jolt, but all values without quotes and I can`t  test a type.  I think a have two ways: 1) insert all data t

Hi Teams,I got this warning on CDP-DS cluster ECS Instance : "Couldn't parse an alert list from Infrastructure Prometheus ,This happens when the Prometheus is not available, the credentials are wrong or another error is returned."All pod status is running, how to troubleshoot this issue 

I need to know if there are any coming updates to using the full functionality(update, delete or create table) of Iceberg from impala on CDP Private Cloud 7.1.9.  

Hi,Can Presto and Trino be installed and configured in Cloudera CDP?if yes, please provide details.

Hi,Anyone knows if there is compatibility issue for using python exntension with the latest python version 3.12. To me the 3.12 doesnt seem to work well with the python extension and I keep getting the following error :AttributeError: 'FileFinder' object has no attribute 'find_module'2024-05-16 22:12:52,884 ERROR python.ExtensionManager Failed to load Python extensions from module file <Unknown Module File>. This module will be ignored.Traceback (most recent call last):File "F:\NIFI-PORD-ENV-2.0.0-M2\python\framework\ExtensionManager.py", line 186, in __discover_extensions_from_pathsmodule = finder.find_module(name)The error seems to go away when using 3.11.* 

Hi all,I developed my Spark application using Spark 3.4, and it runs on a different platform. Now, I want to run it on another platform that uses Spark 2.4. When I submit the job using the command belowspark-submit --class Main --master yarn --deploy-mode client --num-executors 6 --executor-memory 10G --driver-memory 20G --executor-cores 4 --driver-cores 4 --conf spark.yarn.submit.waitAppCompletion=true --conf spark.executor.extraJavaOptions="-XX:+UseG1GC -XX:MaxGCPauseMillis=20" --conf spark.driver.extraJavaOptions="-XX:+UseG1GC -XX:MaxGCPauseMillis=20" --jars /home/cdsw/mysql-connector-java-5.1.49.jar /home/cdsw/MyApplication.jaran error appears as shown:Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/connector/catalog/TableProvider at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.j

我正在尝试使用 Sqoop 将数据从 PostgreSQL 提取到 HDFS，但一直收到错误2024-07-02 13:26:20,913 INFO mapreduce.JobSubmitter：拆分数：1 2024-07-02 13:26:21,183 INFO mapreduce.JobSubmitter：提交作业令牌：job_1719844718570_0005 2024-07-02 13:26:21,183 INFO mapreduce.JobSubmitter：使用令牌执行：[] 2024-07-02 13:26:21,336 INFO conf.Configuration：未找到 resource-types.xml 2024-07-02 13:26:21,337 INFO resource.ResourceUtils：无法找到“resource-types.xml”。2024-07-02 13:26:21,382 INFO impl.YarnClientImpl：已提交的应用程序 application_1719844718570_0005 2024-07-02 13:26:21,410 INFO mapreduce.Job：跟踪作业的 URL：http://master:8088/proxy/application_1719844718570_0005/2024-07-02 13:26:21,411 INFO mapreduce.Job：正在运行的作业：job_1719844718570_0005 2024-07-02 13:26:25,455 INFO mapreduce.Job：作业 job_1719844718570_0005 正在运行uber 模式：false 2024-07-02 13：26：25,456 INFO mapreduce.Job：map 0％reduce 0％2024-07-02 13：26：25,480 INFO mapreduce.Job：作业 job_1719844718570_0005 失败，状态为 FAILED，原因是：应用程序 application_1719844718570_0005 由于 AM 而失败 2 次，appattempt_1719844718570_0005_000002 的容器退出，退出代码为：1

After installing CDP 7.1.9 on AWS ec2 machines and RDS Postgres as backend database, below databases are created, where scm is the Cloudera Manager database: List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges-------------------------+-------------------------+----------+-------------+-------------+----------------------- amon | amon | UTF8 | en_US.UTF-8 | en_US.UTF-8 | classic-clusters | classic-clusters | UTF8 | en_US.UTF-8 | en_US.UTF-8 | cluster-proxy | cluster-proxy | UTF8 | en_US.UTF-8 | en_US.UTF-8 | das | das | UTF8 | en_US.UTF-8 | en_US.UTF-8 | db-alerts | db-alerts | UTF8 | en_US.UTF-8 | en_US.UTF-8 | db-clusteraccessmanager | db-clusteraccessmanager | UTF8 | en_US.UTF-8 | en_US.UTF-8 | db-dex

Hi    For example, the hive component of cluster CDH often has internal pause. How to tune it?    CDH version : CDH 6.3.2

 == sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log2024-07-08 15:10:06,345 WARN scm-web-117:com.cloudera.server.web.cmf.EventsController: (7 skipped) Exception querying eventscom.cloudera.cmon.MgmtServiceLocatorException: There is no configured event storeat com.cloudera.cmon.MgmtServiceLocator.getEventStoreQueryProxy(MgmtServiceLocator.java:314)at com.cloudera.server.web.cmf.events.EventDao.findEvents(EventDao.java:398)at com.cloudera.server.web.cmf.EventsController.query(EventsController.java:167)at com.cloudera.server.web.cmf.EventsController$$FastClassBySpringCGLIB$$690db1aa.invoke(<generated>)at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvo