Solved

Unable to execute the Sqoop command after configuring the existing environment into the new environment.we are using Sqoop import/export command to get the data from other DB to the HADOOP. But it's failed with following error. But we can do eval command and list command for same DB.Error: java.lang.RuntimeException: java.sql.SQLException: ORA-12541: Cannot connect. No listener at host prxd10-scan.intranet.slt.com.lk port 1521. (CONNECTION_ID=fnQPj6XGQbKYLTFy6NYkAA==) https://docs.oracle.com/error-help/db/ora-12541/but we are doing some Sqoop job with HADOOP and other DBs.please help me!

Hi, On our Cloudera Cluster version 7.11.3, added new Nifi and nifi registry parcels and csds. Restarted cloudera-scm-server and cloudera-scm-agents and cloudera manager services, but still not seeting any of the nifi or nifi registries when I add service. Nifi version 1.26.0Checked scm-server and scm-agent logs no errors. What else am I missing ,Any idea why ?

Hi Team, I am using 3 - node Nifi cluster, version : 1.23.2. I observed that Nifi throws error for deleting any processor/parameter context if any node is down. Can you please share the reason of it? Is there any document from Apache that clearly list out the restrictions of task list if any one node is down?Thanks,Priyanka

How does the ZKFC running in the namenode server maintain a open session in zookeeper?

Which configuration should I change for set specific NIC for HDFS data-node data replication?I want to separate traffic for Service between Replication. 

1000 requests are queued in NIFI , invoke HTTP processor is showing socket timeout exception , i have added a self loop in invoke http processor so that it keep on retrying . These requests are stuck permanently.My issue is endpoint which invoke http processor is trying to invoke is up and running .  i checked from postman i am able to get success response. 2023-11-08 12:17:34,783 ERROR [Timer-Driven Process Thread-138] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=b48ff3b8-befb-32c2-b670-83445e1873f2] Routing to Failure due to exception: java.net.SocketTimeoutException: connect timed out: java.net.SocketTimeoutException: connect timed outjava.net.SocketTimeoutException: connect timed outat java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketI

Hello,Is there a way to check the replication factor of a particular folder in HDFS?While we have default replication set to 3 in CM for some reason files being uploaded in a particular folder shows up with replication factor of 1. RegardsWert

I've got an instance of hue server on my host configured to use infra-solr, and it's tell me that it's got stale configuration on solr_url because the server is changed. If I restart the hue service for the first minute everything is good, but just some minute later it tells me again that's got stale configuration. The solr_url value isn't changed, there's the same url that was before i restart the service. Seems like it cannot update hue.ini config fileAny help?

Hello,is it possible to create seperate individual list from one list based on some conditions. e.g i have a list that have category & product information so i have to create individual list based on category/item id with excludeInd flag 0/1,List A - CategoryIds List -> Exclude List B - CategoryIds List -> Include List C - ProductIds List -> Exclude List D -ProductIds List -> Include Input{ "correlationId": "3242a7b4-7929-43b3-9fb6-0d7e173373af", "sourceSystemId": "PCS", "messageTimestamp": "2024-05-03T00:00:00", "id": 1234, "promotionId": 2303, "offerId": 2303, "description": "Members Get Tees $15", "customerDescription": "Members Get Tees $15", "levelCode": 0, "typeCode": 3, "templateId": 4, "couponCode": null, "isCouponCodeRequired": 0, "startDatetime": "2024-06-20T00:00:00", "endDatetime": "2024-06-24T23:59:00", "cancelationDatetime": null, "comments": null, "distributionRuleCode": null, "exclusiveDiscountInd": 0, "tiers"

Hi to all,let's say i have a flow file with some given attributes.What i want to achieve is to add flow attribute with a new one those value is retrieved from SQL Server query, where 2 existing attributes is passed in where clause(of course will return only one result) Just for exampleCurrent Flow file:year = 2018brand = audiI would add a third attribute called "model" with value retrieved from a following query:"SELECT model FROM cars WHERE year=${year} AND brand=${brand}" I've searched around but not still found a good answer....Tried evaluating lookup services but still not usable in this scenario. Can anyone can point me on the right direction? Many Thanks

Hello,Could you assist me in retrieving the single branchName value from priceRuleAttributes list based on id, When i am doing without id match its working fine and giving me single branchName value for each index but with id match its combining all values at 1st index.Input  { "Data": { "ID": "09878666", "DATE": "2022-01-01", "ARRAY_ONE": [ { "NAME": "test_1", "priceRuleAttributes": [ { "type": "CFA", "id": "PR_BRANCH_NAME", "values": [ "MEMBERS" ] }, { "type": "CFA", "id": "PR_PRICE_RULE_PROMO", "valueBoolean": true }, { "type": "CFA", "id": "PR_TARGET_SEGMENT", "values": [ "1" ] } ] }, { "NAME": "test_2", "priceRuleAttributes": [ { "type": "CFA", "id": "PR_BRANCH_NAME",

Hi Team, I have been trying to generate a scenario like three separate modules to join through Fork Enrichment with Join Enrichment process.For example:  There are three processor groups A , B and CMy existing code designed like join the A and B modules with Fork Enrichment with Join Enrichment process by adding strategy SQL, and getting the expected result.Now, latest requirement to join the Module C into SQL query with existing flow( SELECT a. id, b.name, c. city FROM A a JOIN B b ON a.id = b.id JOIN C c ON c.id = a.id)Is it possible to implement one original and two enrichment flows with Fork Enrichment with Join Enrichment, or any alternative approachPlease advice.  

Hi everyone!I have json from kafka: [ {"before" : {"old_key1" : old_value1,"old_key2" : old_value2,"old_key3" : old_value3,"old_key4" : old_value4},"after" : {"new_key1" : new_value1,"new_key2" : new_value2,"new_key3" : new_value3,"new_key4" : new_value4},"table" : "table_name","op" : "u", -- operation ['u-update', 'c-insert', 'd-delete']"tm" : 1720539441000000000 - timestamp of changes} ]I need execute on database sql like:update table_nameset new_key1=new_value1, new_key2=new_value2, new_key3=new_value3, new_key4=new_value4where old_key1=old_value1 and old_key2=old_value2 and old_key3=old_value3 and old_key4=old_value4; What's the best way to update rows in a database with before and after values by NiFi. I can`t use a groovy script because our team know sql only and if I produce it on groovy, I must support this alone. I have tried create sql by jolt, but all values without quotes and I can`t  test a type.  I think a have two ways: 1) insert all data t

I need to know if there are any coming updates to using the full functionality(update, delete or create table) of Iceberg from impala on CDP Private Cloud 7.1.9.  

Hi all,I developed my Spark application using Spark 3.4, and it runs on a different platform. Now, I want to run it on another platform that uses Spark 2.4. When I submit the job using the command belowspark-submit --class Main --master yarn --deploy-mode client --num-executors 6 --executor-memory 10G --driver-memory 20G --executor-cores 4 --driver-cores 4 --conf spark.yarn.submit.waitAppCompletion=true --conf spark.executor.extraJavaOptions="-XX:+UseG1GC -XX:MaxGCPauseMillis=20" --conf spark.driver.extraJavaOptions="-XX:+UseG1GC -XX:MaxGCPauseMillis=20" --jars /home/cdsw/mysql-connector-java-5.1.49.jar /home/cdsw/MyApplication.jaran error appears as shown:Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/connector/catalog/TableProvider at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.j

 == sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log2024-07-08 15:10:06,345 WARN scm-web-117:com.cloudera.server.web.cmf.EventsController: (7 skipped) Exception querying eventscom.cloudera.cmon.MgmtServiceLocatorException: There is no configured event storeat com.cloudera.cmon.MgmtServiceLocator.getEventStoreQueryProxy(MgmtServiceLocator.java:314)at com.cloudera.server.web.cmf.events.EventDao.findEvents(EventDao.java:398)at com.cloudera.server.web.cmf.EventsController.query(EventsController.java:167)at com.cloudera.server.web.cmf.EventsController$$FastClassBySpringCGLIB$$690db1aa.invoke(<generated>)at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvo

Hi .. I have nifi running with ldap-user-group-provider integrated with AD. However, there are some usernames with mixed case sAMAccountName, lets say `sAMAccountName=Michael`.Logging into Nifi as `Michael` works (permission already given to an AD group which this user is a member of), but when logging into NiFi as `michael` or any username that does not match with the exact case, while the login is successful, i'm getting insufficient permission error.have tried to set group membership enforce case sensitivity to false, but seems like did not work, any steps I might have missed?and extract from authorizers.xml:<property name="Page Size">300</property><property name="Group Membership - Enforce Case Sensitivity">false</property><property name="Sync Interval">10 mins</property><property name="User Search Base">DC=company,DC=org</property><property name="User Search Scope">SUBTREE</property><property name="User Identity A

Hi there,Maybe you guys can shed a light on an issue we’ve been having since we went live with one of our flows using alot of processors running on the primary node. We are using a 2 node cluster where we run a couple of getsqs (time driven, 1min) and executesql (cron scheduling, spread out during the day both on primary node. We do this so only one task is ran resulting in just on trigger flow file. Other wise we get 2 sqs events for the same event and 2 executesqls queries.What we have been seeing is that the jvm heap utilization on the primary node is above 70%, this increased gradually the past few days from 25%, so didn’t peak straight to 70%. The secondary node on the other hand is on a stable below 10%. So quite clear that there is a correlation between the processors running only on primary and the heap used.Our questions are:1. When a primary node processor triggers, will the flow file be processed by the following processors in the flow only on the primary node? Or can t

HiAny expected date for a possible SP1 for cdp 7.1.9 ?Is 7.1.9 / 7.11.3 compatible with RHEL 8.10 ?

I have downloaded some huge LLM Model files from huggingface to local filesystem.How can i shared them between CML Projects，so that it looks like exist under /home/cdsw/models?

Hello,We are on CDP PB 7.1.9 and goal is to monitor YARN applications and performance along with few HDFS metrics on Grafana via dashboards and ultimately trigger alerts. Metrics has to be collected via Prometheus agent and shared with Grafana.At this point, I have downloaded a Prometheus agent and created a below yml configuration. I understand this will collect all the metrics, which is intentional to start with.lowercaseOutputName: truerules: # All Gauge type Hadoop Metrics - pattern: 'hadoop<name=.*><>(Count|Value)' name: hadoop_${1}_gauge type: GAUGE # All Counter type Hadoop Metrics - pattern: 'hadoop<name=.*><>(Count|Value)' name: hadoop_${1}_counter type: COUNTERPrometheus agent jar and this config file is stored at /var/lib/prometheus_jmx_config/.For now testing this only on resource manager instances to verify if metrics are getting collected.Following few articles and Grafana documentation, I understood that I will need to run Promethe

How can we add Local Users in Hue when HUE is integrated with AD? Attempting to add a user using add/sync LDAP user returns an error stating no user found in LDAP. We have some users who are not on AD we would them to be added to HUE as Local Users. Have searched but could not get any information on thisCM 7.6.1Cloudera Runtime - 7.1.7 Appreciate any input on this. Thanks 

Hi Team,We are using Hadoop version 2.10.2 version and facing below error while starting  server. Due to company security policy no execution permission is enabled on /tmp directory.   Is there any better alternate way by which we can resolve this?  I am new to Hadoop project and help is much appreciated. Many Thanks in advance.Logs Message:  2024-06-13 09:57:00,871 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManagerjava.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-7115603708108814032.8: /tmp/libleveldbjni-64-1-7115603708108814032.8: failed to map segment from shared object]at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.

When the wizard generates credentials, it reports Insufficient access (50) ldap error, like this:/opt/cloudera/cm/bin/gen_credentials_ad.sh failed with exit code 50 and output of <<+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf3782202571582054951.keytab+ PRINC=HTTP/cradle3302t.priv.cwru.edu@ADS.CASE.EDU+ USER=PruWGPfsVZ+ PASSWD=REDACTED+ DELETE_ON_REGENERATE=true+ SET_ENCRYPTION_TYPES=false+ ENC_TYPES_MASK=4+ USERACCOUNTCONTROL=66048+ ACCOUNTEXPIRES=0+ OBJECTCLASSES='objectClass: topobjectClass: personobjectClass: organizationalPersonobjectClass: user'+ EXTRA_ATTRIBUTES=+ DIST_NAME='CN=PruWGPfsVZ,OU=cradle33,OU=Hadoop,OU=Research Computing,OU=Information Technology Services,OU=Delegate

I know I can run 'hbase version' in command line to check the HBase Version.Where can I find the version of hbase from Cloudera Manager?