Solved

ERROR orm.CompilationManager: Sqoop requires a JDK that can compile Java code cloudera

It looks like CDH 7.1.7 SP1 is vulnerable to CVE-2022-42889.   Here is the announcement from Apache which indicates the mitigation is to "Upgrade to Apache Commons Text 1.10.0".   https://lists.apache.org/thread/n2bd4vdsgkqh2tm14l1wyc3jyol7s1om   There was another community thread about Text4Shell in NiFi, but CVE-2022-42889 is NOT just a NiFi issue.   CDH 7.1.7 SP1 (even p1057) includes the vulnerable common-jars 1.6/1.7 and 1.9.   /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/jars/commons-text-1.6.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/jars/commons-text-1.7.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/jars/commons-text-1.9.jar.../opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/lib/solr/server/solr-webapp/webapp/WEB-INF/lib/commons-text-1.6.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/lib/streams_messaging_manager/libs/commons-text-1.9.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.

Hi Team, Spark job is hang or struck due to below error. can any one please help here.. 22/12/05 22:29:55 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5e29988e0x0, quorum=localhost:2181, baseZNode=/hbase22/12/05 22:29:55 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)22/12/05 22:29:55 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Connection refusedat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1126).... 22/12/05 22:30:12 INFO zookeeper.ClientCnxn: Op

Hi guys, do you know how to get the version mapping such as hadoop between cdh and apache community? Just like how to get the difference of hadoop between hadoop-cdh5.16.2 and 2.6.0-cdh5.5.0.

When  trying to connect to a dblink to user the QueryDatabaseTable, NiFi is returning this error:• 14:24:36 BRT ERRORQueryDatabase Table[id=c3576641-0184-1000-ad73-c81ab57c11bb]QueryDatabase Table[id=c3576641-0184-1000-ad73-c81ab57c11bb] failed toprocess session due to Cannot invoke method public default java.sql.Connectionorg.apache.nifi.dbcp.DBCPService.getConnection(java.util.Map) throwsorg.apache.nifi.processor.exception. Process Exception on Controller Service withbecause the Controlleridentifier bf90fd3d-0184-1000-4a92-d8b25677a705Service's State is currently ENABLING; Processor Administratively Yielded for 1sec: org.apache.nifi.controller.service.Controller Service DisabledException: Cannotinvoke method public default java.sql.Connectionorg.apache.nifi.dbcp.DBCPService.getConnection(java.util.Map)throwsorg.apache.nifi.processor.exception. Process Exception on Controller Service withidentifier bf90fd3d-0184-1000-4a92-d8b25677a705 because the ControllerService's State is curr

Hi, I want ask a shared variable between non-associated Processors. I have 2 separate Process Groups.One group monitors the status of the API called log manager, and changes the variable 'Circuit_Break' to 'enable' through UpdateAttribute when the API status is bad. Another group receives syslog as udp and put it in an API called log manager. At this time, if the variable called 'Circuit_Break' is 'enable' through RouteOnAttribute in the middle, alert with slack and try to execute ControlRate. thisis Group 2 thisis Group 1  Is it possible to use variable shared by different process groups?If not available, are there any other good ways? I would like to periodically monitor the API and give a slight delay according to the API status.When the API is in a bad state, I want to give it time to recover by adjusting the throughput(by ControlRate).  

I am getting below input flow file: Input:[ {"original" : {"Id" : "100008","Date" : "2022-09-22"},"enrichment" : {"Record" : "Record 11","Description" : "Invalid values for Id or Date"}}, {"original" : {"Id" : "100009","Date" : "2022-09-02"},"enrichment" : {"Record" : "Record 2","Description" : "Invalid values for Id or Date"}} ] Expected output:[{"original" : {"Id" : "100009","Date" : "2022-09-02"},"enrichment" : {"Record" : "Record 2","Description" : "Invalid values for Id or Date"}}] and Output I am getting is the below using Groovy script:["{\"original\":{\"Id\":\"100009\",\"Date\":\"2022-09-02\"},\"enrichment\":{\"Record\":\"Record 2\",\"Description\":\"Invalid values for Id or Date\"}}"]  Groovy Script:import org.apache.commons.io.IOUtilsimport java.nio.charset.StandardCharsetsimport groovy.json.*import java.util.ArrayListdef flowFile = session.get()if(!flowFile) returntry {flowFile = session.write(flowFile,{ inputStream, outputStream ->def text = IOUt

Hi , I am new to the groovy... I want to know, how json array can be formed for dynamic variables from json flow file thorough groovy script.. please let me know, if you have an idea 

Can you help me to find where do Nifi stores templates?I am running Nifi on a docker container.PS: there is no dir under the folder conf.Here is the WORKDIR: 

SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]2022-12-05 13:22:59: Starting HiveServer2WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]S

I am trying to re-create a ranger-audits collection for Infra-Solr with numShards=2 instead of 1.     # curl -k --negotiate -u : "https://fqdn.example.net:8995/solr/admin/collections?action=CREATE&name=ranger_audits&collection.configName=ranger_audits&numShards=2" {"responseHeader":{"status":400,"QTime":1730},"failure":{"fqdn.example.net:8995_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at https://ufqdn.example.net:8995/solr: Error CREATEing SolrCore 'ranger_audits_shard2_replica_n2': Unable to create core [ranger_audits_shard2_replica_n2] Caused by: Could not initialize class java.time.zone.ZoneRulesProvider"},"success":{"fqdn.example.net:8995_solr":{"responseHeader":{"status":0,"QTime":1278},"core":"ranger_audits_shard1_replica_n1"}},"Operation create caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: ranger

I want to collect the metadata from HDFS. But when i searched , it looks like there is no atlas hook for HDFS available like it is there for kafka , hive and other data sources. Can I get the HDFS hook  install steps or the steps to create a custom hook. 

Hello guys, I having a problem with Airflow on CDE. This make me boring almost a week. I am using the ECS containers on Data Services 1.3.4-b211. On CDE, I can run a Spark job, open a Grafana UI, but Airflow UI can not. Thanks for viewing my issue Any advice would be greatly appreciated

I have installation of ECS (Embedded Container Service / Experience Compute Service )  v 1.3.3 , CDP 7.1.7.1000. Cloudera ML and DWX were ok to start and use. But, CDE (Data Engineering XP) service always fail to start with log saying {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstalllationFailed","Attempt":"3"}.   Not sure what is best way to start troubleshooting with such log displyed only.

I need to change the column order in a Hive table, but I want to make sure I can do it without deleting any of the existing data.

Hello,We are able to login to Hue UI locally with default user name / password, the issue arises when choosing AD Authentication it does not accept the username / password. Error in Hue Access log[25/Nov/2022 14:00:29 -0800] DEBUG Initiating TLS[25/Nov/2022 14:00:29 -0800] DEBUG search_s('DC=mtest,DC=com', 2, '(&(sAMAccountName=%(user)s)(sAMAccountName={0}))') returned 0 objects:[25/Nov/2022 14:00:29 -0800] DEBUG Authentication failed for User1_dev: failed to map the username to a DN.[25/Nov/2022 14:00:29 -0800] WARNING 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" --- Failed login for user: User1_dev[25/Nov/2022 14:00:29 -0800] INFO 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" returned in 1062ms 200 114908[25/Nov/2022 14:00:40 -0800] DEBUG 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" -Error in Runcpserver.log[25/Nov/2022 14:43:01 -0800] resource ERROR Error logging return call POST http://localhost:7187/api

Hi everyone I have spent several days trying to solve the following case. I need to join two flowfiles (master-detail) using Nifi flow, I tried using queryrecord and mergerecord but i couldn't did it. I have some json based files in a HDFS (Apache Spark) i need to make a join and put de result in a database table (using nifi v1.11.4). I can read HDFS files but i cant' perform a join.   Example: master_file content: {   "FALLA_ID" : 40217,  "FALLA_FECHA" : "1998-01-01 00:00:00",  "FALLA_RPT_NRO" : 25,  "FALLA_FRECUENCIA" : "60.0000" }   detail_file content: { "FALLA_ID" : 40217,"PRINCIPAL" : 1,"FALLA_DTL_ID" : 1,"FALLA_CLASE" : 1,"TPF_FALLA_ID" : 1 }, { "FALLA_ID" : 40217,"PRINCIPAL" : 1,"FALLA_DTL_ID" : 2,"FALLA_CLASE" : 2,"TPF_FALLA_ID" : 5 }   Expected_File content: {   "FALLA_ID" : 40217,  "FALLA_FECHA" : "1998-01-01 00:00:00",  "FALLA_RPT_NRO" : 25,  "FALLA_FRECUENCIA" : "60.0000",   "PRINCIPAL" : 1,   

Hi all, I'm having problems getting the JavaScript I wrote for ExecuteScript to work properly. What I'm trying to do:My flowfiles contain rows of data from a SELECT statement. Each flowfile is searched line by line for non-ASCII characters, and all non-ASCII characters that are found are replaced with empty strings before the data is loaded to a target database. I'd like to grab the value for the table's unique identifier column and write it to a new table, say "nifi_replaced_data" so that I can easily look in the source database and see the row with the non-ASCII character that NiFi replaced in the target database. For example, I might be processing a table with the columns:column_1, column_2, unique_id, column_4And the flowfile, which is tab delimited, might look like:somedata    bad__character   AA11BB22   someotherdataotherdata no_bad_character CC33DD44 someotherdata The process:Each flowfile is checked in it's entirety for no

I need to create a flow that update rows in my database based on csv file.I have :1- GetFilestatusclient_idActive123Inactive456 2- UpdateAttribute to update the Baobab_file (the schema is already in the AvroSchemaRegistry (collection) 3- SplitRecord to get each line statusclient_idActive123 4- ExecuteSQL : update table_client set current_status = status where id = client_id How can I do this I tried ${status} and \status but no luck Thank you

 Cloudera Manager (Trial) 7.4.4Cloudera Runtime 7.1.7 (Parcels)  Cloudera SCM log: 2022-07-28 19:19:59,113 ERROR CommandPusher-1:com.cloudera.cmf.command.CommandHelpers: GenerateCredentials - Execution error:java.io.IOException: Encountered error with /opt/cloudera/cm/bin/gen_credentials.sh: Cannot access generated keytab file /var/run/cloudera-scm-server/cmf2117942825672975512.keytabat com.cloudera.cmf.security.GenerateCredentialsCommand.generateKeytab(GenerateCredentialsCommand.java:533)at com.cloudera.cmf.security.GenerateCredentialsCommand.createKeytab(GenerateCredentialsCommand.java:637)at com.cloudera.cmf.security.GenerateCredentialsCommand.generateMissingKeytabs(GenerateCredentialsCommand.java:350)at com.cloudera.cmf.security.GenerateCredentialsCommand.access$300(GenerateCredentialsCommand.java:70)at com.cloudera.cmf.security.GenerateCredentialsCommand$2$1.run(GenerateCredentialsCommand.java:745)at com.cloudera.cmf.security.GenerateCredentialsCommand$2$1.run(

Yarn's application priority can be found in yarn 8088 resource manager's website,How do the priority work in yarn? Now the version of CDH I use is 6.3.2.Hadoop 3.0.0 Hive 2.1.1 I use hive on spark.Can I use the config to manage the application priority? For some reason some hive sqls should have high application priority when those sqls are appending and run ahead of the other.  Instead of running the same time and equal share the compute resources.   When I set the following config in Hive, it seems do not work well in Yarn.MapReduce"-Dmapreduce.job.priority=xx"Flink"-yD yarn.applicaiton.priority=xx"Spark"spark.yarn.priority=xx" In hive sql."set spark.yarn.priority=10;"It does not work ...  

Orginal data:{"data": [{"id": "1234569","Date": "2022-08-22"},{"id": "1234567","Date": "2022-08-22"},{"id": "34567","Date": "2022-08-22"}]}API Response:[ {"Record 2" : "Invalid values for emp info"}, {"Record 3" : "Invalid values for emp info"} ]Where Record 1,Record 2,Record 3 is array of json object index. Here we have three records in orginal data, Record 1 passed successfully and remaining two failed and due to that in response we are getting Record 2 and Record 3i.e. if index of orginal data = value of Record (for example original data's index 1 is passed successfully so it should not merge. In orginal Response Index 2 failed, so it should merge with Record 2 and so on.. ) then final output should look like below:{"data": [{"id": "1234567","Date": "2022-08-22","Record 2" : "Invalid values for emp info"},{"id": "34567","Date": "2022-08-22","Record 3" : "Invalid values for emp info"}]}How can I do that? Please help.

Hi Team, I have an requirement: Input:{"headers": {"comments": "3"},"data": [{"Id": "100000","Date": "2022-09-22"},{"Id": "100001","Date": "2022-09-02"}]} Jolt Spec:[{"operation": "shift","spec": {"data": {"*": {"Id": "[&1].Id","Date": "[&1].Date","$": "[&1].array_Ind"}}}}, {"operation": "modify-default-beta","spec": {"*": {"Record": "=concat('Record ',@(1,array_Ind))"}}}] I am getting below output:[ {"array_Ind" : "0","Id" : "100000","Date" : "2022-09-22","Record" : "Record 0"}, {"array_Ind" : "1","Id" : "100001","Date" : "2022-09-02","Record" : "Record 1"} ] Expected output: [ {"array_Ind" : "1","Id" : "100000","Date" : "2022-09-22","Record" : "Record 1"}, {"array_Ind" : "2","Id" : "100001","Date" : "2022-09-02","Record" : "Record 2"} ]   Can you please help 

I am getting info messages like the one below while reading and writing files in hdfs.can anyone explain the meaning of this message, will it cause any issues? I have 3 Namenode(open-source Hadoop) in HA. INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-errorat org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:108)at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2094)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1550)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3342)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1208)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(Clien

I am trying to process a CSV file and convert it to a JSON in a specific format. My CSV file is as followsSTARTPI,0010002,25,king,address,phonePE,3.2,company1PE,1.9,company2STOPSTARTPI,0010003,25,prince,address,phonePE,3.2,company1PE,1.9,company2STOPI want help in extracting records from START till STOP. I was able to do the JSON conversion.