Find answers, ask questions, and share your expertise
Recently active
we want to find the **approach / test /cli / API** that gives us the results about HeartBeat Lost between Ambari agent to Ambari serverHeartBeat Lost could be as results of poor connection between Ambari agent to Ambari server or because Ambari server was down for along time , etc Note - from Ambari GUI the machine with **HeartBeat Lost** state usually colored by yellow state clarification: the case as described here appears when `ambari-agent status` is in running state as the followingambari-agent statusFound ambari-agent PID: 119315ambari-agent running.Agent PID at: /run/ambari-agent/ambari-agent.pidAgent out at: /var/log/ambari-agent/ambari-agent.outAgent log at: /var/log/ambari-agent/ambari-agent.log
Hi Team,Can someone please help with best practices for tuning queue manager in CDP? Thanks!
Hello, I am trying to use hbase-spark in order to query over Hbase with spark-sql, but I am stuck with some of these exceptions: java.lang.NullPointerExceptionor java.lang.NoSuchMethodError: org.apache.hadoop.hbase.util.ByteStringer.wrap([B)Lcom/google/protobuf/ByteString; Contents : 1- Details on the platform 2- Details on the problem3- Description of the attached content 1- A Cloudera cluster running CDH-6.3.4 2- I run a (POC) Java application with spark-submit in yarn cluster mode. What the application does is to sequencially : - Create a Hbase table and populate it using the Java API - Use a spark-session to read the hbase table (using .format("org.apache.hadoop.hbase.spark") ) - Perform some queries on the Dataframe. For now, I only got partial success on the last step. I can show the contents of the dataframe with this for example : Dataset<Row> sqlDF1 = sqlContext.sql("SELECT * FR
Hello.I would like to help about check the history log of sql query command of each account on Impala and Hive. I have been check on `/var/log/hive` (the Hive log) but not expected.Can you figure out for me where is the log save on ? And any have another way to check it ?.Thank you for you help.
ERROR orm.CompilationManager: Sqoop requires a JDK that can compile Java code cloudera
It looks like CDH 7.1.7 SP1 is vulnerable to CVE-2022-42889. Here is the announcement from Apache which indicates the mitigation is to "Upgrade to Apache Commons Text 1.10.0". https://lists.apache.org/thread/n2bd4vdsgkqh2tm14l1wyc3jyol7s1om There was another community thread about Text4Shell in NiFi, but CVE-2022-42889 is NOT just a NiFi issue. CDH 7.1.7 SP1 (even p1057) includes the vulnerable common-jars 1.6/1.7 and 1.9. /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/jars/commons-text-1.6.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/jars/commons-text-1.7.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/jars/commons-text-1.9.jar.../opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/lib/solr/server/solr-webapp/webapp/WEB-INF/lib/commons-text-1.6.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1057.32088321/lib/streams_messaging_manager/libs/commons-text-1.9.jar/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.
Hi Team, Spark job is hang or struck due to below error. can any one please help here.. 22/12/05 22:29:55 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5e29988e0x0, quorum=localhost:2181, baseZNode=/hbase22/12/05 22:29:55 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)22/12/05 22:29:55 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Connection refusedat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1126).... 22/12/05 22:30:12 INFO zookeeper.ClientCnxn: Op
Hi guys, do you know how to get the version mapping such as hadoop between cdh and apache community? Just like how to get the difference of hadoop between hadoop-cdh5.16.2 and 2.6.0-cdh5.5.0.
Starting Apache NIFI 1.16.0 on windows 10 , with jdk 1.8.0_45 installed is failing to start with the following error in nidi-app.log:Caused by: java.lang.IllegalStateException: Flow controller TLS configuration is invalid at org.apache.nifi.controller.FlowController.<init>(FlowController.java:501) at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:403) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:86) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169) &n
When trying to connect to a dblink to user the QueryDatabaseTable, NiFi is returning this error:• 14:24:36 BRT ERRORQueryDatabase Table[id=c3576641-0184-1000-ad73-c81ab57c11bb]QueryDatabase Table[id=c3576641-0184-1000-ad73-c81ab57c11bb] failed toprocess session due to Cannot invoke method public default java.sql.Connectionorg.apache.nifi.dbcp.DBCPService.getConnection(java.util.Map) throwsorg.apache.nifi.processor.exception. Process Exception on Controller Service withbecause the Controlleridentifier bf90fd3d-0184-1000-4a92-d8b25677a705Service's State is currently ENABLING; Processor Administratively Yielded for 1sec: org.apache.nifi.controller.service.Controller Service DisabledException: Cannotinvoke method public default java.sql.Connectionorg.apache.nifi.dbcp.DBCPService.getConnection(java.util.Map)throwsorg.apache.nifi.processor.exception. Process Exception on Controller Service withidentifier bf90fd3d-0184-1000-4a92-d8b25677a705 because the ControllerService's State is curr
Hi, I want ask a shared variable between non-associated Processors. I have 2 separate Process Groups.One group monitors the status of the API called log manager, and changes the variable 'Circuit_Break' to 'enable' through UpdateAttribute when the API status is bad. Another group receives syslog as udp and put it in an API called log manager. At this time, if the variable called 'Circuit_Break' is 'enable' through RouteOnAttribute in the middle, alert with slack and try to execute ControlRate. thisis Group 2 thisis Group 1 Is it possible to use variable shared by different process groups?If not available, are there any other good ways? I would like to periodically monitor the API and give a slight delay according to the API status.When the API is in a bad state, I want to give it time to recover by adjusting the throughput(by ControlRate).
I am getting below input flow file: Input:[ {"original" : {"Id" : "100008","Date" : "2022-09-22"},"enrichment" : {"Record" : "Record 11","Description" : "Invalid values for Id or Date"}}, {"original" : {"Id" : "100009","Date" : "2022-09-02"},"enrichment" : {"Record" : "Record 2","Description" : "Invalid values for Id or Date"}} ] Expected output:[{"original" : {"Id" : "100009","Date" : "2022-09-02"},"enrichment" : {"Record" : "Record 2","Description" : "Invalid values for Id or Date"}}] and Output I am getting is the below using Groovy script:["{\"original\":{\"Id\":\"100009\",\"Date\":\"2022-09-02\"},\"enrichment\":{\"Record\":\"Record 2\",\"Description\":\"Invalid values for Id or Date\"}}"] Groovy Script:import org.apache.commons.io.IOUtilsimport java.nio.charset.StandardCharsetsimport groovy.json.*import java.util.ArrayListdef flowFile = session.get()if(!flowFile) returntry {flowFile = session.write(flowFile,{ inputStream, outputStream ->def text = IOUt
Hi , I am new to the groovy... I want to know, how json array can be formed for dynamic variables from json flow file thorough groovy script.. please let me know, if you have an idea
Can you help me to find where do Nifi stores templates?I am running Nifi on a docker container.PS: there is no dir under the folder conf.Here is the WORKDIR:
SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]2022-12-05 13:22:59: Starting HiveServer2WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]S
I have set up Cloudera Manager (Trial) 7.4.4 in Ubuntu 20.04.4 LTS (Focal Fossa) as a single node cluster in AWS EC2. All the services HDFS, Hue , Hive , Spark , Livy , Solr , ZooKeeper , Yarn are in good health and running. I can login to Hue editor and open spark notebook. But I'm getting the issue as The Spark session could not be created in the cluster when trying to execute notebook. I The Spark session could not be created in the cluster: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(Spark
I have requirement to execute a PostgreSQL procedure in Nifi, I tried to use Put Sql processor , but ended up getting error. Could someone let me know what I'm doing wrong. This is how my entire flow looks like. -querying from a table and putting the record to another stage table using the first 'PutSql' processor. And afterwards the second putsql processor to call my stored procedure.
I am trying to re-create a ranger-audits collection for Infra-Solr with numShards=2 instead of 1. # curl -k --negotiate -u : "https://fqdn.example.net:8995/solr/admin/collections?action=CREATE&name=ranger_audits&collection.configName=ranger_audits&numShards=2" {"responseHeader":{"status":400,"QTime":1730},"failure":{"fqdn.example.net:8995_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at https://ufqdn.example.net:8995/solr: Error CREATEing SolrCore 'ranger_audits_shard2_replica_n2': Unable to create core [ranger_audits_shard2_replica_n2] Caused by: Could not initialize class java.time.zone.ZoneRulesProvider"},"success":{"fqdn.example.net:8995_solr":{"responseHeader":{"status":0,"QTime":1278},"core":"ranger_audits_shard1_replica_n1"}},"Operation create caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: ranger
I want to collect the metadata from HDFS. But when i searched , it looks like there is no atlas hook for HDFS available like it is there for kafka , hive and other data sources. Can I get the HDFS hook install steps or the steps to create a custom hook.
Hello guys, I having a problem with Airflow on CDE. This make me boring almost a week. I am using the ECS containers on Data Services 1.3.4-b211. On CDE, I can run a Spark job, open a Grafana UI, but Airflow UI can not. Thanks for viewing my issue Any advice would be greatly appreciated
Hello, unfortunately I couldn't find an answer for this question, although it seems to be easy enough. I have a json file of this format, which I want to generate separate flowfiles from singularizing the values in NestedKey2: My input:{"Key1" : [ {"NestedKey1" : "Value1","NestedKey2" : [ "Value2", "Value3", "Value4", "Value5"],"NestedKey3" : "Value6"} ]} Desired output:Flowfile 1:{"Key1" : [ {"NestedKey1" : "Value1","NestedKey2" : "Value2","NestedKey3" : "Value6"} ]} Flowfile 2:{"Key1" : [ {"NestedKey1" : "Value1","NestedKey2" : "Value3","NestedKey3" : "Value6"} ]} ... and so on. Thanks for any suggestions.
I have installation of ECS (Embedded Container Service / Experience Compute Service ) v 1.3.3 , CDP 7.1.7.1000. Cloudera ML and DWX were ok to start and use. But, CDE (Data Engineering XP) service always fail to start with log saying {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstalllationFailed","Attempt":"3"}. Not sure what is best way to start troubleshooting with such log displyed only.
Hi guys, I am able to connect to my Hive2 server with the following command: beeline jdbc:hive2://<hostname>:10001/;ssl=true;sslTrustStore=/etc/hadoop/conf/hive-truststore.jks;trustStorePassword=<redacted>;transportMode=https;httpPath=cliservice However, when I try to connect with Hue I get the following error: 500 Server Error: Server Error for url: https://<hostname>:10001/cliservice/ <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /cliservice/. Reason: <pre> java.lang.IllegalArgumentException: Invalid sign, original = <long string> current = <another long string></pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.25.v20180904</a><hr/> </body> </html> (error 500) I
I need to change the column order in a Hive table, but I want to make sure I can do it without deleting any of the existing data.
Hello,We are able to login to Hue UI locally with default user name / password, the issue arises when choosing AD Authentication it does not accept the username / password. Error in Hue Access log[25/Nov/2022 14:00:29 -0800] DEBUG Initiating TLS[25/Nov/2022 14:00:29 -0800] DEBUG search_s('DC=mtest,DC=com', 2, '(&(sAMAccountName=%(user)s)(sAMAccountName={0}))') returned 0 objects:[25/Nov/2022 14:00:29 -0800] DEBUG Authentication failed for User1_dev: failed to map the username to a DN.[25/Nov/2022 14:00:29 -0800] WARNING 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" --- Failed login for user: User1_dev[25/Nov/2022 14:00:29 -0800] INFO 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" returned in 1062ms 200 114908[25/Nov/2022 14:00:40 -0800] DEBUG 10.216.176.10 -anon- - "POST /hue/accounts/login HTTP/1.1" -Error in Runcpserver.log[25/Nov/2022 14:43:01 -0800] resource ERROR Error logging return call POST http://localhost:7187/api
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.