Q&A

Hi,   I have a requirement where-in I have to take a count(*) using selecthiveQL and if count is 0 then I have to insert a row.  My flow is selectHive QL → RouteOnAttribute where in I am trying to capture the 0  Json is as below: [ {"_c0" : 0} ] can some one please tell me how to capture this zero, I tried multiple ways but not able to.  Also if anyone has better solution pls let me know. This is quite urgent. 

I have Nifi 1.9.2 and I'm trying to run it into 3 different ports on the same machine (Localhost). I can't seem to find Anything on the internet that can help me with that.Can we do this?I need your help guys, Thanks in advance

The version of CDH5.16.1 I used recently has the following error, which makes it impossible to use the cm management platform to execute some commands, such as distributing the Gateway configuration. I hope to provide some solutions. 2019-11-07 14:00:02,161 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.AbstractRoleHandler: (1 skipped) Unable to generate configuration for GATEWAY ba se group 2019-11-07 14:00:02,161 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.AbstractRoleHandler: Unable to generate configuration for HIVESERVER2: hive-HIVE SERVER2-0f0c09243092e98c6d762d9b05f72d25 2019-11-07 14:00:02,163 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Failed to invoke command (GlobalPoolsRefresh) for schedu le (1) java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:968) at

Hi, everyone, my CDH cluster can't execute commands on the cm management platform, such as pushing the client configuration, killing the jobs on the yarn, etc. The error log of the server is as follows. Thank you for your suggestions. Thank you. com.cloudera.cmf.command.CmdExecException: com.cloudera.cmf.command.CmdExecException: java.lang.RuntimeException: java.lang.NullPointerException A server error has occurred. The full stack trace is not shown here due to security reasons. See Cloudera Manager Server log for details. WARN 986168735@scm-web-253:com.cloudera.cmf.command.flow.CmdStep: Unexpected exception during command work com.cloudera.cmf.command.CmdExecException: java.lang.RuntimeException: java.lang.NullPointerException at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:119) at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885) at com.cloude

Hi, We are planing for CDP cluster deployment for Ms Azure cloud , Any link to get the repo file 

This problem has been bothering me for a long time The oozie role on Cloudera Manager starts to report an error. The information is as follows. How do I troubleshoot it, and where is oozie.keytab? , which logs can be queried to help solve? SERVER[xxxx.com] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], failure to login: for principal: oozie/xxxx.com@XXXX.COM from keytab oozie.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24) - PREAUTH_FAILED

I run a PC with Ubuntu 16.04 LTS. I have installed CDH 5.16.1 (installation by parcels) and had accidentally allowed it to install Oracle Java 7 into my default Java directory (/usr/lib/jvm). Prior to my CDH installation I had installed OpenJDK 8 on my PC in the same location.    I then downloaded the JDBC for Postgresql for Java 8 and higher postgresql-42.2.5.jar and placed it in the recommended directory (/var/lib/sqoop).   In attempting to connect to my database and list the tables present, I got the following error:   Exception in thread "main" java.lang.UnsupportedClassVersionError: org/postgresql/Driver : Unsupported major.minor version 52.0at java.lang.ClassLoader.defineClass1(Native Method)at java.lang.ClassLoader.defineClass(ClassLoader.java:800)at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)at java.net.URLClassLoader.access$100(URLClassLoader.java:71)at java.net.

I install Kafka in Ambari and configure two listeners for every broker, one for inner traffic and one for outer traffic. E.g., broker A has two ip addresses: 192.168.1.1 and 88.88.88.88, and its configuration is like follows:   SASL_PLAINTEXT://192.168.1.1:9092,EXTERNAL://88.88.88.88:19092listener.security.protocol.map=SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXTsecurity.inter.broker.protocol=SASL_PLAINTEXT   With this configurations, Kafka works fine but keeps alerting, the message is:   Connection failed: [Errno 111] Connection refused to 192.168.1.1:19092    This is not valid because ip and port do not match. And I suspect that it could just split by colon and use the last element. So I exchange the order of the two listeners:   EXTERNAL://88.88.88.88:19092,SASL_PLAINTEXT://192.168.1.1:9092   To my surprise, it still alerts and the message changes to    Connection failed: [Errno 111] Connection refused to 88.88.88.88:9092  

Hi, We are getting the error while executing Hive query on spark as execution engine. Hive version: 1.2.1, Spark version : 1.6   set hive.execution.engine=spark;set spark.home=/usr/hdp/current/spark-client;set hive.execution.engine=spark;set spark.master=yarn-client;set spark.eventLog.enabled=true;set spark.executor.memory=512m;set spark.executor.cores=2;set spark.driver.extraClassPath=/usr/hdp/current/hive-client/lib/hive-exec.jar;   Query ID = svchdpir2d_20191106105445_a9ebc8a2-9c28-4a3d-ac5e-0a8609e56fd5Total jobs = 1Launching Job 1 out of 1In order to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number>In order to limit the maximum number of reducers:set hive.exec.reducers.max=<number>In order to set a constant number of reducers:set mapreduce.job.reduces=<number>Starting Spark Job = c6cc1641-20ad-4073-ab62-4f621ae595c8Status: SENTFailed to execute spark task, with exception 'java.lang.IllegalStateExcept

hive metastore is consuming complete heap(tried max heap : 24G) after sometime  the metastore is not responding , the metastore is ok if we disable ranger hive authentication  this issue is reported in (https://jira.apache.org/jira/browse/HIVE-20568) ,It seems ranger plugin is causing memory leaks during audit when it encounters hive db names containing '_' just wondering to know if there is a work around for it in HDP-3.1.0 or the upgrade from HDP-3.1 to HDP-3.1.4 will fix it,  Thanks in advance. Current Version  and config Information :========================HDP-3.1hive-3.1Hive Authentication - RangerJDK-1.8mysql -7.5mysql-connector-8.0.13.jar    

I'm running a from-scratch cluster on AWS EC2. I have an external table (partitioned) defined with data on S3. I'm able to query this table and receive results to the console with a simple select * statement:hive> set hive.execution.engine=tez;hive> select * from external_table where partition_1='1' and partition_2='2';[correct results returned]Running a query that requires Tez doesn't return the results to the console:hive> set hive.execution.engine=tez;hive> select count(*) from external_table where partition_1='1' and partition_2='2';Status: Running (Executing on YARN cluster with App id application_1572972524483_0012)OK+------+| _c0 |+------++------+No rows selected (8.902 seconds)However, if I dig in the logs and on the filesystem, I can find the results from that query:(yarn.resourcemanager.log) org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1572972524483_0022

/usr/local/Hbase/bin# ./hbase shellJava HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by org.jruby.java.invokers.RubyToJavaInvoker (file:/usr/local/Hbase/lib/jruby-complete-1.6.8.jar) to method java.lang.Object.registerNatives()WARNING: Please consider reporting this to the maintainers of org.jruby.java.invokers.RubyToJavaInvokerWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future releaseArgumentError: wrong number of arguments (0 for 1)method_added at file:/usr/local/Hbase/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10method_added at file:/usr/local/Hbase/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129Pattern at file:/usr/local/Hbase/lib

After upgrading the Cloudera Manager from 6.2.0 to 6.3.1, the CDSW won't come online anymore. In the health logs, this line stands out:   MaxRetryError: HTTPSConnectionPool(host='xxx.xxx.xxx.xxx', port=6443): Max retries exceeded with url: /api/v1/secrets (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f364478d210>: Failed to establish a new connection: [Errno 111] Connection refused',))  The port is closed on the CDSW master node, and I couldn't find any service in the docs that depends on it. Is it Kubernetes? Further, from cdsw validate it seems that some chains are missing from iptables: The following chains are missing from iptables: [KUBE-EXTERNAL-SERVICES, WEAVE-NPC-EGRESS, WEAVE-NPC, WEAVE-NPC-EGRESS-ACCEPT, KUBE-SERVICES, WEAVE-NPC-INGRESS, WEAVE-NPC-EGRESS-DEFAULT, WEAVE-NPC-DEFAULT, WEAVE-NPC-EGRESS-CUSTOM]  However, I cannot remember any step in the installation process that required to set such rules, so I a

I'd like to use API to set CM monitoring thresholds so I can deploy the modified ones on other CMs.  For example, to set 'Cloudera Manager Agent Log Directory Free Space Monitoring Absolute Thresholds' Warning at 5GB and Critical to 2GB instead of using the default values.   I am not sure with  API to use by looking at  this webpage: https://archive.cloudera.com/cm6/6.0.0/generic/jar/cm_api/apidocs/index.html   Would anyone provide some help?   Thanks, Kevin

How do I add the queue name to the Apache Zeppelin JDBC interpreter settings?

Hi,   We are currently using HDP 2.6.4 image for our development environment and plan to use the same for client deliveries in production environment. This HDP image supports Java8 whereas the company standard says we need support for Java11 for any client deliveries. How can we tackle this problem if we plan to use HDP in our production environment?

Hello ,I installed Ambari server and Ambari agent but I have  ten day that I try to connect ,I receive this error in ambari-agent.log:  INFO 2019-11-06 09:46:39,682 NetUtil.py:62 - Connecting to https://localhost:8440/caWARNING 2019-11-06 09:46:39,683 NetUtil.py:93 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refusedWARNING 2019-11-06 09:46:39,683 NetUtil.py:116 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds...INFO 2019-11-06 09:46:49,683 NetUtil.py:62 - Connecting to https://localhost:8440/caWARNING 2019-11-06 09:46:49,684 NetUtil.py:93 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refusedWARNING 2019-11-06 09:46:49,685 NetUtil.py:116 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds...  And in Ambari-server I have:  05 Nov 2019 11:28:49,329 INFO [ambari-hearbeat-monitor] TopologyManager:487 - Hearbeat for host localhost.localdomain l

Hello! This is a very rudimentary request perhaps, so fair warning! However, recently I installed CFM and had NiFi working fine without NiFi Toolkit CA and NiFi Registry services.I wanted to add LDAP authentication, so I removed the NiFi service, and then added the 3 services in the order that is recommended by the CFM 1.0.1 Documentation, and proceeded to configure the services for LDAP. Now, the NiFi Node fails to stay online. Everything else seems to be in order, but NiFi Node shuts off very quickly after it reports good health in SCM.My first guess is that it could be an issue with Registry, because I am using only 1 CFM host, yet I have all 3 services running on the same, and additionally I am using the same host as the NiFi Gateway. I did not read anything about this in the documentation, but perhaps I've overlooked something simple due to my lack of experience. Did I mess something up by setting the Gateway as the same host as the NiFi node?Any info you can offer to help me trou

Hello all,   I am having an odd issue with NIFI. The cluster stands, and it works but I am unable to log in. I am getting "Untrusted proxy CN=server.domain.com, OU=NIFI" That is not the OU for the zookeeper node I am using. It should be OU=nifi_test. I discovered this in nifi-user.log:   2019-10-28 14:36:26,958 INFO [NiFi Web Server-132] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=nifi_admin, OU=nifi_test) GET <a href="https://server.domain:port/nifi-api/flow/current-user" target="_blank">https://server.domain:port/nifi-api/flow/current-user</a> (source ip: ipaddress)2019-10-28 14:36:26,962 INFO [NiFi Web Server-132] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=nifi_admin, OU=nifi_test2019-10-28 14:36:27,273 INFO [NiFi Web Server-133] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=nifi_admin, OU=nifi_test><CN=server.domain:port, OU=NIFI>) GET <a href="https://server.domain:port/nifi-api/flow

I just noticed that the SmartSense Activity Explorer Zeppelin Notebooks have been failing to run on a production HDP 2.6.1 cluster. I'm not sure how long the issue has been occurring since the dashboards haven't been used much until now. Whenever we try to run the paragraphs, we immediately get an error about unable to make a connection. No other information is given. We are able to connect to Phoenix through psql.py, so we know Phoenix is working properly, just not the dashboard. We've tried restarting the activity explorer, which hasn't fixed the issue. Has someone seen this issue? Any ideas? I'm including the logs we are seeing below. ==> activity-explorer.log <==2019-11-05 10:34:42,555 INFO [qtp1209702763-1653] NotebookServer:711 - New operation from 10.142.131.4 : 62057 : admin : GET_NOTE : 2BPD7951H2019-11-05 10:34:42,558 WARN [qtp1209702763-1653] VFSNotebookRepo:292 - Get Note revisions feature isn't supported in class org.apache.zeppelin.notebook.repo.VFSNotebookRe

Hello everyone,    I can't complete the first installation of my Cluster on AWS. When I try to install my instances on my nodes, the error given by Cloudera Manager is the following:     HTTP ERROR 502Problem accessing /cmf/process/287/logs. Reason: Connection refused (Connection refused)Could not connect to host.     I know the error is very simple, but I am struggling to find out the solution.   My first thought was about the /etc/hosts file, that it is the following for each host in my cluster:     127.0.0.1 localhost.localdomain localhost13.48.140.49 master.sysdatadigital.it master13.48.181.38 slave1.sysdatadigital.it slave113.48.185.39 slave2.sysdatadigital.it slave213.53.62.160 slave3.sysdatadigital.it slave313.48.18.0 slave4.sysdatadigital.it slave4     Moreover, I checked with the following command the hostname in each node:     python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(sock

Hi all, Is it possible to save the uuid of a processor in an attribute? For example,   i have the processor ExecuteSQL, which has this uuid I wish I could save the uuid like this:id = $ .uuid It's possible? Thanks!

Error: Unable to instantiate class [org.apache.zeppelin.server.LdapRealm] for object named 'ldapRealm'.  The same is true for 'ldapGroupRealm'Zeppelin 0.8 on HDP 3.1.0.0 Configuration:# authentication settingsldapRealm = org.apache.zeppelin.server.LdapRealmldapRealm.contextFactory.environment[ldap.searchBase] = DC=tantor,DC=netldapRealm.userDnTemplate = uid={0},OU=users,DC=tantor,DC=netldapRealm.contextFactory.url = ldap://infra01.tantor.net:389ldapRealm.contextFactory.authenticationMechanism = SIMPLE# general settingssessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManagersecurityManager.cacheManager = $cacheManagersecurityManager.sessionManager = $sessionManagersecurityManager.sessionManager.globalSessionTimeout = 86400000shiro.loginUrl = /api/login Exception: ERROR [2019-11-06 12:49:24,565] ({main} EnvironmentLoader.java[initEnvironment]:146) - Shiro environment initialization failedorg.apache.shiro.config.ConfigurationException: Unable to

I'm trying to load druid-avro-extensions by setting it on druid.extensions.loadList config property, but the broker is not starting beause the extension is not found: Exception in thread "main" io.druid.java.util.common.ISE: Extension [/usr/hdp/current/druid-broker/extensions/druid-parquet-extensions] specified in "druid.extensions.loadList" didn't exist!? Edit: the extensions is not located on the spcified directory, but it should be there since it is a core extension. What can I do to fix this issue and why is not the extension there?   What is happening? Thanks.  

I'm trying to create a custom CUDA-capable engine image with the CDSW GPU-Guideon our airgapped CDSW-Cluster. We have a CDSW Cluster with 1 Master-Node and 2 Woker-Nodes (one of the Worker-Nodes is equipped with NVIDIA GPUs)I used the following command on the GPU-Host, which is also given in CSDW GPU-Guide, to build the CUDA docker image:docker build --network host -t <company-registry>/cdsw-cuda:8 . -f cuda.DockerfileBeforehand we configured these firewall rules on our GPU-Host:http://developer.download.nvidia.com/*http://archive.ubuntu.com/ubuntu/*http://security.ubuntu.com/ubuntu/*Unfortunately I get these error messages: (Full output on stdout is attached)Reading package lists...W: The repository 'http://archive.cloudera.com/kudu/ubuntu/xenial/amd64/kudu xenial-kudu5 Release' does not have a Release file.W: The repository 'http://ppa.launchpad.net/deadsnakes/ppa/ubuntu xenial Release' does not have a Release file.E: Failed to fetch http://archive.cloudera.com/kudu/u