Q&A

Hi, I'm setting up Impala Admission Control.For the user.<username> placement rules, there is "Not recommended" remark "Use the resource pool that matches the username. (Not Recommended)"(https://www.cloudera.com/documentation/enterprise/5-16-x/topics/cm_mc_resource_pools.html)In my use-case, specific limitation for a set (a dozen) of users should be applied. At the same time the group management is relatively hard, therefore, I would prefer the root.[username] approach. I would like to better understand the drawbacks of this approach: are there any technical limitations from Impala point of view, or it is just a bad practice due to it is harder to maintain&support&manage (from Administrator's perspective)?

Hi all,is it possible to move a cluster from a workspace to another workspace?Let's say I have created a cluster in my personal workspace and want to move it to another workspace where more people have access to. Is this possible?Otherwise I have to carefully think about which cluster to spawn in which workspace in advance because it cannot be changed later on.Thanks in advance!Alex

Hi Cloudera Community,  Why Cloudera don't recommended the creation of root.[username] pool on the Resource Pools configuration ? I can't find docummentation realted with this. Best Regards,  

Hi, I foun this error in HUE server each time I pick a query from query history in Hive Editor. Only if the query has NOT expired... Any idea, suggestion?  11:17:02 AMERRORdecoratorsError running fetch_result_sizeTraceback (most recent call last): File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 97, in decorator return func(*args, **kwargs) File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 264, in fetch_result_size response['result'] = get_api(request, snippet).fetch_result_size(notebook, snippet) File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py", line 82, in decorator return func(*args, **kwargs) File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py", line 329, in fetch

Hi all,is there a way to define custom hostnames when deploying a cluster with the WebUI of Cloudbreak or do we have to use the CLI to do that? The documentation just mentions the CLI:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/advanced-cluster-options/content/cb_custom-hostnames.html Is it possible to add the corresponding customDomain entries to the request that the Cloudbreak cluster creation wizard creates at the end without having to switch to the CLI?Thanks!Alex

Hello, is it possible for one single user to allocate more than 50% of memory on cdsw because I'm not able to start a session with more than 50% of memory?Actually this makes sense if this is a limitation from cdsw but most of the time I use cdsw alone in our company and I wanted to ask if there is a parameter which I can set to allocate more memory. Thanks.

1. Created HDF cluster in AWS using cloudbreak 2.9. 2. Deleted cloudbreak stack using AWS CloudFormation - I still can use HDF cluster without cloudbreak.Question is can I import existing HDF cluster in AWS to Cloudbreak?BrMargus Roo

The list of supported CentOS versions for CDH clusters managed by Cloudera Altus Director 2.8 and 6.0 do not include CentOS 7.6, however we have several clusters managed by Director 2.8 running CentOS 7.6. Are the docs out of date?

Hello,I am observing quite strange behaviour of anyDelineatedValue expression when I use it with 'OR' operation.If I use 'AND' operation there is no issues.Input attribute value:org.entitlement.list=AA,CC,DD,FFExpression used in RouteOnAttribte/UpdateAttribute processor :${anyDelineatedValue("${org.entitlement.list}", ","):equals('BB'):or(${anyDelineatedValue("${org.entitlement.list}", ","):contains('EE')})}Above expression is giving ArrayIndexOutOfBoundException for OR operation for the given input (where both the values which expression is looking is not present)If I use 'AND' operation expression works fine. What is the difference here? If I pass BB or EE in the input then expression wont give exception !Looks like defect, can some one please suggest if anything wrong in this?ThanksMahendra

We are receiving Json messages from upstream system via Kafka topic. Requirement is to store these messages into HDFS at certain interval. Since we are storing into HDFS we want to merge certain number of these Records in to single file. As per NiFi documentation we are using "MergeRecords" processor for that. About the in coming Records:##These are the multi-line JSon messages with nested structure.Those are based on the same schema (they are picked from single Kafka topic)Those are validated messages and even NiFi processor is able to parse it. so apparently no issues with JSon messages from Schema point of viewPresent ConfigurationBelow is the snapshot of the Processor Configuration. NiFi version: 1.8 Expected behaviorFor the Above configuration its expected that MergeRecords should have weighted for one of the thresholds i.e. Maximum records(100000) or Maximum Bean size(100KBs). Observed BehaviorBut its observed that bean is getting bundled pretty before either of the threshold is

Hi all,  We are going through the hassle of rebuilding our boxes with CentOS7 as there is no upgrade path from CentOS6. Now I have looked over these Kudu documentation to migrating to multiple kudu master but the problem is the first warning it says it is unsafe to do this so I guess my question would be can I take one master making it a 2 node master then follow the documentation on migrating to a 3 node master setup. what process would I follow to remove the 3rd master other than just deleteing the role as I can only imagine how kudu will handle this. Also we are running kudu 1.4 

Hi All,    We have upgrader he VM resources , like CPU and storage is added to each VM in the Cloudera manager.      we have done this one by one VM , and also we have upgraded the edgeserv as well, where we have proxy is installed to access the datanodes. our appliaction is using the impala api URL to access the data from the datanode via the proxy using edgeserver.   from java side we can see the the error  java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;   and checked the edgeserver logs request is recived in proxy and redirct to the data nodes.   in the data nodes logs we can see error logs   E0327 11:57:09.115049 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)I0327 11:57:09.115571 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13)

I've been trying to install CDS 2.3.4(SPARK2_ON_YARN-2.3.0.cloudera4) on my cluster with CM-5.13.1 + CDH5.13.1, the SPARK2 parcel distributed successfully but activated hung there, what could I do to continue the installation, or how to rollback it ? Appreciate your help!     My environment, CM-5.13.1 CDH-5.13.1 installed with parcels JDK-1.8.0_144 Python2.7 Scala-2.11   SPARK2 csd file and parcel file here， [root@vm ~]# ls -ltr /opt/cloudera/csd/total 28-rw-r--r--. 1 root root 19037 Apr 3 13:58 SPARK2_ON_YARN-2.3.0.cloudera4.jar[root@vm ~]#[root@vm ~]# ls -ltr /opt/cloudera/parcel-repo/total 2140796-rw-r--r--. 1 root root 191904064 Apr 3 13:57 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel-rw-r--r--. 1 root root 41 Apr 3 13:57 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha-rw-r--r--. 1 root root 5171 Apr 3 13:57 manifest.json-rw-r-----. 1 root root 7521 Apr 3 14:00 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.

Hi Everyone,M logging into my hadoop env. after a long time, infact last time when i have changed the password for root login on web-shell-client i stored it and system doesn't accept that changed password any more not sure why and what happened.Now all m trying to figure out is how & where to go , to be able reset my root passwordRegards:Bhasker.VJsglp

I used handlehttprequest / handlehttpresponse: when i got a http request, then call 2 other apis at same time to get result, then merge 2 api result. if I run this service at 1 thread, it is ok, but got 2 errors when I run it via more than 1 thread: 1. because FlowFile did not have an 'http.context.identifier' attribute 2. Failed to respond to HTTP request for StandardFlowFileRecord because FlowFile had an 'http.context.identifier' attribute of but could not find an HTTP Response Object for this identifier any suggestion? thanks in advance.

Hi All,@mattclarke,@mattburgess,@markpayneI want to generate sequence number in my nifi cluster (3 nodes), I was using update attribute processor with store state locally option , but this is not serving my purpose as each node is generating its own value incrementally and this is creating duplicate values while loading data to target table.I would be grateful if i can get alternate solution to achieve this batchid generation in cluster mode.Thanks in advance!!

Hi expertsThe scenario is this:Cluster with one Spark History Server on host Y.Using curl I added another Spark History Server to a host X.I also created a different config group for the new SHS, with it's own specific spark.yarn.history.address with host XThe goal is that all clients will write their logs to the default SHS on host Y and whoever needs the new SHS will have to set it manually in his spark app arguments.Problem is, after adding the new SHS and performing "refresh configs" for spark-clients all of them are getting the new SHS address in their /etc/spark2/conf/spark-defaults.conf (spark.yarn.historyServer.address = X:18081)Where do i set which spark.yarn.historyServer.address the clients will be using ?ThxAdi

I have created a flow to read json messages from kafka, and covert them to avro so that I can use ConvertAvroToParquet. My issue is that I can only store one avro message in a parquet file. Is there away to store multiple records within a parquet file using ConvertAvroToParquet?

While trying to install the shell interpreter for zeppelin (v 0.8.0), I fail with this message:# sudo /usr/hdp/current/zeppelin-server/bin/install-interpreter.sh --name shellOpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/zeppelin/lib/interpreter/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/zeppelin/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/zeppelin/lib/slf4j-simple-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Install shell(org.apache.zeppelin:zeppelin-shell:0.8.0) to /usr/hdp/current/zeppelin-server/interpreter/shell ... or

So ZK is working fine. Perfect. Make sure HBase is down and clear everything:hadoop fs -rm -r /hbase/*echo "rmr /hbase" | zookeeper-client Then try to start HBase. JMS

Hi there! I have different warnings when I try to run Cloudera Express, such as 8 GB Ram, and at least 2 processors... How can I check my configuration settings and in case I have to change it, how can I do it? Is it disallowed because is it suspended? Do I have to power it off before? Do I have to create a new VM from zero again in order to change the configuration settings? Thanks in advance! 

hi allDFSIO is a built-in benchmark tool for HDFS I/O test. we created the Base Directory to Store Test Results/benchmarks/TestDFSIOand when we run the test we get: hadoop jar /usr/hdp/2.6.4.0-91/spark2/jars/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar TestDFSIO -write -nrFiles 16 -fileSize 1GB -resFile /tmp/USER-dfsio-write.txt19/03/31 05:39:57 INFO fs.TestDFSIO: TestDFSIO.1.819/03/31 05:39:57 INFO fs.TestDFSIO: nrFiles = 1619/03/31 05:39:57 INFO fs.TestDFSIO: nrBytes (MB) = 1024.019/03/31 05:39:57 INFO fs.TestDFSIO: bufferSize = 100000019/03/31 05:39:57 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO19/03/31 05:39:59 INFO fs.TestDFSIO: creating control file: 1073741824 bytes, 16 filesjava.io.IOException: Permission denied: user=root, access=WRITE, inode="/benchmarks/TestDFSIO/io_control/in_file_test_io_0":hdfs:hdfs:drwxr-xr-xwhere we are wrong?

Hi,We have to reset the admin ambari password and have tried "ambari-admin-password-reset" but I do not see it on the server. Please advice how can I reset the password.Ambari database - MySQL, Ambari version - 2.7.3 and HDP 3.1Thank youSampath

 Can we take advantage of Hive table partitions when querying with impala Are there any issues or problems we might run into given this scenario. We currently have partitioned hive tables... will we be missing anything if we dont convert to impala tables

If I am in the process of creating multiple flow files during an ExecuteScript processor, and an error occurs before I'm done, I transfer the original flow file to its failure relationship. My doubt, however, is whether or not I have to do anything with the flow files that I've created thus far.Consider, for example, the following groovy:def flowFile = session.get()if (!flowFile) returndef filesDir = flowFile.getAttribute('filesDirectory')// See https://community.hortonworks.com/questions/106878/split-one-nifi-flow-file-into-multiple-flow-file-b.htmldef flowFiles = [] as List<FlowFile>try { // Get a list of all the trades in the portfolio directory. File folder = new File(filesDir); File[] listOfFiles = folder.listFiles(); for (File file : listOfFiles) { if (file.isFile()) { // Create a new flow file with that single trade. def newFlowFile = session.create(flowFile) flowFiles << newFlowFile // p