Solved

I've been trying to install CDS 2.3.4(SPARK2_ON_YARN-2.3.0.cloudera4) on my cluster with CM-5.13.1 + CDH5.13.1, the SPARK2 parcel distributed successfully but activated hung there, what could I do to continue the installation, or how to rollback it ? Appreciate your help!     My environment, CM-5.13.1 CDH-5.13.1 installed with parcels JDK-1.8.0_144 Python2.7 Scala-2.11   SPARK2 csd file and parcel file here， [root@vm ~]# ls -ltr /opt/cloudera/csd/total 28-rw-r--r--. 1 root root 19037 Apr 3 13:58 SPARK2_ON_YARN-2.3.0.cloudera4.jar[root@vm ~]#[root@vm ~]# ls -ltr /opt/cloudera/parcel-repo/total 2140796-rw-r--r--. 1 root root 191904064 Apr 3 13:57 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel-rw-r--r--. 1 root root 41 Apr 3 13:57 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha-rw-r--r--. 1 root root 5171 Apr 3 13:57 manifest.json-rw-r-----. 1 root root 7521 Apr 3 14:00 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.

Hi Everyone,M logging into my hadoop env. after a long time, infact last time when i have changed the password for root login on web-shell-client i stored it and system doesn't accept that changed password any more not sure why and what happened.Now all m trying to figure out is how & where to go , to be able reset my root passwordRegards:Bhasker.VJsglp

I used handlehttprequest / handlehttpresponse: when i got a http request, then call 2 other apis at same time to get result, then merge 2 api result. if I run this service at 1 thread, it is ok, but got 2 errors when I run it via more than 1 thread: 1. because FlowFile did not have an 'http.context.identifier' attribute 2. Failed to respond to HTTP request for StandardFlowFileRecord because FlowFile had an 'http.context.identifier' attribute of but could not find an HTTP Response Object for this identifier any suggestion? thanks in advance.

Hi All,@mattclarke,@mattburgess,@markpayneI want to generate sequence number in my nifi cluster (3 nodes), I was using update attribute processor with store state locally option , but this is not serving my purpose as each node is generating its own value incrementally and this is creating duplicate values while loading data to target table.I would be grateful if i can get alternate solution to achieve this batchid generation in cluster mode.Thanks in advance!!

Hi expertsThe scenario is this:Cluster with one Spark History Server on host Y.Using curl I added another Spark History Server to a host X.I also created a different config group for the new SHS, with it's own specific spark.yarn.history.address with host XThe goal is that all clients will write their logs to the default SHS on host Y and whoever needs the new SHS will have to set it manually in his spark app arguments.Problem is, after adding the new SHS and performing "refresh configs" for spark-clients all of them are getting the new SHS address in their /etc/spark2/conf/spark-defaults.conf (spark.yarn.historyServer.address = X:18081)Where do i set which spark.yarn.historyServer.address the clients will be using ?ThxAdi

I have created a flow to read json messages from kafka, and covert them to avro so that I can use ConvertAvroToParquet. My issue is that I can only store one avro message in a parquet file. Is there away to store multiple records within a parquet file using ConvertAvroToParquet?

While trying to install the shell interpreter for zeppelin (v 0.8.0), I fail with this message:# sudo /usr/hdp/current/zeppelin-server/bin/install-interpreter.sh --name shellOpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/zeppelin/lib/interpreter/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/zeppelin/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/zeppelin/lib/slf4j-simple-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Install shell(org.apache.zeppelin:zeppelin-shell:0.8.0) to /usr/hdp/current/zeppelin-server/interpreter/shell ... or

Hi there! I have different warnings when I try to run Cloudera Express, such as 8 GB Ram, and at least 2 processors... How can I check my configuration settings and in case I have to change it, how can I do it? Is it disallowed because is it suspended? Do I have to power it off before? Do I have to create a new VM from zero again in order to change the configuration settings? Thanks in advance! 

hi allDFSIO is a built-in benchmark tool for HDFS I/O test. we created the Base Directory to Store Test Results/benchmarks/TestDFSIOand when we run the test we get: hadoop jar /usr/hdp/2.6.4.0-91/spark2/jars/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar TestDFSIO -write -nrFiles 16 -fileSize 1GB -resFile /tmp/USER-dfsio-write.txt19/03/31 05:39:57 INFO fs.TestDFSIO: TestDFSIO.1.819/03/31 05:39:57 INFO fs.TestDFSIO: nrFiles = 1619/03/31 05:39:57 INFO fs.TestDFSIO: nrBytes (MB) = 1024.019/03/31 05:39:57 INFO fs.TestDFSIO: bufferSize = 100000019/03/31 05:39:57 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO19/03/31 05:39:59 INFO fs.TestDFSIO: creating control file: 1073741824 bytes, 16 filesjava.io.IOException: Permission denied: user=root, access=WRITE, inode="/benchmarks/TestDFSIO/io_control/in_file_test_io_0":hdfs:hdfs:drwxr-xr-xwhere we are wrong?

Hi,We have to reset the admin ambari password and have tried "ambari-admin-password-reset" but I do not see it on the server. Please advice how can I reset the password.Ambari database - MySQL, Ambari version - 2.7.3 and HDP 3.1Thank youSampath

 Can we take advantage of Hive table partitions when querying with impala Are there any issues or problems we might run into given this scenario. We currently have partitioned hive tables... will we be missing anything if we dont convert to impala tables

If I am in the process of creating multiple flow files during an ExecuteScript processor, and an error occurs before I'm done, I transfer the original flow file to its failure relationship. My doubt, however, is whether or not I have to do anything with the flow files that I've created thus far.Consider, for example, the following groovy:def flowFile = session.get()if (!flowFile) returndef filesDir = flowFile.getAttribute('filesDirectory')// See https://community.hortonworks.com/questions/106878/split-one-nifi-flow-file-into-multiple-flow-file-b.htmldef flowFiles = [] as List<FlowFile>try { // Get a list of all the trades in the portfolio directory. File folder = new File(filesDir); File[] listOfFiles = folder.listFiles(); for (File file : listOfFiles) { if (file.isFile()) { // Create a new flow file with that single trade. def newFlowFile = session.create(flowFile) flowFiles << newFlowFile // p

Kudu rebalance tool crashes - when I run it in command line as well as when I use Cloudera Manager UI. Here is the stderr displayed in the Cloudera Manager: + exec kudu cluster rebalance master1.domain.com,master2.domain.com,master3.domain.com --max_moves_per_server=5 --max_run_time_sec=0 --max_staleness_interval_sec=300terminate called after throwing an instance of 'std::regex_error' what(): regex_error*** Aborted at 1553857242 (unix time) try "date -d @1553857242" if you are using GNU date ***PC: @ 0x7fbfc3637207 __GI_raise*** SIGABRT (@0x3ec00005c50) received by PID 23632 (TID 0x7fbfc5c83a00) from PID 23632; stack trace: *** @ 0x7fbfc5642680 (unknown) @ 0x7fbfc3637207 __GI_raise @ 0x7fbfc36388f8 __GI_abort @ 0x7fbfc3f467d5 __gnu_cxx::__verbose_terminate_handler() @ 0x7fbfc3f44746 (unknown) @ 0x7fbfc3f44773 std::terminate() @ 0x7fbfc3f44993 __cxa_throw @ 0x7fbfc3f99dd5 std::__throw_regex_error() @ 0x931c32 std

We have already 10+ GB logs for Kudu Masters and 30+ GB logs for Kudu Tablet Servers. How to limit the number of log files to keep? The Kudu Configuration Reference mentiones only a limit on a log file size (--max_log_size), but no limit for the number of log files.

Short version   cloudera_scm_server is crashing during final configuration.  I reach Add Cluster - Configuration - Command Details and fail on Formatting the name durectories of the current Namenode and Creating Oozie Database Tables.   It's a hard failure - the cloudera_scm_server crashes.   I'm probably missing some little detail in my setup but haven't found anything yet in either this community or the usual onlne resources.   cloudera_scm_server.log shows:   2019-03-28 17:04:09,030 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2019-03-28T17:04:09.030Z2019-03-28 17:04:09,734 WARN scm-web-92:com.cloudera.server.cmf.descriptor.components.DescriptorFactory: Could not generate client configs for service: YARN (MR2 Included) Caused by: com.cloudera.cmf.service.config.ConfigGenException: Unable to generate config of 'mapreduce.application.framework.path'   a

Hi All,I am having an issue in restarting Ranger Service via Ambari. I deployed a cluster via cloudbreak then added the Ranger service, I then had to make a config change and when I go to Ranger > Restart > Restart All Affected I get the attached issue:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 241, in <module> RangerAdmin().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 971, in restart self.stop(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 61, in stop Execute(format('{params.ranger_stop}'), environment={'JAVA_HOME': params.java_home}, user=params.unix_user) File "/usr/lib/ambari-agent/lib/resource_management/core

How to denied use rm with -skiptrash on HDFS?I want to block any attempts to delete a file with the -skiptrash option.

Hi, I use Impala UI, under /queries tab, in order to get runtime metrics for queries. I'm a little bit confused about the 'duration' and 'waiting time' metrics. What is being computed and tracked in each of them? I have also timed how long it takes to run the query (since statement is submitted until all rows are fetched) from the client side (I use Ibis framework to submit queries to Impala) and I can see a much smaller runtime than the one reported in the UI. Why is that? Is perhaps the metrics reported an aggregated computing time across all impalad nodes? Thanks,Paulo.

Hi All - We restarted cloudera manager today with a change in mysql database. we see connnection refusedexceptions on all agent logs and cloudera manager as in below screenshot.Why this happens ?  

Good day guys, im newby in Cloudera and wanted to ask 2 questions. 1) I got 20TB of data and i should migrate it to 10 servers, do i need to have 20TB of disk on each server ?2) How do i organize the right HDFS model (NameNode, DataNode, SecondaryNameNone) on those 10 servers ? Thanks, i hope to receive the answer very soon )  

Hi All, i tired to start and stop my clodera manger, wheire i need to upgrader the VM with RAM,and storagefirst i tried to stop the clodera manger#service cloudera-scm-server stopclouderamanger is already stoppedbut still the UI i was able to access. so i found the process id and kill the process.  kill -9 pidand stoped the agent.service cloudera-scm-agent stop and shutdown the VM and allaocted the resourses . and started the VM . now i started the service .service cloudera-scm-server startservice cloudera-scm-agent start after startign the service . i see error in the logs2019-03-22 07:25:40,874 INFO 1894048261@scm-web-0:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-ACTIVITYMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx2019-03-22 07:25:41,317 INFO 1894048261@scm-web-0:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure fo

Hi community,I am studying ksu for some use cases and found this link:https://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-user/ksu.htmlI have a user1 with KDC entry and keytab. Just before running ksu, I kinit user1 to get Kerberos ticket:[user1@server1 ~]$ klistTicket cache: FILE:/tmp/krb5cc_1003293697Default principal: user1@EXAMPLE.COMValid starting Expires Service principal02/18/2019 09:13:12 02/19/2019 09:13:12 krbtgt/EXAMPLE.COM@EXAMPLE.COMThen, I want user1 to ksu user2. For this to work, I have created a .k5login file on user2 home directory with user1@EXAMPLE.COM on its content.Than, I launch ksu with user1 but found this issue:[user1@server1 ~]$ ksu user2ksu: Server not found in Kerberos database while verifying ticket for serverAuthentication failed.Looking for an error on /var/log/krb5kdc.log, I found that one:UNKNOWN_SERVER: authtime 0, user1@EXAMPLE.COM for krbtgt/UNKNOWN_DOMAIN@EXAMPLE.COM, Server not found in Kerberos databaseAs the error states, service principal

Hello.Is there a way to change in Hue the default database for user to another one?

I created a UDA function which was written in C++. When I try to test it on some tables, it seems that when the table is stored as several parquet files, the UDA function will cause impala to crash. The error message is: Socket error 104: Connection reset by peer. However, the UDA function works fine with tables that stored in only one file.I checked the error logs but didn't find any useful information. The Version of Impala:Version 2.11.0-cdh5.14.4 Here is the UDA function:#include "funnel-uda.h"#include <sstream>#include <iostream>#include <algorithm>#include <vector>#include <string>#include <stdlib.h>using namespace impala_udf;using namespace std;template <typename T>StringVal ToStringVal(FunctionContext* context, const T& val) { stringstream ss; ss << val; string str = ss.str(); StringVal string_val(context, str.size()); memcpy(string_val.ptr, str.c_str(), str.size()); return string_val;}