Solved

Brand new NiFi user here. I created a process group and have the QueryDatabaseTable → PutFile processors linked. I funnel the output for failures and successes from the PutFile processor. I'm getting data passed through without issue, however, it is not writing the output file to my C:\users\<username>\nifi_output on my local machine. I have changed the configuration of the PutFile to try \\<machine-name>\C$\users\<username>\nifi_output for the output file and that hasn't worked either. I haven't received any errors. Like I said, I'm a newbie, so I don't know where to look any further. Thanks!

Hello, I am regularly seeing this error in the state-change.log on my production cluster and was wondering if this is something to worry about.   prod.cmla.metrics-15 in response to UpdateMetadata request sent by controller 1005 epoch 84 with correlation id 1083 (state.change.logger)[2019-08-21 04:18:18,862] TRACE Controller 1005 epoch 84 received response {error_code=0} for a request sent to broker  (id: 1005 rack: null) (state.change.logger)[2019-08-21 04:18:18,863] TRACE Controller 1005 epoch 84 received response {error_code=0} for a request sent to broker  (id: 1002 rack: null) (state.change.logger)

Can we have a shell script to delete the nodes in Ambari which were already deleted in Azure

Hi   I have been searching for sometime for a command reference/API manual for PySpark-Kudu and I have been unsuccessful so far. Does Cloudera have something that can be of help?   Thanks.

I am using HDP-2.6.5.0 with kafka 1.0.0; I have to process large (16M) messages so i setmessage.max.bytes=18874368replica.fetch.max.bytes = 18874368socket.request.max.bytes =18874368From Ambary/Kafka configs screen and restarted Kafka servicesWhen I try to send 16M messages:/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list <broker-ip>:6667 --topic test < ./big.txtI still have the same error:ERROR Error when sending message to topic test with key: null, value: 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)org.apache.kafka.common.errors.RecordTooLargeException: The message is 16777327 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.I tried to set max.request.size in producer.properties file but still have the same error. What am I missing?Thank you,

Hello,   Cloudera version 5.8(Parcels), Impala version is 2.6 and we are facing lot of issues. So we decided to upgrade the Impala.   What all Impala versions are compatible with CDH 5.8. Please provide me the documentations to follow and Impala parcels location.   Please suggest me the upgrade possibilities.

I am trying to install HDP 3.1 with Ambari 2.7 following this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/set_up_the_ambari_server.htmlAll software installation completed and mysql database had been created. I then setup ambari with "ambari setup" followed by "ambari start". But I got follwing errors. I had listed the screen output for ambari setup and part of ambari-server.log to show the error message. Please help. Thanks,HarrySetup and start ambari-server [root@msl-dpe-perf80-100g resources]# ambari-server setupUsing python /usr/bin/pythonSetup ambari-serverChecking SELinux...SELinux status is 'disabled'Customize user account for ambari-server daemon [y/n] (n)? Adjusting ambari-server permissions and ownership...Checking firewall status...Checking JDK...Do you want to change Oracle JDK [y/n] (n)? Check JDK version for Ambari Server...JDK version found: 8Minimum JDK version is 8 for Ambari. Skipping to setup differe

Hi Hortonworks Team,   We are getting error "DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020" during pip install. I can see that HDP3.0/HDF3.2 are having python 2.7. My question is post January 1st, 2020. will HDP3.0/HDF3.2 stop working or we will not be able to install any python library through pip or there will be new release of HDP/HDF sandboxes with python version > 2.7.

Hello Community, I have a CDH 5.14 cluster hosted on EC2 machines which is kerberized with Active Directory and have Sentry for authorization of databases.  I want to use SSSD to secure my Linux hosts (RHEL 7.x) with Active Directory.  I have been going through this post but there are few queries which are bothering me to proceed forwards: 1. There are service-users (Hive, YARN, etc.) in AD that are already created during Kerberization of my cluster. So, if I go ahead and implement SSSD, then will these pre-existing service-users be able to communicate? 2. If something goes wrong will I be able to rollback? If yes, how?  

after confirming hosts these two are the warnings   Process Issues (2) The following processes should not be runningProcess /usr/jdk64/jdk1.8.0_112/bin/java -c. Running on 1 host /usr/jdk64/jdk1.8.0_112/bin/java c...  Running on 1 host

Hi,   We are getting the below error whenever we are trying to access HDFS from console. This error is coming after enabling the kerberos authentication in the cluster. We are getting this error when we are trying to access from a customized user 'edtuser'. But we are not getting error when we are accessing error from 'root' user. and 'hdfs' user.Please find the error below:-   [edtuser@nladfmrvu11 ~]$[edtuser@nladfmrvu11 ~]$ hadoop fs -ls /19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.2

Hi,   Is it possible to upgrade Enterprise version of Cloudera Manager to Express ? What happend if licence expired and I need to upgrade to newest Cloudera Manager that I can upgrade cluster to higher version?   

stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 43, in BeforeStartHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 37, in hook setup_unlimited_key_jce_policy() File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/shared_initialization.py", line 197, in setup_unlimited_key_jce_policy __setup_unlimited_key_jce_policy(custom_java_home=params.java_home, custom_jdk_name=params.jdk_name, custom_jce_name = params.jce_policy_zip) File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/shared_initialization.py", line 235, in __setup_unlimited_key_jce_policy if jcePolicyInfo.is_unlimited_key_jce_policy(): File "/usr/lib/ambari-agent/lib/resource_management/core/resources/jcepolicyinfo.py", line 39, in is_unlimite

Hello, I had trouble to post questions on the community. It keeps marking my question as spam and I dispute to moderator get no answers. Does anyone know how to post questions without being treated as spam thanks,Harry

Hello All, I had a question around the upcoming CDSW 1.6 release. I understand from a recent webinar that the release would come out in the 2nd half of July of 2019. Has a more specific date been announced yet? In addition, will the new CDSW 1.6 be able to run on newer versions of RHEL other then what is currently supported by CDSW 1.5 (7.2, 7.3, 7.4, 7.5). Specifically, will it run on 7.6? Thanks,Robert

We already have a Kafka in our CDH cluster and we wish to set up a newer version of Kafka in the same cluster. I have tried adding new Kafka parcel in CM and it seems only one version of Kafka parcel can be enabled at the same time. I wonder whether it is possible. Thanks.

I am facing a StringIndexOutOfBoundsException while converting CSV to AVRO format using ConvertRecord 1.2.0 processor in NiFi. Can someone please help me.

this is the error   2019-08-21 16:13:10,828 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 408, in <module> NameNode().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 138, in start upgrade_suspended=params.upgrade_suspended, env=env) File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 264, in namenode create_hdfs_directories(name_service) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 336, in create_hdfs_directori

 Caused by: java.sql.SQLException: Connections using insecure transport are prohibited while --require_secure_transport=ON. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)

in server.out file this is the error   Wed Aug 21 11:54:54 IST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.   in server.log  file this is the error   Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:146)a

Hello All, Due to memory issues we had to remove some components from our test cluster and one of them was oozie. Currently I am getting an error in Hue "Missing required value: Oozie Service", would appreciate any help/guide in getting this resolved.    

I used configuration hive-site.xml what I already setted. than I reinstalled hadoop, hive and started hive-metastore. But I could not connect MariaDB with hive-client. Metastore log showed me MariaDB syntax error. I know hive support only MySQL. But I'm always using MariaDB and I've never gotten any problem.   I guess it is wrong espace query. I searched what is query. I thought hive ran this query. SELECT “THIS”.”NAME” AS NUCORDER0 FROM “DBS” “THIS” WHERE (LOWER(“THIS”.”NAME”) LIKE ? ESCAPE ‘\\' ) ORDER BY NUCORDER0    But I think this query using only one backslash character. so I though there is escape issue. Can I fix or avoid this problem?   This is log of exception. NestedThrowablesStackTrace:java.sql.SQLSyntaxErrorException: (conn=31) You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''\' ORDER BY NUCORDER0' at line 1at org.mariadb.jdbc.internal.util.exceptions.Exceptio

[root@sxd-cdh-nn parcels]# spark-submit --versionWelcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ version 2.4.0-cdh6.1.1/_/Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201Branch HEADCompiled by user jenkins on 2019-02-08T07:10:49ZRevision HEADUrlType --help for more information.     we have on top of this installed spark2 with 2.2    [root@sxd-cdh-nn parcels]# spark2-submit --versionWelcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ version 2.2.0.cloudera1/_/Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201Branch HEADCompiled by user jenkins on 2017-07-13T00:28:58ZRevision 39f5a2b89d29d5d420d88ce15c8c55e2b45aeb2eUrl git://github.mtv.cloudera.com/CDH/spark.gitType --help for more information.   Could you please let me know how to uninstall spark-submit --version to 2.4?   we only want spark2 with 2.2 version    thanks Romesh

Hi   By mistake i deleted one file in ambari agent that is rm -f /run/ambari-agent/ambari-agent.pid   after that its showing this error   ambari-agent restartRestarting ambari-agentVerifying Python version compatibility...Using python /usr/bin/pythonambari-agent is not running. No PID found at /run/ambari-agent/ambari-agent.pidVerifying Python version compatibility...Using python /usr/bin/pythonChecking for previously running Ambari Agent...Checking ambari-common dir...Starting ambari-agentVerifying ambari-agent process status...ERROR: ambari-agent start failed. For more details, see /var/log/ambari-agent/ambari-agent.out:====================No handlers could be found for logger "root"Failed to start ping port listener of: Could not open port 8670 because port already used by another process:UID PID PPID C STIME TTY TIME CMDroot 23758 23754 0 Aug19 pts/4 00:00:17 /usr/bin/python /usr/lib/ambari-====================Agent out at: /var/log/ambari-agent/ambari-agent.outAgent

I would like to downgrade Ambari server (from 2.7.3 to 2.6.2). This is a new installation and since Ambari was never upgraded, there are no database back up files (as referenced here: https://community.hortonworks.com/questions/175812/downgrade-procedure-from-ambari-version-2610-to-lo.html). Therefore, I modified the instructions as follows (omitting the old database import steps and re-running ambari-server setup). Will this work without negatively affecting the other HDP 2.6.4 components? If not, is there a procedure to downgrade Ambari in a new installation?1. Stop the ambari-server.# ambari-server stop2. Stop all the ambari-agents on all hosts.# ambari-agent stop3. Download the ambari.repo of the downgraded version on all agent hosts and ambari server host.Example:# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo 4. (omitting this step) Then take a backup of "/etc/ambari-server/conf" and "/etc/ambari-agent