Solved

Hi all, there are process groups in my NiFi (1.10.0) which contain disabled processors.These process groups are versioned and all changes are committed. As soon as the NiFi instance is restarted these process groups are marked as changed and "show local changes" here:     I would like to know if this behaviour of NiFi is "normal". So far I have not noticed this fact.It is misleading when uncommitted changes are shown and it is not possible to avoid that disabled processors are included.   Does anyone know the phenomenon described above?Is there a solution for it? Thanks for any advice.

Hello All,     When upgrading HDP cluster to 3.1.0 version -Restarting yarn DNS resgistry failed due to  ERROR 'resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh -H -E /usr/hdp/3.1.0.0-78/hadoop-yarn/bin/yarn --config /usr/hdp/3.1.0.0-78/hadoop/conf --daemon start registrydns' returned 1. ERROR: Cannot set priority of registrydns process 123223'. I ignored it and proceeded with the upgrade. But at Finalize Pre check the task failed and the only option available is to delete the unhealthy and host and proceed with the upgrade.   How can I resolve this issue. @jsensharma

Hello,   We are having difficulty downloading files from the open-source HDP platform through the links  https://www.cloudera.com/downloads/hdp.html  https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/ambari_repositories.html https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/hdp_315_repositories.html   Before we had this access, considering that HDP is an Open-Source platform.   Without access to these links above, we have no way to install and test the HDP.   The link for HDF (https://www.cloudera.com/downloads/hdf.html) is already no more pointing to HDF, but now for CDF. The HDF are no more available?   Thanks.

I'm looking at the Spark parcel that is included with CDH 6.3, and seeing something strange.  Specifically, the source code under python/pyspark doesn't seem to match up with what you see in the upstream (Apache) repository.   Take one file as an example. /opt/cloudera/parcels/CDH/lib/spark/python/pyspark/context.py   Jumping to line 114 of this file, we see:     self._callsite = first_spark_call() or CallSite(None, None, None) if gateway is not None and gateway.gateway_parameters.auth_token is None: raise ValueError( "You are trying to pass an insecure Py4j gateway to Spark. This" " is not allowed as it is a security risk.")     Now, this file should come from Spark 2.4.0 (see both /opt/cloudera/parcels/CDH/lib/spark/RELEASE and /opt/cloudera/parcels/CDH/lib/spark/python/pyspark/version.py files, which indicate this).  However, if you look at the upstream source, at the same line

  hi all   we have HDP version 2.6.5 and ambari 2.6.2.2   we configured the following    Metrics Service operation mode=distributed hbase.rootdir=hdfs://hdfsha:8020/user/ams/hbase   and all the rest according to - https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode   so ambari configured to save the data on hdfs instead the local disk   after metrics collector service restart we saw the following from log     2020-02-20 20:09:15,770 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.2020-02-20 20:11:35,769 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.     and     2020-02-20 20:30:10,306 INFO org.apache.hadoop.metrics2.impl.MetricsSys

Hi all,  Does any one know how to create timeseries graph for httpfs service from cloudera manager.  Currently we have httpfs service on 3 servers and We need to build a graph on top of it. Is there any query which I Can use of it to create a graph.   Thanks, Raghu.

I have enabled Kerberos authentication for Kafka as the documentation suggests and indeed producing to topics and consuming from topics requires authentication. Surprisingly, topic creation and deletion do not require authentication. Could somebody tell me whether this goes wrong ? Really appreciate it.   CDH version: 5.15.1 CDK version: 4.1.0 test command:     kafka-topics --create --zookeeper <zookeeper-host>:2181 --replication-factor 2 --partitions 3 --topic test2kafka-topics --delete --zookeeper <zookeeper-host>:2181 --topic test2      

I have run the following test case several times and got the same result.Context:1. My HDP cluster uses the simple mode to determine user identity. Kerberos is not enabled.2. Below is the permission on hdfs folder /data/test   drwxrwxr-x - hdfs hadoop 0 2020-02-07 13:33 /data/test   So hdfs (the super user) is the owner and hadoop is the owner group. Both the owner user and owner group has write permission on the /data/test folder. Steps: On an edge node, I used ID command to confirm that the logged on user "testuser" is in hadoop group.   $ iduid=1018(testuser) gid=1003(hadoop) groups=1003(hadoop),10(wheel), 1002(hdfs) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023   However, testuser still ran into "Permission Denied".   $ hadoop fs -put ./emptyfile1.txt /data/testput: Permission denied: user=testuser, access=WRITE, inode="/data/test":hdfs:hadoop:drwxrwxr-x   Then I used hdfs account to changed the folder owner to be testuser.   &nbsp

Hello everyone,need help please.when i execute the following request on the impala-shell (Cloudera cdh5.16)CREATE TABLE FCT_OC_ACTIVE_ALARMS_COUNTERS (       TIME TIMESTAMP NOT NULL ENCODING RLE,       OPERATIONCONTEXTID BIGINT NOT NULL ENCODING RLE,       SEVERITYID BIGINT NOT NULL ENCODING RLE,       ALARMTYPEID BIGINT NOT NULL,       PROBABLECAUSEID BIGINT NOT NULL,       SPECIFICPROBLEMSID BIGINT,       MANAGEDOBJECTNAME STRING NOT NULL,           YEARMONTH INT NOT NULL,           NEXTDELTATIME TIMESTAMP,       AOACKNOWLEDGED BIGINT,       AOHANDLED BIGINT,       AONOTHANDLED BIGINT, 

Problem Statement : when using a local repository, installation fails due to empty baseurls being written to the Ambari repository (even though local repository baseurl values were provided). The installation fails with below error : 020-02-19 11:58:52,681 - Reporting component version failedTraceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 363, in execute self.save_component_version_to_structured_out(self.command_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 223, in save_component_version_to_structured_out stack_select_package_name = stack_select.get_package_name() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 109, in get_package_name package = get_packages(PACKAGE_SCOPE_STACK_SELECT, service_name, component_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", l

Hi, Is migration of existing on-prem CDH clusters to CDP cloud supported? If yes, can anyone direct me to the documentation related to this that might be helpful. If not supported as of now, when can we expect this to be available?   Thanks in advance.

Please can you give me an example of how i would extract the "test" values from the below json    I tried the following but not winning ,    $.outputs.name.test1  $.test1 $.outputs.test1   I am using the processor "evaluate json path"   The below is a response from a webservice I am calling . I want the "test" values to be inserted in a sql table    "outputs": [{"name": "test1","value": "1"},{"name": "test2","value": 0.9512272865085364},{"name": "test3","value": 0.9512272865085364},{"name": "test4","value": "1"},{"name": "test5","value": 4.4659},{"name": "test6","value": 0.048772713491463618}]}

How to Generate Certs to secure 3 node nifi cluster and configs   All, I am new to nifi and started to secure nifi  can some share how to generate certs using toolkit standalone   Do I need to generate certs on all nodes   node1.xyz.com node2.xyz.com node3.xyz.com   I understand initial admin identity but what is   Node Identity for 3 nodes i have to update in authorizers.xml file     

While exporting data to Teradata, duplicate records in the Hive table are not exported, even though the Teradata table has a single non unique index.  Is there any way to change this behavior and force the duplicate records into Teradata?

Hi,    There is an issue when having a lot of (17.000 +)  components in NiFi's canvas resulting in a very large template.   When the flow.xml.gz in /var/lib/conf/flow.xml.gz starts to get large we need to increase maximum heap size for the toolkit JVM (Ambari > NiFi > Advanced nifi-toolkit-env > Toolkit java options) in order for NiFi to start. Normally this solves the issue setting it to -Xms1024m -Xmx7168m or any reasonably high maximum value.   The issue is when flow.xml.gz starts to exceed ~70MB, NiFi can't be started even with large amounts of memory (we've tried up to 70GB).   IDK if this is a memory problem (e.g. we need more RAM) or if we've hit a wall of some kind.   Error:       ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.lang.StringCoding.encode(StringCoding.java:350) at java.lang.String.getBytes(String.java:941)

Nifi 1.11.1 cannot find JSONToAttributes and ConvertJSONToAvro, Any suggestions???

Anyone in here know.How to command to run infra solr instance and infra solr client manually ?

Hi,   Can some one help me how to import pandas and Numpy in Livy2.   I'm using zeppelin and using the Livy2. In the Livy i'm creating a notebook and provide the following.   %pyspark import pandas as py   I see the below error as "No module found"   If i use the same using the pyspark interpreter it looks good and i don't have any issues. Need some help on this.   Thanks Sambasivam.        

Hi,   I had to reboot one of my data node. While rebooting I came to know the data node that I am rebooting also acts as Journal node.   The cluster is kerberized and HA enabled. After reboot, both Namenodes are coming as Standby. I tried all rebooting methods, no luck..   Here are the zookeeper Logs: 2020-02-07 20:22:47,823 WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(211)) - Transport-level exception trying to monitor health of NameNode at admin1.XXXX.io/XX.4.48.11:8020: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/XX.XX.48.11:43263 remote=admin1.XXXX.io/XX.XX.48.11:8020] Call From admin1.XXXX.io/XX.XX.48.11 to admin1.XXXX.io:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/XX.XX.48.11:43263 remote=admi

Hello,   Thank you for reading this question. Recently, one of Hadoop clusters we have has been upgraded to Hadoop 3.0 ( CDH 6.3.2 ). I'm curious that the copied data from Hadoop 3.0 ( by distcp ) can be used by Hadoop 2.6 cluster. I was able to  distcp from Hadoop 3.0 cluster to Hadoop 2.6, but I found out this document. https://hadoop.apache.org/docs/r3.0.3/hadoop-distcp/DistCp.html#Copying_Between_Versions_of_HDFS   Can you give me some comment for this?   Thank you very much in advance.

Hello,   After running a security tool on our cluster, a report found the tomcat version as outdated and vulnerable to certain threats. We have tried to find ways to upgrade Tomcat on our cluster but are not having any success with it.   I realize this is similar to a post https://community.cloudera.com/t5/Support-Questions/Apache-tomcat-compatibility/m-p/159988 which concerns HDP but there are no responses on that post either.   Does anyone have some experience with this? If so, I would be grateful for some pointers. Thanks.   Regards, CaptainJay

Hi All,   Can we install Druid in CDH 6.x version ? if so please help me with installation steps .    Thanks in advance

Hi, I have csv files and I want to move the content of files along with its meta data (File name, source (To be hard coded), control number (Part of file name - to be extracted from file name itself) thru NiFi. So here is the sample File name and layout - File name - 12345_user_data.csv (control_number_user_data.csv) source - Newyork CSV File Content/columns -  Fields - abc1, abc2, abc3, abc4  values - 1,2,3,4   Postgres Database table layout Table name - User_Education fields name  - control_number, file_name, source, abc1, abc2, abc3, abc4 Values -  12345, 12345_user_data.csv, Newyork, 1,2,3,4 I am planning to use below processors -  ListFile FetchFile UpdateAttributes PutDatabaseRecords LogAttributes But I am not sure how to combine the actual content with the meta data to load into one single table. Please help      

Currently we are seeing our HDFS DFS directory is getting filled up and we have to remove the data at faster rate. We currently have 12 datanodes and 4 masternodes 1 edgenode. Can I delete the files from HDFS from masternodes and edgenodes at once? I have created a script on edgenode which deletes the HDFS files but speed is really slow.How can I delete multiple files at a time ? Can I place that script on multiple server and delete the files?

Waiting for server start...............ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information.