Q&A

I have run the following test case several times and got the same result.Context:1. My HDP cluster uses the simple mode to determine user identity. Kerberos is not enabled.2. Below is the permission on hdfs folder /data/test   drwxrwxr-x - hdfs hadoop 0 2020-02-07 13:33 /data/test   So hdfs (the super user) is the owner and hadoop is the owner group. Both the owner user and owner group has write permission on the /data/test folder. Steps: On an edge node, I used ID command to confirm that the logged on user "testuser" is in hadoop group.   $ iduid=1018(testuser) gid=1003(hadoop) groups=1003(hadoop),10(wheel), 1002(hdfs) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023   However, testuser still ran into "Permission Denied".   $ hadoop fs -put ./emptyfile1.txt /data/testput: Permission denied: user=testuser, access=WRITE, inode="/data/test":hdfs:hadoop:drwxrwxr-x   Then I used hdfs account to changed the folder owner to be testuser.   &nbsp

Hello everyone,need help please.when i execute the following request on the impala-shell (Cloudera cdh5.16)CREATE TABLE FCT_OC_ACTIVE_ALARMS_COUNTERS (       TIME TIMESTAMP NOT NULL ENCODING RLE,       OPERATIONCONTEXTID BIGINT NOT NULL ENCODING RLE,       SEVERITYID BIGINT NOT NULL ENCODING RLE,       ALARMTYPEID BIGINT NOT NULL,       PROBABLECAUSEID BIGINT NOT NULL,       SPECIFICPROBLEMSID BIGINT,       MANAGEDOBJECTNAME STRING NOT NULL,           YEARMONTH INT NOT NULL,           NEXTDELTATIME TIMESTAMP,       AOACKNOWLEDGED BIGINT,       AOHANDLED BIGINT,       AONOTHANDLED BIGINT, 

Problem Statement : when using a local repository, installation fails due to empty baseurls being written to the Ambari repository (even though local repository baseurl values were provided). The installation fails with below error : 020-02-19 11:58:52,681 - Reporting component version failedTraceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 363, in execute self.save_component_version_to_structured_out(self.command_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 223, in save_component_version_to_structured_out stack_select_package_name = stack_select.get_package_name() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", line 109, in get_package_name package = get_packages(PACKAGE_SCOPE_STACK_SELECT, service_name, component_name) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", l

Hi, Is migration of existing on-prem CDH clusters to CDP cloud supported? If yes, can anyone direct me to the documentation related to this that might be helpful. If not supported as of now, when can we expect this to be available?   Thanks in advance.

Please can you give me an example of how i would extract the "test" values from the below json    I tried the following but not winning ,    $.outputs.name.test1  $.test1 $.outputs.test1   I am using the processor "evaluate json path"   The below is a response from a webservice I am calling . I want the "test" values to be inserted in a sql table    "outputs": [{"name": "test1","value": "1"},{"name": "test2","value": 0.9512272865085364},{"name": "test3","value": 0.9512272865085364},{"name": "test4","value": "1"},{"name": "test5","value": 4.4659},{"name": "test6","value": 0.048772713491463618}]}

How to Generate Certs to secure 3 node nifi cluster and configs   All, I am new to nifi and started to secure nifi  can some share how to generate certs using toolkit standalone   Do I need to generate certs on all nodes   node1.xyz.com node2.xyz.com node3.xyz.com   I understand initial admin identity but what is   Node Identity for 3 nodes i have to update in authorizers.xml file     

While exporting data to Teradata, duplicate records in the Hive table are not exported, even though the Teradata table has a single non unique index.  Is there any way to change this behavior and force the duplicate records into Teradata?

Hi,    There is an issue when having a lot of (17.000 +)  components in NiFi's canvas resulting in a very large template.   When the flow.xml.gz in /var/lib/conf/flow.xml.gz starts to get large we need to increase maximum heap size for the toolkit JVM (Ambari > NiFi > Advanced nifi-toolkit-env > Toolkit java options) in order for NiFi to start. Normally this solves the issue setting it to -Xms1024m -Xmx7168m or any reasonably high maximum value.   The issue is when flow.xml.gz starts to exceed ~70MB, NiFi can't be started even with large amounts of memory (we've tried up to 70GB).   IDK if this is a memory problem (e.g. we need more RAM) or if we've hit a wall of some kind.   Error:       ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.lang.StringCoding.encode(StringCoding.java:350) at java.lang.String.getBytes(String.java:941)

Nifi 1.11.1 cannot find JSONToAttributes and ConvertJSONToAvro, Any suggestions???

Anyone in here know.How to command to run infra solr instance and infra solr client manually ?

Hi,   Can some one help me how to import pandas and Numpy in Livy2.   I'm using zeppelin and using the Livy2. In the Livy i'm creating a notebook and provide the following.   %pyspark import pandas as py   I see the below error as "No module found"   If i use the same using the pyspark interpreter it looks good and i don't have any issues. Need some help on this.   Thanks Sambasivam.        

Hi,   I had to reboot one of my data node. While rebooting I came to know the data node that I am rebooting also acts as Journal node.   The cluster is kerberized and HA enabled. After reboot, both Namenodes are coming as Standby. I tried all rebooting methods, no luck..   Here are the zookeeper Logs: 2020-02-07 20:22:47,823 WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(211)) - Transport-level exception trying to monitor health of NameNode at admin1.XXXX.io/XX.4.48.11:8020: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/XX.XX.48.11:43263 remote=admin1.XXXX.io/XX.XX.48.11:8020] Call From admin1.XXXX.io/XX.XX.48.11 to admin1.XXXX.io:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/XX.XX.48.11:43263 remote=admi

Hello,   Thank you for reading this question. Recently, one of Hadoop clusters we have has been upgraded to Hadoop 3.0 ( CDH 6.3.2 ). I'm curious that the copied data from Hadoop 3.0 ( by distcp ) can be used by Hadoop 2.6 cluster. I was able to  distcp from Hadoop 3.0 cluster to Hadoop 2.6, but I found out this document. https://hadoop.apache.org/docs/r3.0.3/hadoop-distcp/DistCp.html#Copying_Between_Versions_of_HDFS   Can you give me some comment for this?   Thank you very much in advance.

Hello,   After running a security tool on our cluster, a report found the tomcat version as outdated and vulnerable to certain threats. We have tried to find ways to upgrade Tomcat on our cluster but are not having any success with it.   I realize this is similar to a post https://community.cloudera.com/t5/Support-Questions/Apache-tomcat-compatibility/m-p/159988 which concerns HDP but there are no responses on that post either.   Does anyone have some experience with this? If so, I would be grateful for some pointers. Thanks.   Regards, CaptainJay

Hi All,   Can we install Druid in CDH 6.x version ? if so please help me with installation steps .    Thanks in advance

Hi, I have csv files and I want to move the content of files along with its meta data (File name, source (To be hard coded), control number (Part of file name - to be extracted from file name itself) thru NiFi. So here is the sample File name and layout - File name - 12345_user_data.csv (control_number_user_data.csv) source - Newyork CSV File Content/columns -  Fields - abc1, abc2, abc3, abc4  values - 1,2,3,4   Postgres Database table layout Table name - User_Education fields name  - control_number, file_name, source, abc1, abc2, abc3, abc4 Values -  12345, 12345_user_data.csv, Newyork, 1,2,3,4 I am planning to use below processors -  ListFile FetchFile UpdateAttributes PutDatabaseRecords LogAttributes But I am not sure how to combine the actual content with the meta data to load into one single table. Please help      

Currently we are seeing our HDFS DFS directory is getting filled up and we have to remove the data at faster rate. We currently have 12 datanodes and 4 masternodes 1 edgenode. Can I delete the files from HDFS from masternodes and edgenodes at once? I have created a script on edgenode which deletes the HDFS files but speed is really slow.How can I delete multiple files at a time ? Can I place that script on multiple server and delete the files?

Waiting for server start...............ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information.  

My current environment is CDH 6.3.2 Impala v3.2.0-cdh6.3.2 kudu 1.10.0-cdh6.3.2 Somehow, creating table with kudu storage gives IllegalArgumentException. It was ok with kudu 1.7.0-cdh5.16.2 CREATE TABLE test_mlee( id BIGINT, name STRING, PRIMARY KEY(id))PARTITION BY HASH PARTITIONS 16STORED AS KUDUERROR: IllegalArgumentException: null  Any comment is appreciated Thank you in advance.

Hi,   I'm upgrading CDH from 5.13.0 to 6.3.1 and can not proceed after "Upgrade HDFS Metadata".   On a 3rd step, after "Starting the JournalNodes." and "Starting metadata upgrade on Active NameNode of nameservice nameservice1.". I cannot proceed with "Waiting for NameNode (master02) to start responding to RPCs.".   It just freezes, but according to log records, it will not continue because the process failed. And it is failed because of a quorum of JournalNodes is not succeeded. master03.ib (10.12.0.3) is not responding.   What can I do? What can cause the issue? Can I run following steps manually?   The logs says the following Spoiler (Highlight to read) 6:43:03.610 PM WARN QuorumJournalManager Waited 55044 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.12.0.1:8485] 6:43:04.611 PM WARN QuorumJournalManager Waited 56045 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.

Hello ,   Can anyone experienced issue while using ConsumeImap Processor for consuming messages from Secure Email server using Imap port 993.   We are getting below error PKIX path building failed. unable to find valid cerification path   Please write here if anyone faced issue like this.   Thanks and Regards  

We want to store our sensitive information such as passwords, private keys in Vault and retrieve it from Vault on need basis, kindly share your inputs which is much needed.

Hi Everyone,We have enabled hive compaction on transnational table in order to improve fetch query performance.Somehow it's not working at my end, am facing attached error which i observed in "hivemetastore.log". Can somebody please let us know, How this issue can be resolved ? 

Hi,Newbie here. Suddenly one of the nodes lost the heartbeat. Tried to restart ambari-agent and ambari-server. However, the error still persists. Here is the ambari-agent log.   WARNING 2020-02-11 15:24:08,318 base_alert.py:138 - [Alert][ranger_admin_password_check] Unable to execute alert. argument of type 'NoneType' is not iterableINFO 2020-02-11 15:24:14,721 security.py:141 - Encountered communication error. Details: SSLError('The read operation timed out',)ERROR 2020-02-11 15:24:14,721 Controller.py:226 - Unable to connect to: https://xxx1:8441/agent/v1/register/xxx2.comTraceback (most recent call last):File "/usr/lib/ambari-agent/lib/ambari_agent/Controller.py", line 175, in registerWithServerret = self.sendRequest(self.registerUrl, data)File "/usr/lib/ambari-agent/lib/ambari_agent/Controller.py", line 549, in sendRequestraise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))IOError: Request to https://xxx1.com:8441/agent/v1/register/xxx2.com failed due

Hi, folks,   You could help me fill up my hard drive with NiFi. /var/lib/nifi/content_repository   It's taking up 100% of my space, can you help me get it back or can I just delete it all?   Greetings