Solved

Hello All,Off late we have been getting concerning alerts for Reports Manager (The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.) occasionally reports manager service also goes to bad state.I believe, this has to do with Reports Manager Heap Memory, but am in a doubt hence wanted to get a confirmation before playing around with it. I have attached a snippet of the error, which I get.Also would like to know if there is any reference guide / document by which, we can calculate and periodically adjust such items.HDFS Used – 10TB / Total 90TBFSImage – 980MCM - 6.16.2 / CDH - 5.16.2The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.[root@MY Server cloudera-scm-headlamp]# tail -n 300 mgmt-cm

hi all  when we look on https://supportmatrix.hortonworks.com/ And choose the RHEL 7.7  HDP or AMBARI versions are not marked from some unclear reason! so we cant captured the HDP or ambari version for RHEL 7.7 is it problem in support-matrix page ?  or something else ? any way if we want to use RHEL 7.7 what are the HDP and AMBARI Versions? that fit this RHEL version ?     

Hello   I get this error when restarting components :   500 status code received on POST method for API: /api/v1/clusters/Analytics_Hadoop/requests   I deleted oozie service    I am using postegres database Thank you in advance

I am reading "Enable the Atlas Book & Bridge for Apache HBase" https://atlas.apache.org/Hook-HBase.html, and get stuck on the step belowLink Atlas hook jars in HBase classpath - 'ln -s <atlas package>/hook/hbase/* <hbase-home>/lib/'Where is the HBase classpath in HDP?I checked the file hbase_site.xml but still got lost. From the file, # Extra Java CLASSPATH elements. Optional.export HBASE_CLASSPATH=${HBASE_CLASSPATH} Is the classpath env variable specified somewhere else?

Wondering if its possible to add a GPU node to Cloudera Enterprise 5.14.4. If yes, is there any documentation available around it.    Thanks

I added an old server already used as datanode in my cluster. The process get completed, but as soon as the datanode is been added I get the following error message on cloudera manager: "The DataNode has 1 volume failure but disks"   I don't have any error message, just one warning as soon as I added the host:     5:27:37.578 PM WARN Storage Failed to add storage directory [DISK]file:/data/2/dfs/dn/org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/2/dfs/dn is in an inconsistent state: Can't format the storage directory because the current/ directory is not empty. at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:495) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:600) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:279) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java

# The database type# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.com.cloudera.cmf.db.type=mysql How can that be changed to postgres. I tried so many things and now out of options.  I am doing a POC and at first started the regular installation bu installing agent,server.daemons and bringing up CM server without a DB and then unstalled all retried using the cloudera-manager-installer.bin . After a day of searching i cannot get the installtion to complete. The embedded database does not install with the below error  $$$$sudo cat /var/log/cloudera-manager-installer/7.start-cloudera-scm-server-db.logJob for cloudera-scm-server-db.service failed because the control process exited with error code. See "systemctl status cloudera-scm-server-db.service" and "journalctl -xe" for details.$$$$ systemctl status cloudera-scm-server-db.service● cloudera-scm-server-db.service - LSB: Cloudera SCM Server's Embedded DBLoaded: loaded (/etc/rc.d/init.

We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry   We get the following error on one of the zookeeper server   [2019-11-12 07:44:20,719] ERROR Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer)java.io.IOException: Unreasonable length = 198238896at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:629)at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:601)at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:591)at org.apache.zookeeper.se

I have a 4 node CDH cluster and want to add CDSW. Does adding CDSW entail adding a new host onto the cluster or is CDSW ran on top of the CDH cluster?

Hello to all, I'm trying to execute a command /opt/cloudera/parcels/CDH/lib/spark/bin/spark-beeline -u 'jdbc:hive2:<myserver>:10000' -f /root/testquery.sql and get an error Error: Could not find or load main class org.apache.hive.beeline.BeeLine spark-beeline is a script (taken from spark repository with some path modifications): set -o posixexport JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera# Figure out if SPARK_HOME is setif [ -z "${SPARK_HOME}" ]; then source "$(dirname "$0")"/find-spark-homefiCLASS="org.apache.hive.beeline.BeeLine"exec "/opt/cloudera/parcels/CDH/lib/spark/bin/spark-class" $CLASS "$@"  I think that I need to add a classpath to Hive Beeline (maybe/opt/cloudera/parcels/CDH/lib/hive/lib/hive-beeline.jar). I tryed to add --driver-class-path /opt/cloudera/parcels/CDH/lib/hive/lib/hive-beeline.jar and --conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hive/lib/hive-beeline.jar" options to command witho

Bionic supported CDH parcel is package release is 1425774 but it is not in the manifest.json file. https://archive.cloudera.com/cdh6/6.2.1/parcels/manifest.json   Is there a new package being prepared for 6.2.1 bionic?  

stderr:   /var/lib/ambari-agent/data/errors-2079.txt /Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart self.status(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 63, in status check_service_status(env, name='collector') File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/status.py", line 41, in check_service_status check_process_status(pid_file) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning()ComponentIsNotRunningThe above exception was the cause of the following exception:Traceback (most recent call last): File "/var/lib/ambari-agent/cach

  we have two SPARK2_THRIFTSERVER on node01/03 in our ambari server we want to delete both SPARK2_THRIFTSERVER on both nodes we try the following API , but without success , any idea where we are wrong?   component_name : SPARK2_THRIFTSERVER   the REST API: ( we try to delete the thrift server on one of the nodes )   curl -iv -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://node02:8080/api/v1/clusters/HDP/hosts/node01/SPARK2_THRIFTSERVER * About to connect() to node02 port 8080 (#0)* Connected to node02 (45.3.23.4) port 8080 (#0)* Server auth using Basic with user 'admin'> DELETE /api/v1/clusters/HDP/hosts/node01/SPARK2_THRIFTSERVER HTTP/1.1> Authorization: Basic YWRtaW46YWRtaW4=> User-Agent: curl/7.29.0> Accept: */*> X-Requested-By: ambari>< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found  

Hi, We have a process that generates text files in a ftp server. The process generates the files by writing multiple passes and could take from say more than a minute to a few minutes to generate the files.   The nifi ListSFTP in this case generates multiple files as we are using tracking timestamps. The file's timestamp is getting updated continuously during the generation process.   We dont have control over the existing process that generates the files and has no notification mechanism in place that can tell us when the file generation is complete.     What would be the best practice to handle this scenario in a List+FetchSFTP flow to avoid duplicate files generated in the target system.    Please advice.   Thank you

Hi Team,   Could you please help me to understand why installing Hive Service requires PIG clients to be installed. Also, is it okay to remove PIG clients after installing hive. 

/Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in <module> AmsCollector().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 355, in execute self.execute_prefix_function(self.command_name, 'post', env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 382, in execute_prefix_function method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 424, in post_start raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file))resource_management.core.exceptions.Fail: Pid file /var/run/ambari-metrics-collector//hbase-ams-master.pid doesn't exist after starting of the component./

sudo -u hdfs hdfs dfsadmin -reportConfigured Capacity: 860500408320 (801.40 GB)Present Capacity: 1417964708 (1.32 GB)DFS Remaining: 322059428 (307.14 MB)DFS Used: 1095905280 (1.02 GB)DFS Used%: 77.29%Replicated Blocks:Under replicated blocks: 520Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0Low redundancy blocks with highest priority to recover: 520Pending deletion blocks: 0Erasure Coded Block Groups:Low redundancy block groups: 0Block groups with corrupt internal blocks: 0Missing block groups: 0Low redundancy blocks with highest priority to recover: 0Pending deletion blocks: 0 -------------------------------------------------Live datanodes (1): Name: 192.168.24.32:50010 (xxxxxxxxxxxxx)Hostname: gaian-lap386.comDecommission Status : NormalConfigured Capacity: 860500408320 (801.40 GB)DFS Used: 1095905280 (1.02 GB)Non DFS Used: 808787101696 (753.24 GB)DFS Remaining: 322059428 (307.14 MB)DFS Used%: 0.13%DFS Remaining%: 0.04%Configured Cache C

Hi,   I'm trying to figure out if the following scenarios are possible:1. I have hundreds of tables that need to use the same flow, but have different intervals, different source hostnames and destinations?How to build such a flow?  also I can't figure out how to use dynamic hosts/schemas/table names... We maintain a table with with all this info but how to execute it with NiFi?2. If I need to load a file on multiple clusters (each table different clusters) in parallel - how can this be achieved?   tnx!

Hi all, I'm have a 9 host cluster on HDP 2.6.5.  Recently, a host went down and had to be rebuilt.  It was hosting ZK server, DataNode and YARN server.  Can I safely delete the host from Ambari web UI as I'm unable to turn off the services? Thanks.

Hello,   I have a basic question about about namenode format. If we perform namenode format, will clean up all the metadata info in namenode like directory structure, etc.   After it completed, if we restart namenode, would it rebuild the directory structure from datanode information? Or we are lost.    

Hi Everyone, I am new to Hadoop Ecosystem. I have installed Cloudera on Ubuntu and while i was performing simple sqoop from Postgres to Hive/HDFS getting following error. Appreciate the help: Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=nthumu, access=WRITE, inode="/user/hive/warehouse/postgres":hdfs:hive:drwxrwx--xat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1855)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1839)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1798)at org.apach

hi all     I want to ask this important question   lets say we have the following:   HDP cluster with  :   3 masters machine ( active/standby name-node ) , ( active/standby resource manager ) 3 datanode machines     - each data-node machine have 4 disks for HDFS ( not include the OS ) 3 kafka machines    - each kafka machine have one disk of 10T ( not include the OS )   now we want to install from scratch all cluster include HDP and ambari   but save the data on datanode machines and kafka topics data machine by the following:   we umount the disks on datanode machines and kafka machines       example   on datanode machine ( note - /etc/stab is already configured )   umount /grid/data1 umount /grid/data2 . . .     so the second scratch installation we install all the cluster ( by blueprint ) , but without data-node HDFS disks , and kafka  topic disk ( scratch installa

How to install Apache Livy on Azure databricks?

Hello, I don't know if this is the means to ask for help but because professional purposes, I have to remove a question I've made here in the community, excuse me, and I hope you can help me. Thanks.

I have installed latest version of Nifi-1.10.0 , everything works locally , however on my Linux server i am having issues when i start NIFI , below is the stack trace. 2019-11-14 21:22:31,464 INFO [main] org.apache.nifi.NiFi Launching NiFi...2019-11-14 21:22:31,812 INFO [main] org.apache.nifi.security.kms.CryptoUtils Determined default nifi.properties path to be '/ngs/app/ssewiret/nifi/./conf/nifi.properties'2019-11-14 21:22:31,818 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 143 properties from /ngs/app/ssewiret/nifi/./conf/nifi.properties2019-11-14 21:22:31,826 INFO [main] org.apache.nifi.NiFi Loaded 143 properties2019-11-14 21:22:31,838 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 333172019-11-14 21:22:31,862 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap2019-11-14 21:22:31,909 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 198 NAR files with