Q&A

We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry   We get the following error on one of the zookeeper server   [2019-11-12 07:44:20,719] ERROR Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer)java.io.IOException: Unreasonable length = 198238896at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:629)at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:601)at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:591)at org.apache.zookeeper.se

I have a 4 node CDH cluster and want to add CDSW. Does adding CDSW entail adding a new host onto the cluster or is CDSW ran on top of the CDH cluster?

Hello to all, I'm trying to execute a command /opt/cloudera/parcels/CDH/lib/spark/bin/spark-beeline -u 'jdbc:hive2:<myserver>:10000' -f /root/testquery.sql and get an error Error: Could not find or load main class org.apache.hive.beeline.BeeLine spark-beeline is a script (taken from spark repository with some path modifications): set -o posixexport JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera# Figure out if SPARK_HOME is setif [ -z "${SPARK_HOME}" ]; then source "$(dirname "$0")"/find-spark-homefiCLASS="org.apache.hive.beeline.BeeLine"exec "/opt/cloudera/parcels/CDH/lib/spark/bin/spark-class" $CLASS "$@"  I think that I need to add a classpath to Hive Beeline (maybe/opt/cloudera/parcels/CDH/lib/hive/lib/hive-beeline.jar). I tryed to add --driver-class-path /opt/cloudera/parcels/CDH/lib/hive/lib/hive-beeline.jar and --conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hive/lib/hive-beeline.jar" options to command witho

Bionic supported CDH parcel is package release is 1425774 but it is not in the manifest.json file. https://archive.cloudera.com/cdh6/6.2.1/parcels/manifest.json   Is there a new package being prepared for 6.2.1 bionic?  

stderr:   /var/lib/ambari-agent/data/errors-2079.txt /Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart self.status(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 63, in status check_service_status(env, name='collector') File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/status.py", line 41, in check_service_status check_process_status(pid_file) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning()ComponentIsNotRunningThe above exception was the cause of the following exception:Traceback (most recent call last): File "/var/lib/ambari-agent/cach

  we have two SPARK2_THRIFTSERVER on node01/03 in our ambari server we want to delete both SPARK2_THRIFTSERVER on both nodes we try the following API , but without success , any idea where we are wrong?   component_name : SPARK2_THRIFTSERVER   the REST API: ( we try to delete the thrift server on one of the nodes )   curl -iv -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://node02:8080/api/v1/clusters/HDP/hosts/node01/SPARK2_THRIFTSERVER * About to connect() to node02 port 8080 (#0)* Connected to node02 (45.3.23.4) port 8080 (#0)* Server auth using Basic with user 'admin'> DELETE /api/v1/clusters/HDP/hosts/node01/SPARK2_THRIFTSERVER HTTP/1.1> Authorization: Basic YWRtaW46YWRtaW4=> User-Agent: curl/7.29.0> Accept: */*> X-Requested-By: ambari>< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found  

Hi, We have a process that generates text files in a ftp server. The process generates the files by writing multiple passes and could take from say more than a minute to a few minutes to generate the files.   The nifi ListSFTP in this case generates multiple files as we are using tracking timestamps. The file's timestamp is getting updated continuously during the generation process.   We dont have control over the existing process that generates the files and has no notification mechanism in place that can tell us when the file generation is complete.     What would be the best practice to handle this scenario in a List+FetchSFTP flow to avoid duplicate files generated in the target system.    Please advice.   Thank you

Hi Team,   Could you please help me to understand why installing Hive Service requires PIG clients to be installed. Also, is it okay to remove PIG clients after installing hive. 

/Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in <module> AmsCollector().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 355, in execute self.execute_prefix_function(self.command_name, 'post', env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 382, in execute_prefix_function method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 424, in post_start raise Fail("Pid file {0} doesn't exist after starting of the component.".format(pid_file))resource_management.core.exceptions.Fail: Pid file /var/run/ambari-metrics-collector//hbase-ams-master.pid doesn't exist after starting of the component./

sudo -u hdfs hdfs dfsadmin -reportConfigured Capacity: 860500408320 (801.40 GB)Present Capacity: 1417964708 (1.32 GB)DFS Remaining: 322059428 (307.14 MB)DFS Used: 1095905280 (1.02 GB)DFS Used%: 77.29%Replicated Blocks:Under replicated blocks: 520Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0Low redundancy blocks with highest priority to recover: 520Pending deletion blocks: 0Erasure Coded Block Groups:Low redundancy block groups: 0Block groups with corrupt internal blocks: 0Missing block groups: 0Low redundancy blocks with highest priority to recover: 0Pending deletion blocks: 0 -------------------------------------------------Live datanodes (1): Name: 192.168.24.32:50010 (xxxxxxxxxxxxx)Hostname: gaian-lap386.comDecommission Status : NormalConfigured Capacity: 860500408320 (801.40 GB)DFS Used: 1095905280 (1.02 GB)Non DFS Used: 808787101696 (753.24 GB)DFS Remaining: 322059428 (307.14 MB)DFS Used%: 0.13%DFS Remaining%: 0.04%Configured Cache C

Hi,   I'm trying to figure out if the following scenarios are possible:1. I have hundreds of tables that need to use the same flow, but have different intervals, different source hostnames and destinations?How to build such a flow?  also I can't figure out how to use dynamic hosts/schemas/table names... We maintain a table with with all this info but how to execute it with NiFi?2. If I need to load a file on multiple clusters (each table different clusters) in parallel - how can this be achieved?   tnx!

Hi all, I'm have a 9 host cluster on HDP 2.6.5.  Recently, a host went down and had to be rebuilt.  It was hosting ZK server, DataNode and YARN server.  Can I safely delete the host from Ambari web UI as I'm unable to turn off the services? Thanks.

Hello,   I have a basic question about about namenode format. If we perform namenode format, will clean up all the metadata info in namenode like directory structure, etc.   After it completed, if we restart namenode, would it rebuild the directory structure from datanode information? Or we are lost.    

Hi Everyone, I am new to Hadoop Ecosystem. I have installed Cloudera on Ubuntu and while i was performing simple sqoop from Postgres to Hive/HDFS getting following error. Appreciate the help: Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=nthumu, access=WRITE, inode="/user/hive/warehouse/postgres":hdfs:hive:drwxrwx--xat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1855)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1839)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1798)at org.apach

hi all     I want to ask this important question   lets say we have the following:   HDP cluster with  :   3 masters machine ( active/standby name-node ) , ( active/standby resource manager ) 3 datanode machines     - each data-node machine have 4 disks for HDFS ( not include the OS ) 3 kafka machines    - each kafka machine have one disk of 10T ( not include the OS )   now we want to install from scratch all cluster include HDP and ambari   but save the data on datanode machines and kafka topics data machine by the following:   we umount the disks on datanode machines and kafka machines       example   on datanode machine ( note - /etc/stab is already configured )   umount /grid/data1 umount /grid/data2 . . .     so the second scratch installation we install all the cluster ( by blueprint ) , but without data-node HDFS disks , and kafka  topic disk ( scratch installa

How to install Apache Livy on Azure databricks?

Hello, I don't know if this is the means to ask for help but because professional purposes, I have to remove a question I've made here in the community, excuse me, and I hope you can help me. Thanks.

I have installed latest version of Nifi-1.10.0 , everything works locally , however on my Linux server i am having issues when i start NIFI , below is the stack trace. 2019-11-14 21:22:31,464 INFO [main] org.apache.nifi.NiFi Launching NiFi...2019-11-14 21:22:31,812 INFO [main] org.apache.nifi.security.kms.CryptoUtils Determined default nifi.properties path to be '/ngs/app/ssewiret/nifi/./conf/nifi.properties'2019-11-14 21:22:31,818 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 143 properties from /ngs/app/ssewiret/nifi/./conf/nifi.properties2019-11-14 21:22:31,826 INFO [main] org.apache.nifi.NiFi Loaded 143 properties2019-11-14 21:22:31,838 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 333172019-11-14 21:22:31,862 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap2019-11-14 21:22:31,909 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 198 NAR files with

I'm trying to change an existing variable. If the variable was defined in a process group and this process group is then copied, I get the following message when I try to change the variable.   "Unable to complete variable update request: Failed to update Variable Registry because failed while performing step: Applying updates to Variable Registry"   NiFi 1.10

Me and My colleagues are having a discussion regarding the pros and cons on running a Cloudera cluster on Physical Servers versus running the cluster on several Virtual Machines on a Hyper Converge servers.

I am using CDH 6.1.1 Cluster.Cluster is configured to use Spark as the execution engine for Hive.  Is there anything wrong with using SparkSQL on this Cluster?Is it ok to create Hive Tables and change data using SparkSQL? Since SparkSQL uses the Hive Metastore, I suspect that there may be a conflict between SparkSQL and Hive on Spark. In addition, please refer to documentation on how to intergrate Cloudera CDH Hive with Apache Zeppelin's Spark interpreter.  Thank you. 

I have a kudu table with schema:  create table test_table( `time` timestamp not null, -- `id` string not null, -- ..... primary key(`time`,`id`))partition by hash(id) partitions 6stored as kudu; and I try to use spark to copy the data to a parquet table in hdfs:  val df = spark.read.options(Map("kudu.master" -> kuduMasters, "kudu.table" -> KuduTable)).format("kudu").load .where("time> '2019-10-29 08:05:10' AND time < '2019-10-29 08:05:30'") df.write .mode("append") .parquet("hdfs://parquet") But the performance is low and the job seems to be doing a full table scan against the kudu table (from spark UI, I can see the "Scan Kudu impala::table" is the number of entire table).For comparison I did a copy using impala's "insert into from" which is much faster and the "where" predicate seems to be working. Is this full table scan behavior expected or am I missing something here? The kudu version is 1.10.0

Dear Community,   I am trying to install an ambari cluster using Use Public Repository. I have already configured the Password Less on all the servers and I could connect from ambari node to the others using ssh root@node1,2... I copied the right ssh private key however the confirmation of hosts failed with:   Error while bootstrapping:/usr/bin/env: ambari-python-wrap: No such file or directory  I could not find any ambari-agent folder , should I have? I did not install agent manually ..   Please help Thanks and Regards

I import data from Postgresql to hdfs and hdfs to S3 in my daily operation. (sqoop import [postgres to hdfs] & distcp [from hdfs to s3])I wanted to remove intermediate step (hdfs) and directly import data to S3 bucket by using sqoop.However, same sqoop string fails in the end of the import operation.  sqoop import -Dmapreduce.map.memory.mb="8192" -Dmapreduce.map.java.opts="-Xmx7200m" -Dmapreduce.task.timeout=0 -Dmapreduce.task.io.sort.mb="2400" --connect $conn_string$ --fetch-size=20000 --username $user_name$ --p $password$ --num-mappers 20 --query "SELECT * FROM table1 WHERE table1.id > 10000000 and table1.id < 20000000 and \$CONDITIONS" --hcatalog-database $schema_name$ --hcatalog-table $table_name$ --hcatalog-storage-stanza "STORED AS PARQUET LOCATION s3a://path/to/destination"--split-by table1.id    I also tried   --target-dir s3a://path/to/destination   instead of   ....... LOCATION s3a://path/to/desti

All, I have a HDFS/Spark/YARN gateway node that has been assigned the roles via cloudera manager. However, Kerberos is preventing me from accessing HDFS from the gateway node. I could generate TGT via kinit command. I could access HDFS from other nodes in the hadoop cluster using the same user id and same steps. All kerberos configuration files in the gateway node seems to be in place. Any suggestions what i could have missed?