Q&A

Hello everyone,I'm trying to install the CDP trial version and the wizard is taking hours to finish. I'm not an expert, but the log seems okay. I did the command below to keep tracking the log.  tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log and I got these as a result and never ends... 2020-04-20 02:29:15,382 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (3 skipped) AgentAvroServlet: heartbeat processing stats: average=3ms, min=1ms, max=196ms.2020-04-20 02:30:15,422 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (3 skipped) AgentAvroServlet: heartbeat processing stats: average=3ms, min=1ms, max=196ms.2020-04-20 02:31:15,453 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (3 skipped) AgentAvroServlet: heartbeat processing stats: average=3ms, min=1ms, max=196ms.2020-04-20 02:32:15,502 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServ

I am running an Hive query (my execution engine is Tez) on an external table with gz files. The error message i get is "java.io.EOFException: Unexpected end of input stream". I have done a check on HDFS for corrupt files and it returns a healthy status. Anyone come across a similar issue?Caused by: java.io.EOFException: Unexpected end of input streamat org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:145)at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)at java.io.InputStream.read(InputStream.java:101)at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:208)at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:246)at org.apache.hadoop.mapred.LineRecordReader.next

I am running Hortonworks Sandbox HDP 2.6.5 on VirtualBox. I have increased the size of my virtual hard disk (.vdi) to 500 GB. However, when I login to Ambari and view the size of my disk, it shows 106 GB only. What should I do to increase the HDFS capacity from 106 GB to 500 GB?

Team , We have upgraded NIFI from 1.5 to 1.11 and trying to configure ranger authorization at group level instead of user level . But its not working . Below are the properties i have given in authorizers.xml <userGroupProvider><identifier>ldap-user-group-provider</identifier><class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class><property name="Authentication Strategy">LDAPS</property><property name="Manager DN">CN=xxxx,OU=Users,OU=Standard Users,OU=Users and Computers,OU=Atlanta Corporate,OU=CCI,DC=CORP,DC=COX,DC=com</property><property name="Manager Password">xxxxx</property><property name="TLS - Keystore">/etc/nifi/conf/signed-keystore.jks</property><property name="TLS - Keystore Password">xxx</property><property name="TLS - Keystore Type">JKS</property><property name="TLS - Truststore">/etc/nifi/conf/coxca-truststore.jks</property><property name=

yesterday i deleted some topics in my kafka with below command ./kafka-topics.sh — zookeeper localhost:2181 — delete — topic <topic_name> with the configuration of delete.topic.enable=true after 20 hrs i checked those topics still not deleted. how can we force delete those topics. and why it will take this much time to delete the topic?

We are doing spark-submit from airflow (added it as a custom parcel into CDP 7.1)Airflow is built with python 3 however default python version on CDP is python2. As a result during spark-submit getting this issue: WARN net.ScriptBasedMapping: Exception running /etc/hadoop/conf.cloudera.yarn/topology.py 10.228.86.42ExitCodeException exitCode=1: File "/opt/cloudera/parcels/Airflow-1.10.10-python3.7.7_1.2.3/lib/python3.7/site.py", line 177 file=sys.stderr) ^SyntaxError: invalid syntax Added PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON to spark-defaults as well as spark-env.sh pointing to python3. Also added spark.yarn.appMasterEnv.PYTHONHASHSEED = 0 however the problem remains. As soon as python version is being changed to python3 on the workers (basically the only available python becomes python 3) spark-submit starts working. I was wondering if there is something I am missing. Thanks

Hi, I am running cdh 5.14.0 cluster version both on Prod and DR cluster. Both cluster are Kerberos and TLS enabled with the same kdc server. Could anyone guide me on how to do BDR between Kerberos enabled clusters. As I was not able to find any proper guide on the Cloudera website.   Please help me or provide any docs related to this. Appreciate your help.   Thanks  pdev

Hello, I am trying to install Apache Metron in a cluster created with Ambari and HDP 2.6.5.0. When I select Metron as a service to install, I get stuck when trying to get to the next step, the next button shows as always disabled. This is not happening with any other service.   I followed this guides: Github community.cloudera.com My environment specs are: Java JDK 1.8 in all hosts Metron RPM packages in /localrepo in all hosts root user in all hosts Ambari server 2.7.5.0 6 Centos 7 host : metron1 (9.6 GB, 4 cores, 100G SSD) NameNode ZooKeeper Server Kafka Broker Zeppelin Notebook metron2 (9.6 GB, 4 cores, 100G SSD) SNameNode App Timeline Server ResourceManager History Server ZooKeeper Server Nimbus Kafka Broker metron3 (15.5 GB, 4 cores, 500G SSD) HBase Master Kafka Broker Elasticsearch Master metron4 (11.6 GB, 3 cores, 50G SSD) HBase Master Oozie Server Nimbus DRPC Server Storm UI Server Kafka Broker metron5 (11.6 GB, 3 cores, 50G SSD) Kafka Broker metron6 (15

Is the a configuration file for Cloudera Management Service?I'd like to have the service roles' configuration stored in repository. This way, service configuration can be shared between environments without the need to change the configuration via CM ui or rest API on each environment.

Hi there, I'm facing this problem for 2 days now and maybe you guys can find it.I'm not the only one facing this issue but none of the given solutions fixed my problem. Hive metastore is running and hiveserver2 is not saying connection refused I will put my exact error messages here too.   Connection failed on host localhost:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/alerts/alert_hive_thrift_port.py", line 204, in execute ldap_password=ldap_password) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/hive_check.py", line 84, in check_thrift_port_sasl timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE, File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-age

How is data quality managed?Which tool should be used for Hadoop security - Apache Ranger or Apache Sentry?

Need to download hdp-3.1.5 and ambari-2.7.5 but it says I need to register.Does one needs to be a paid customer of Cloudera to download the latest versions?

Hi,being pretty new to Nifi, I am strugeling to extract interface monitoring data Assuming a very simple ingest flow like:HttpListener -> KafkaPublisher How can I get throughput information, i.e. number of records put (e.g.) every minute to a topic? There is NifiSummary -> Processors -> Status History, which might be useful. Can the statistics be accessed programatically and how? Best regardsJaro

Newbie question, apologies. We have a need to backup a Kafka cluster, so that we can restore to a given point in time (as far as possible according to backup granularity) in case of problems, e.g. bad data. Replication would not help here, since bad data could be replicated.Does anyone out there have such a use case, and did you solve it (with Cloudera or open-source tools)?Thanks in advance.

I will deploy CDH 6.3.3 and it has Hadoop 3.0.0 , Is there any way to use later version of Hadoop like 3.1.2 as more stable instead and still being supported with Cloudera CDH 6.3.3 ?

I am using PostgreSQL for the Cloudera Manager Server and other services that use databases. I'm using a RHEL 7.4 Azure VM and following the official documentation -https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/prepare_cm_database.htmlI have completed step 4(Install Databases). In Step 5 : Set up the Cloudera Manager Database, on executing the scm_prepare_database.sh script I'm getting the following output-  [root@machine1 ~]# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh postgresql scm scmEnter SCM password:JAVA_HOME=/usr/java/jdk1.8.0_181-clouderaVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera

hi all we have HDP 2.6.4 cluster with 365 data-nodes ( workers machines )  we intend to add to this cluster 67 new data-nodes , but for now these new data-nodes should not be active , in spite the Linux OS is up and machine is up so is it possible to disable HDFS and YARN on these new machines?  what is the preferred way?  for example - decommission ? , or just maintenance mode? or maybe both them?  the target is to avoid from spark jobs to run on the new data-nodes machines and disable the HDFS Temporarily  

Hello,I need to reboot my Data node to fix an underlying network issue. OS team needs around 3 hours (approx.) to complete this activity, what would be the best approach:Decommission and recommission host / dn.Move DN to Offline Mode (feature in CM 5.16.x)Any help / guidance is appreciated. RegardsAmn

hi all we are examining the option to installed new Hadoop cluster based on CDH cloudera packages I will appreciate to get feedback's / notes / remarks / else ,  from someone that have the experienced of installing the CDH packages , and how this Hadoop looks after installing the  Cloudera open source ( CDH )    

During an overly ambitious cleanup effort, the IAM user / access key used by our Altus Director server was deleted from AWS and now we can no longer manage our clusters.  I would like to either update the access key used by director, or remove the key and force director to rely on IAM role.  Please advise.

Hi Community,Suppose that i changed one of the values in the node hdfs-site.xml manually, will cloudera manager override this config?example : in /etc/hadoop/conf i want to change the dfs.nameservices in  hdfs-site.xml manually and i want to how cloudera manager can override this. *** I don't want to make such a change but i want to know if cloudera manager can override this.

I'm running Ubuntu 18.04 LTS in my machine.I'm trying to install CDH 6.1.While installing agents I got error message like this below     Please help give soliution. Thank you Regard,solomonchinni

Hi Team, I  have updated nifi to 1.11.4 from 1.5.0 and the ranger policy is not working as expected .I have set up policy with /process-groups/<process group uuid>  and it opens up access to all the process groups which are present in canvas . example :- /process-groups/992fc902-0172-1000-348e-b0bae84f520f.  - i gave read and write to this process group .But i am able to go into all the process groups which are available .   regardsBharad

We are using Livy 0.7.0 with CDH 6.3 version. We are able to submit the job through Livy to CDH Cluster. We wanted to restrict the User access to the Livy session through Livy ACL.  As per the Apache Documentation added the below parameter values and restarted the Livy. But still livy is allowing all the users to access.    Any other settings needs to be enabled for ACL to work.   # Whether to enable Livy server access control, if it is true then all the income requests will# be checked if the requested user has permission.# livy.server.access-control.enabled = false livy.server.access-control.enabled=true # Allowed users to access Livy, by default any user is allowed to access Livy. If user want to# limit who could access Livy, user should list all the permitted users with comma separated.livy.server.access-control.allowed-users = raj   Thanks Raj

from ambari we can capture all version , by click on `Admin` button and the click on `stack and version`finally click on `Versions` , then we get the following details  we want to know how to capture all these version by using ambari rest api?we trycurl -u admin:admin -H 'X-Requested-By:admin' 'http://localhost:8080/api/v1/clusters/HDP/configuratons/service_config_versions'but its not return any info