Q&A

Hi, we have a couple of brokers that need to be life-cycled out.  We have reassigned all topics to the other brokers in the cluster and Stopped the Kafka role on that host.  To totally remove it, is it as simple as choosing the Instance and then from the Action Menu -> Delete?  CM -> Kafka -> Instances -> (choose instance) -> Actions Menu -> Delete I don't see a "Decommission" option like with a Datanode.  Anything need to be done in ZooKeeper or does the "Delete" option handle that?  Thank you!

Hi,    I'm planning to bring the Kafka as a Service. I would need any client or command-line tools to add a cluster. Right now, I'm able to add from Cloudera Manager, but I would want to do it from the CLI. Do we have any API or any leads to get this done?

Hello, After 13th January, all my store streamsets stopped storing data with an error similar like this below. We are collecting data from ETW and IIS log data, that is first collect, then transform, and finally store. This last step is falling. Can someone help?   KUDU_15 - Row key '(string id="3d755d2d-949e-4fba-9925-63f658b44773", int32 year=2020, int32 month=2, int32 day=21, string host="vipe-exch03", string type="etw", string environment="prod", string groupid="iis")' not found ----------------------------------- Error Stack Trace KUDU_15 - Row key '(string id="3d755d2d-949e-4fba-9925-63f658b44773", int32 year=2020, int32 month=2, int32 day=21, string host="vipe-exch03", string type="etw", string environment="prod", string groupid="iis")' not found com.streamsets.pipeline.api.base.OnRecordErrorException: KUDU_15 - Row key '(string id="3d755d2d-949e-4fba-9925-63f658b44773", int32 year=2020, int32 month=2, int32 day=21, string host="vipe-exch03", string type="etw", string en

Hi All, I would like know regarding how can i verify or validate the SSL certificates expiry for all the hadoop components installed on my HDP cluster. Would really appreciate getting any help on methods to check the SSL certificates. Thanks

Hi everyone So I'm an intern and it's my first time using apache NiFi. I wonder if there is any way to create a whole apache NiFi project using java: i.e: I have a button in java I need a code that able me to when I press this button a processor will be dragged and dropped in apache NiFi UI !!Is it possible ?? if it is, how can we make it?

Hello,  I'm trying to connect from beeline/jdbc to ssl enabled Hive (3.1.0) via ssl enabled Knox ( 1.0.0 ) but without success.   1) The same connection works perfectly fine with ssl disabled on Hive side ( Ambari -> Hive -> Configs -> Settings -> Use SSL = false )     beeline -u "jdbc:hive2://knox_gateway:8443/;ssl=true;sslTrustStore=user_keystore.jks;trustStorePassword=password;transportMode=http;httpPath=gateway/default/hive" -n valid_knox_user -p valid_knox_passwordSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See <a href="<a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a>" target=

hi all   in our cluster HDP - 2.6.4 , we have kafka cluster with 11 nodes ( 11 machines )   restart the kafka service , from AMBARI REST API , will stop/start is brutal    when we do it from rest API then its cause corrupted indexes files in kafka topic partitions   what we search is , ambari rest API that do the restart in parallel   for example   first stop kafka01   then   start kafka01    wait until alert gone (from ambari)   then stop kafka02 , and start kafka02   etc....     is this process could be done by ambari rest API?    

Hello, I would like to understand the internal in / outs of Cloudera flow. How CM interacts  with other components in the Hadoop stack ( HDFS / YARN/ IMPALA etc.), would like to know if there are any references material / documents which I can go through to enhance my knowledge in this area. Appreciate any information in this regard. Regards

hi all   in ambari we have kafka service , and this service include 7 kafka brokers   how to stop/start the kafka broker on specific host?   in this example kafka service contain the following kafka broker hosts   kafka01 kafka02 kafka03 kafka04 kafka05 kafka06 kafka07   other repleted post - https://community.cloudera.com/t5/Support-Questions/how-to-identify-alert-from-kafka-service-in-ambari-by-REST/m-p/284675#M211372  

Hi,   I have large HBase table and would like to add a new column family with 1 column. I know about "alter" which adds the to schema to all regions. But it really means adding schema to region ? Is it only change schema or rewrite data ?      Regards, Cezary

Hi Team, @jsensharma @MattWho @Shelton @venkatsambath  I have installed CDH 6.2.0 on CM 6.2.0 with the help of local repository parcel file created by me following the Cloudera documents ( https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_create_local_parcel_repo.html ). I have successfully installed the CDH. however, CDH 6.2 comes with in-built parcels for Spark-2. which I am unable to find in the parcels.  Please assist me on how do I add Spark-2 as a service in my Cloudera 6.2.0 now. Appreciate your assistance.  

Hi All,    Kindly help me in adding services to the cluster. I have been struggling to add Hive and other services as the Test connection is getting failed with the error. Able to find server and databases, and logon failed.    The error in the scm logs state no pg_hba.con available for the host and user. However, have checked them all.  Also, attached are the pictures as I work on a remote desktop where I cannot login to this platform.  Kindly assist.    Thanks, Ansar

move a file from: EX - http://77.17.6.166/tls/file/nn.csvto: hostname:77.17.6.122    remote directory - /tls/fileand rename as hello.csv guide me with processor an there respective configuration 

I plan to get a Cloudera certification and need help on following questions:Question #1. I reviewed the page https://www.cloudera.com/about/training/certification.html, It looks like that CCP Data Engineer is the only certification that has not been suspended or retired.  Am I right on this?Question #2. To prepare DE575, the only recommended Cloudera course is "Spark and Hadoop Developer" training course. according to this page. https://www.cloudera.com/about/training/certification/ccp-data-engineer.html. Should I consider other courses?Questions #3. My workplace uses HDP. Do I need to get familiar with products like CDH before taking the exam? 

Hi allI have the following Nifi template which ingests data from a csv file and put them in a database.   The template is available herehttps://github.com/cetic/fadi/blob/master/examples/basic/basic_example_final_template.xml The template is composed of two processes InvokeHTTP and PutDatabaseRecord   InvokeHTTPPutDatabase It has also two controller services : CSV Reader and DBCPCOnnectionPool   The complete explanation of this small example is available herehttps://github.com/cetic/fadi/blob/master/USERGUIDE.md#3-ingest-measurements The csv file is herehttps://raw.githubusercontent.com/cetic/fadi/master/examples/basic/sample_data.csv   The table creation is: CREATE TABLE example_basic (measure_ts TIMESTAMP NOT NULL,temperature FLOAT (50));   Until Apache Nifi 1.9.2 everything works are expected!Since Apache Nifi 10.0.0, I have an Error in the PutDatabaseRecord processor "org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert

Hi all, tried to use the new parameter thing on PutEmail processor in 1.10.0.   Checked spelling and value a thousand times. Doing the same thing with variables works fine. Any help is appreciated!

  the following API example , will stop the kafka service in ambari export service=kafkacurl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.STOP.$service","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME","service_name":"$service"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service the problem is about the syntax - _PARSE_.STOP.$service and we see that actually service value - kafka not set in - PARSE.STOP.$service so ambri saw the name as - PARSE.STOP.$service and not PARSE.STOP.kafka any idea how we can set the value kafka inside the json syntax?

Hi Folks ,I am facing one issue , i have tried all the methods but its not working can you please help on this .  one column have data like below its there row wise (Original column) IT Strategy& Architecture BDC India [MITX 999],Corporate & IC Solution Delivery [SVII],Operations Solution Delivery [SVIA],Mainframe Service [MLEM],Strategy & Architecture [MLEL] the above 3 rows i need to split into two columns like below 1st columnIT Strategy& Architecture BDC India,Corporate & IC Solution Delivery,Operations Solution Delivery,Mainframe Service,Strategy & Architecture  2nd columns [MITX 999],[SVII],[SVIA],[MLEM], [MLEL] please help , working new for impala data lake . Thank You ..!

Hi all, there are process groups in my NiFi (1.10.0) which contain disabled processors.These process groups are versioned and all changes are committed. As soon as the NiFi instance is restarted these process groups are marked as changed and "show local changes" here:     I would like to know if this behaviour of NiFi is "normal". So far I have not noticed this fact.It is misleading when uncommitted changes are shown and it is not possible to avoid that disabled processors are included.   Does anyone know the phenomenon described above?Is there a solution for it? Thanks for any advice.

Hello All,     When upgrading HDP cluster to 3.1.0 version -Restarting yarn DNS resgistry failed due to  ERROR 'resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh -H -E /usr/hdp/3.1.0.0-78/hadoop-yarn/bin/yarn --config /usr/hdp/3.1.0.0-78/hadoop/conf --daemon start registrydns' returned 1. ERROR: Cannot set priority of registrydns process 123223'. I ignored it and proceeded with the upgrade. But at Finalize Pre check the task failed and the only option available is to delete the unhealthy and host and proceed with the upgrade.   How can I resolve this issue. @jsensharma

Hello,   We are having difficulty downloading files from the open-source HDP platform through the links  https://www.cloudera.com/downloads/hdp.html  https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/ambari_repositories.html https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/hdp_315_repositories.html   Before we had this access, considering that HDP is an Open-Source platform.   Without access to these links above, we have no way to install and test the HDP.   The link for HDF (https://www.cloudera.com/downloads/hdf.html) is already no more pointing to HDF, but now for CDF. The HDF are no more available?   Thanks.

I'm looking at the Spark parcel that is included with CDH 6.3, and seeing something strange.  Specifically, the source code under python/pyspark doesn't seem to match up with what you see in the upstream (Apache) repository.   Take one file as an example. /opt/cloudera/parcels/CDH/lib/spark/python/pyspark/context.py   Jumping to line 114 of this file, we see:     self._callsite = first_spark_call() or CallSite(None, None, None) if gateway is not None and gateway.gateway_parameters.auth_token is None: raise ValueError( "You are trying to pass an insecure Py4j gateway to Spark. This" " is not allowed as it is a security risk.")     Now, this file should come from Spark 2.4.0 (see both /opt/cloudera/parcels/CDH/lib/spark/RELEASE and /opt/cloudera/parcels/CDH/lib/spark/python/pyspark/version.py files, which indicate this).  However, if you look at the upstream source, at the same line

  hi all   we have HDP version 2.6.5 and ambari 2.6.2.2   we configured the following    Metrics Service operation mode=distributed hbase.rootdir=hdfs://hdfsha:8020/user/ams/hbase   and all the rest according to - https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode   so ambari configured to save the data on hdfs instead the local disk   after metrics collector service restart we saw the following from log     2020-02-20 20:09:15,770 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.2020-02-20 20:11:35,769 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.     and     2020-02-20 20:30:10,306 INFO org.apache.hadoop.metrics2.impl.MetricsSys

Hi all,  Does any one know how to create timeseries graph for httpfs service from cloudera manager.  Currently we have httpfs service on 3 servers and We need to build a graph on top of it. Is there any query which I Can use of it to create a graph.   Thanks, Raghu.

I have enabled Kerberos authentication for Kafka as the documentation suggests and indeed producing to topics and consuming from topics requires authentication. Surprisingly, topic creation and deletion do not require authentication. Could somebody tell me whether this goes wrong ? Really appreciate it.   CDH version: 5.15.1 CDK version: 4.1.0 test command:     kafka-topics --create --zookeeper <zookeeper-host>:2181 --replication-factor 2 --partitions 3 --topic test2kafka-topics --delete --zookeeper <zookeeper-host>:2181 --topic test2