Solved

I'm just curious and also unsure... When I'm creating a cluster (in this case, 3 kafka hosts and zookeeper), should I be adding the SCM host to this cluster as well, or does that operate separately?I've read that each cluster should have a Master host, Utility host (with cloudera manager), and 3 worker hosts. Is it a bad idea to add the host that is running SCM to my new cluster, or should I leave it separate? I've noticed that when I add the SCM host to a cluster, it is using a LOT of memory by default, which leads me to think that I should not add the SCM host to the new cluster.

Hi there. I find CDH does not support Ubuntu 18.04 yet, and plan to support ? I hope Ubuntu 18.04 can be supported, and Canonical extends Ubuntu 18.04 support to 10 years, really LTS. So I hope CDH could support Ubuntu 18.04. Thanks

Hello All,We are facing issue with one of our clients cluster, we see some files over 2GB, and would like to know the purpose of these files, and if these could be deleted to make way for some space.1. Eventserver[root@TestBed ~]# du -sh /var/lib/cloudera-scm-eventserver/v3/* |grep G5.4G /var/lib/cloudera-scm-eventserver/v3/_1bx6l.fdt1.2G /var/lib/cloudera-scm-eventserver/v3/_1hgcy.fdt1.4G /var/lib/cloudera-scm-eventserver/v3/_1mz17.fdt 2. Cloudera-Scm-Headlamp[root@TestBed ~]# du -sh /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/* |grep G2.3G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_2t.fdt2.2G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_5m.fdt2.1G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_8f.fdt Also is there a way to limit TS data in SMON & HMON, I understand that 10GB is the minimum requirement for these roles but what could be done if a clients has limited resources. Any help / guidance is appreciated Thanks

I have JSON input of the following format:{  "Id": 1000000,  "ReportName": TestReport,  "Results": [{    "Id": 1,    "ResultId": "1000000-0",    "Query": {      "Id": 001,      "Name": "TestQuery0",    }  }, {    "Id": 2,    "ResultId": "1000000-1",    "Query": {      "Id": 002,      "Name": "TestQuery1",    }  }]}These file can become quite large depending on the number of Results in the Report and I was hoping to convert the single Flowfile to multiple records for processing. However, due to the format of the JSON a SplitRecord will result in one record per split. There is one report per FlowFile and therefore only 1 root level element.I am looking for a method or strategy to split the Flowfile into smaller Records while still maintaining the cohesiveness of the report in the end when it put in HDFS. Current Strategy:Use

Hey, folks, I have a Kerberized server, and for the police I'm using Ranger, via console I can publish and consume the messages of Kafka, But now I must do through Nifi and that I'm very lost someone could help me. Thank you

Hi Cloudera Community!   I`m trying to start going through the tutorial but cannot overcome the following problem:   [cloudera@quickstart java]$ sqoop import-all-tables -m 1 --connect jdbc:mysql://quickstart:3306/retail_db --username=retail_dba --password=cloudera --compression-codec=snappy --as-parquetfile --warehouse-dir=/user/hive/warehouse --hive-import19/10/04 01:46:39 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.16.219/10/04 01:46:39 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.19/10/04 01:46:39 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override19/10/04 01:46:39 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.19/10/04 01:46:39 WARN tool.BaseSqoopTool: It seems that you're doing hive import directly into default19/10/04 01:46:39 WARN tool.BaseSqoopTool: hive warehouse directory which is not supported. Sqoop is19/10/04 01:46:39 WARN tool.BaseSqoopTool

Hi Cloudera Team, I am getting issue while sending message to Kafka topic from outside of sandbox, but same application in sandbox is able to send message to Kafka topic.Below is the screen print of these casesIn First case I am trying to send test message to my Kafka topic; that application runs without giving any error. But at cluster location topic was not showing any messages In Second case when I ran the same application in sandbox messages were send and displayed in Kafka topic.  Could you please assist me what wrong I am doing (or what extra I have to do) to make my First case successful.

Hi, I have just downloaded the quickstart Cloudera VM and when I try to start all services it fails telling me that Service mgmt has 0 Reports Managers. Cloudera Management Service requires at least 1 Reports Manager. I tried to configure Reports Manager by adding a Role instance for Reports Manager, however it requires me to give a databasename and a tablename. My question is what columns or table structure should I have for reports manager? Thanks!

Hello,   I am new to HDP and going thought the tutorials, I got stuck when I tried to launch the DAS UI, I get an error message "cannot connect to server at sandbox-hdp.hortonworks.com, FYI I use Ubuntu"   Why am i getting this error?   Is this OS issue?

Does Phoenix query server support https protocol? My Hadoop system is kerberized and ssl enabled, HTTPS encryption for wire encryption. I did not find any document for PQS request over HTTPS. If it is supported then please point me to the correct configurations parameters.

Hi Experts,we setup Cloudera cluster on  to verify the performanceCloudera was installed with Auto-TLS enabled and we are facing issues to connect to hive. [root@slave1 agent-cert]#  beeline -u jdbc:hive2://slave1:10000;ssl=true;sslTrustStore=/var/run/cloudera-scm-agent/process/72-hive-HIVESERVER2/cm-auto-host_keystore.jks;trustStorePassword=yeap4IhJzRvK5gBGVMeTahoL21BNmBF2TSi46pbQTP6WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Connecting to jdb

How I can want development scheduling algorithm and implement in yarn  Please any help  

I have newly imported HDP 2.6.5 Sandbox after removing the 2.6.4,  and tried to connect via ssh. My host machine Windows 10. My command:   ssh root@sandbox-hdp.hortonworks.com -p 2222   The error output:   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!Someone could be eavesdropping on you right now (man-in-the-middle attack)!It is also possible that a host key has just been changed.The fingerprint for the RSA key sent by the remote host isSHA256:8F0VOvcJIA2FD3Ld90kv5FiD3270onl/+uDeDBNffQ0.Please contact your system administrator.Add correct host key in /c/Users/user/.ssh/known_hosts to get rid of this message.Offending RSA key in /c/Users/user/.ssh/known_hosts:3RSA host key for [sandbox-hdp.hortonworks.com]:2222 has changed and you have requested strict checking.Host key verification failed.

I am trying to install ambari on ubuntu 16.04 LTS on server with zero internet access. I am following the steps mentioned in: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.4.0/bk_ambari-installation/content/ I hosted a local repository using apache http server. I configured to use local repository in /etc/apt/sources.list.d/ambari.list. The file contains #VERSION_NUMBER=2.7.4.0-118#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.jsondeb [trusted=yes] http://my.repository/ambari/ubuntu16/2.7.4.0-118/ Ambari main   when I run sudo apt-get update it is not fetching the binaries. I am attaching log of the process Get:1 http://my.repository/ambari/ubuntu16/2.7.4.0-118 Ambari InRelease [7379 B]Ign:1 http://my.repository/ambari/ubuntu16/2.7.4.0-118 Ambari InRelease Get:2 http://my.repository/ambari/ubuntu16/2.7.4.0-118 Ambari/main amd64 Packages [1513 B] 0% [Connecting to archive.ubuntu.com (91.189.88.31)] [Connecting

Hello,I have a scenario with a Hadoop cluster installed with HDP2.6.5 and a Kafka cluster installed with HDF 3.3.0 with Ranger Service configured.I want to store the Ranger Audit logs in HDFS so I setup in kafka the property xasecure.audit.destination.hdfs.dir pointing to the HDFS directory.Case one: when using the namenode in the URI the logs are stored in HDFS successfully (xasecure.audit.destination.hdfs.dir=hdfs://namenode_FQDN>:8020/ranger/audit)Case two: Using a haproxy, since i have namenode HA enabled and want to point always to the active NN, i get the following error2019-04-02 12:00:13,841 ERROR [kafka.async.summary.multi_dest.batch_kafka.async.summary.multi_dest.batch.hdfs_destWriter] org.apache.ranger.audit.provider.BaseAuditHandler (BaseAuditHandler.java:329) - Error writing to log file.java.io.IOException: DestHost:destPort <ha_proxy_hostname>:8085 , LocalHost:localPort <kafka_broker_hostname>/10.212.164.50:0. Failed on local exception: java.io.IOException:

Hello, I created my account with my client email and I would like to add my company email and my personal one. Is this possible? Thank you,

Hello Friends   You could help me to secure my Kafka servers for both Broker and client. As additional data I am using a KDC. As I see I currently have the security.inter.broker.protocol SASL_PLAINTEXT I don't know if it is correct or how I should secure this service.   Thank you

Hi ,Myself Ravikiran Sharma, I have purchased HDP Certified Spark Developer (HDPCSD2019) Exam on Sunday 22/oct 9:05 pm Indian time. But I did not get any confirmation mail from Hortonworks.I contacted cloudera(Natalie), she told my record is not found in the system. My account got deducted with 250$. its more than one week issue is not fixed. I want the issue to fixed as soon as possible.MY TICKET NUMBER is : 20203Please help.Thanks and RegardsRavikiran Sharma

Is the next-gen cloud-native Cloudera Data Platform available now? Which cloud providers? Is it available for on-prem kubernetes deployments (such as IBM ICP and/or RedHat Openshift)?   Thanks,   Julius

Where do I get a repo or tar file of Cloudera for on premise installation / Upgrade?   Also is CDP sandbox available?

I have a complex flow where I created 3 attributes, [name, date, json_content] extracted from other flow file data that need to go into a database. How can I take these 3 attributes convert them to a new flow file with those columns? The schema I will use has those names.  Schema:{"type": "record","name": "mytable","fields": [{"name": "name","type": [ "string" ]},{"name": "date","type": "type" : [ "null", { "type" : "long", "logicalType" : "timestamp-millis" } ], "default" : null }},{"name": "json_content","type": [ "string" ] }]}

Hello, I saw couple of blogs talks about using Superset with Kudu throw Impala.http://www.bigdatareflections.net/blog/?p=52http://www.bigdatareflections.net/blog/?cat=5 Beside the fact that the blog is from 2017, do I need to install Impala to get it work, or there is another way to plug Kudu with Superset like in the case of Druid (Superset has a native connector to Druid and a majority of Druid’s features are accessible through Superset)? RegardsAnis

Hi, I am looking for the way to convert the CSV to JSON I have a csv file with json object as value,   "name","item""david","{""currency"":""USD"",""content"":""Beauty Lips""}"     I used ConvertRecord processor with CSVReader and JsonRecordSetWriter And the output JSON is   { "name":"david", "item":"{\"currency\":\"USD\",\"content\":\"Beauty Lips\"}"}   But what I expected is   { "name":"david", "item":{"currency":"USD","name":"Beauty Lips"}}     How can I archive this? Thank you   David

Hi! I am trying to restart my Ambari Metrics Monitor on an HDP 3.0 cluster, but the component start is failing with an execution error 255.