Q&A

What are the limitations of a CDP 7.0.3 Trial Version aside on the 60-day trial period? is there any services that is not there in the CDP trial version?

Hello,We have got our Domain renamed from ABC to XYZ.com. Since then neither Cloudera Manager UI nor any other service UI's are accessible. As per my understanding, we would need to update the /etc/hosts file of each node in the cluster to reflect the new Domain name. however it would be great if anyone could advise on the below 2 things:1. Do we really need to get /etc/host/file updated, OR is it taken care by Cloudera2. Where all do we need to update the hostnames with new Domain name to have a fully functional cluster back to operational statusCurrently if I manually enter the complete address (ip+new doamin name) I can login to Cloudera Manager UI but from Cloudera Manager when I try to open any service UI's like NN UI it fails. Any help is much appreciated RegardsWert

How can I do a cloudera upgrade version without having any internet in cluster? May I know any way on how to do it?

Hi all! What I get:JSON with an arbitrary number of entries beneath "table". { "head": { "dnr": "122" }, "table": [ { "zn": 1, "datum": "20200211", <=== yyyyMMdd CET "uhrzvon": "10:40" <=== HH:mm CET }, { "zn": 2, "datum": "20200211", "uhrzvon": "10:47" }, { "zn": 3, "datum": "20200212", "uhrzvon": "08:11" } ]} What I need:Each entry contains "datum" and "uhrzvon" (CET) which I need to join and convert to timestamp (UTC). Because of a later JOLT-Transformation, I thought I can change the original entry of "datum" to use this in JOLT.  { "head": { "dnr": "122" }, "table": [ { "zn": 1, "datum": "2020-02-11 09:40:00.000", <=== yyyy-MM-dd HH:mm:ss.SSS UTC "uhrzvon": "10:40" }, { "zn": 2, "datum": "2020-02-11 09:47:00.000", "uhrzvon": "10:47" },

+ export HADOOP_IDENT_STRING=hdfs+ HADOOP_IDENT_STRING=hdfs+ '[' -n '' ']'+ '[' mkdir '!=' namenode ']'+ acquire_kerberos_tgt hdfs.keytab+ '[' -z hdfs.keytab ']'+ KERBEROS_PRINCIPAL=+ '[' '!' -z '' ']'+ '[' -n '' ']'+ '[' validate-writable-empty-dirs = namenode ']'+ '[' file-operation = namenode ']'+ '[' bootstrap = namenode ']'+ '[' failover = namenode ']'+ '[' transition-to-active = namenode ']'+ '[' initializeSharedEdits = namenode ']'+ '[' initialize-znode = namenode ']'+ '[' format-namenode = namenode ']'+ '[' monitor-decommission = namenode ']'+ '[' jnSyncWait = namenode ']'+ '[' nnRpcWait = namenode ']'+ '[' -safemode = '' -a get = '' ']'+ '[' monitor-upgrade = namenode ']'+ '[' finalize-upgrade = namenode ']'+ '[' rolling-upgrade-prepare = namenode ']'+ '[' rolling-upgrade-finalize = namenode ']'+ '[' nnDnLiveWait = namenode ']'+ '[' monitor-offline = namenode ']'+ '[' refresh-datanode = namenode ']'+ '[' put = namenode ']'+ '[' mkdir = namenode ']'+ '[' setfacl = namenode ']'+

Hello.I'm trying to use my own certs from CA(I converted them to right format)I'm already configurated TLS for cloudera manager server (when I signed on https://{MASTER_IP}:7183 - I see it.File cm_init.txt setsettings AGENT_TLS truesetsettings WEB_TLS truesetsettings NEED_AGENTS_VALIDATION truesetsettings AUTO_TLS_TYPE NONEsetsettings KEYSTORE_PATH /opt/agentcerts/MYKEYSTORE.jkssetsettings KEYSTORE_PASSWORD MYPASSWORDsetsettings TRUSTSTORE_PATH /opt/agentcerts/MYTRUSTSTORE.jkssetsettings TRUSTSTORE_PASSWORD MYPASSWORDAlso I used them for HUE web UI and Hue Load Balancer - its also work too.But, I need to configurate certs to agents too, because I didn't see status of my agents in (hosts ->all hosts tab).I changed file /etc/cloudera-scm-agent/config.ini, there main changes on him: use_tls=1verify_cert_file=/opt/agentcerts/MYCERT.pemclient_key_file=/opt/agentcerts/MYKEY.pemclient_keypw_file=/opt/agentcerts/KEYPASS.pwclient_cert_file=/opt/agentcerts/MYCERT.pemverify_cert_dir=

Hello,How to update DNS part in host name, example from 192.168.0.1.test-1.pl to 192.168.0.1.test.co.pl so that we can open CM GUI via 192.168.0.1.test.co.pl:7180 and other web gui (hue, impala etc) with the new dns name test.co.pl.  RegardsAnm

I have a shell action node that is able to send notifications to a chat application and can be used as a transition route if another action node in the workflow fails. What I'd like to be able to do, is use wf:user() and wf:lastErrorNode() in that notification. My idea was to pass these values to my shell script as arguments, but when I place 'wf:X()' as the argument to my shell action node, it doesn't resolve to anything and just gives me the raw string as the argument. Is there a way to use these in a shell action node? 

Rather than using the automatically generated names that are given to action nodes i.e. shell-12ba, I would like to be able to set the name myself. Is this possible? If so how? 

I keep getting the follwoing health error message:   The filesystem checkpoint is 22 hour(s), 40 minute(s) old. This is 2,267.75% of the configured checkpoint period of 1 hour(s). Critical threshold: 400.00%. 10,775 transactions have occurred since the last filesystem checkpoint. This is 1.08% of the configured checkpoint transaction target of 1,000,000.   What is causing this and how can I get it to stop.   Logs:   Number of transactions: 8 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 6 SyncTimes(ms): 132 Number of transactions: 8 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 7 SyncTimes(ms): 155 Finalizing edits file /dfs/nn/current/edits_inprogress_0000000000000021523 -> /dfs/nn/current/edits_0000000000000021523-0000000000000021530 Starting log segment at 21531 Rescanning after 30000 milliseconds Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). list c

Hello,I am puzzelling for some time if there is possibility to refer the both attributes and content of flow file and transform the content by combining both.Example:Assume a flow file with uuid=2c8a7845-8567-4eed-97ee-4dcd1c18668e and the json content{   "data":"some business data"} The result of the transformation should be a content like{   "nifi-filename": "2c8a7845-8567-4eed-97ee-4dcd1c18668e",   "nifi-content": {       "data": "some business data"   }} How can I reach that.? Best regardsJaro

Hello,I am trying to configure the StandardSSLContextService controller to work with either the PutElasticsearchHttpRecord or PutElasticsearchRecord processors. The Elasticsearch cluster has been secured by Opendistro for ElasticsearchI am able to do the following below successfully:openssl s_client -connect atlas201.cs:9200 -debug -state -cert esnode.pem -key esnode-key.pem -CAfile root-ca.pem But when I try those keys and certs in the StandardSSLContextService I am unable to get it to work. I am not too familiar with Opendistro as this was already a pre-configured cluster that I am working on.Could someone tell me what the following parameters need to be for me to successfully connect and ingest data into the ES cluster? 

I am using MiNiFi(0.5.0) to pick and transfer files from a Linux machine to NiFi(1.9.1). In few cases i can see duplicate files being transferred to NiFi.Flow is setup as belowGetFile ->LogAttribute -> PutFile(archive) -> RemoteProcessGroupLog 1:minifi-app.log:2020-09-13 06:09:00,760 INFO [Timer-Driven Process Thread-4] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237]] (237 bytes) to nifi://nifi1.myorg.com:7074 in 48 milliseconds at a rate of 4.74 KB/sec Log 2 :minifi-app.log:2020-09-13 06:09:01,910 INFO [Timer-Driven Process Thread-5] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.

How can I get credentials to acces Ambari repo ?

We have HDP cluster version `2.6.5` and ambari `2.6.1` version Cluster include 3 masters machines , and 211 data-nodes machines ( workers machines ) , all machines are `rhel 7.2` version Example master1.sys77.com , master2.sys77.com , master3.sys77.com … And data nodes machines as worker01.sys77.com , worker02.sys77.com ----> worker211.sys77.com Now we want to change the domain name to `bigdata.com` instead of `sys77.com` What is the procedure to replace the `domain name` (`sys77.com`) for Hadoop cluster ? ( HDP cluster with ambari )

Dear experts, I have installed apache kafka 2.4 on one node. I am having SSL and SASL(Kerberos) enabled for kafka broker and now enabled SASL for zookeeper. However when starting the broker , i am getting the below error, could you please help on this ? --error log--- [2020-09-07 14:11:09,761] DEBUG Client principal is "kafka/broker0@KAFKA.SECURE". (org.apache.zookeeper.Login)[2020-09-07 14:11:09,761] DEBUG Server principal is "krbtgt/KAFKA.SECURE@KAFKA.SECURE". (org.apache.zookeeper.Login)[2020-09-07 14:11:09,764] INFO TGT valid starting at: Mon Sep 07 14:11:09 IST 2020 (org.apache.zookeeper.Login)[2020-09-07 14:11:09,765] INFO TGT expires: Tue Sep 08 14:11:09 IST 2020 (org.apache.zookeeper.Login)[2020-09-07 14:11:09,765] INFO TGT refresh sleeping until: Tue Sep 08 09:30:58 IST 2020 (org.apache.zookeeper.Login)[2020-09-07 14:11:09,765] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)[2020-09-07 14:11:09,766] DEBUG creating

Hello, Would like some assistance/guidance on Kerberos. Our domain name has changed and since then our applications are unable to connect to Hadoop cluster. We are using MIT Kerberos.   Regards Wert

I haven't worked on Parquet file but i have requirement to convert Parquet file to CSV using NiFi.Appreciate if you could guide me on this.

I have different input files for different sources. Can I create a common Nifi flow which processes all the different kind of files? Each files will be having different types of columns and different types of output table to be loaded. For example, File 1 will be having column A, Column B to be loaded to Table AB, File 2 will be having column C, Column D, Column E to be loaded to Table CDE. Can I achieve this in a single flow or should I create different flows for different types of files?

Hello.I'm using an ExecuteSQL Processor to extract data from Oracle DB. The query has multiple joins and returns a large number of fields.The problem is that ExecuteSQL Processor returns a single flow file (avro format) that is huge. I want to split the flow file (based in a number of rows, for example) and then merge them at the proper moment.I have read about QueryDatabaseTable and GenerateFetchTable which are processors that can split the output flow files but looks like these processors aren't able to perform joins.. Does anyone knows a workaround?

I have a scenario where the columns in a CSV are consistently changing, I want to validate this CSV irrespective of the changes Currently I am validating using a static schema, by placing a static schema in Schema Text property of Validate Record processor. But everytime there is a change in file format(a new column is added or removed) I have to update the schema in Schema text property of processor.Is there a way I can make this dynamic ?Currently I am using NIFI 1.11.4(I couldnt find InferSchema processor or JSON to Avro converter)

Hello, I get the below error message in my NiFi logs when I tried to write a file to HDFS, but when i try to write a file with in the hadoop cluster it works fine, but from NiFi it fails with the below message.My Nifi service is started by root and, when I have a local NiFi instance i am able to write the file to HDFS, where as from the NiFi cluster am unable to do so, any help would be highly appreciated, I am trying another solution, if that works then I will post it over here.  2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056] o.a.n.c.s.StandardProcessScheduler Starting LogMessage[id=b4bc6d2c-0173-1000-0000-00002905a41b]2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056] o.a.n.controller.StandardProcessorNode Starting LogMessage[id=b4bc6d2c-0173-1000-0000-00002905a41b]2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056] o.a.n.c.s.StandardProcessScheduler Starting LogMessage[id=b4bd264b-0173-1000-0000-000018f91304]2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056

Hi All,   Good Afternoon.   We have CDH 5.14 and Impala 2.11 in one of our Cloudera cluster. We have a requirement to upgrade Impala to 3.2 in the same cluster.   Please suggest us the best way to achieve this. Can we just upgrade Impala without upgrading CDH.   Regards, Banshi.

Hello,I tested ListenHTTP on a single node under some load generated by jmeter. I observed that for the one or two client threads the ListenHTTP works fine, however,having 10 client threads leads to 60% request refused by 503,Service Unavailable. Is there any parameter, which helps to come accross the issue?Also the component does not indicate any problems in the GUI. Best regardsJaro

Hello,   I was using Cloudera Enterprise 5.16 (Trial) which is now downgraded to Cloudera Express, since then I am facing issues starting Reports Manager. I am using the default embedded database.   CM / CDH - 5.16.2     2020-09-06 14:14:29,706 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.2020-09-06 14:14:29,711 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 4100.2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 41002020-09-06 14:14:29,725 INFO com.cloudera.enterprise.ssl.SSLFactory: Using default java truststore for verification of server certificates in HTTPS communication.2020-09-06 14:14:29,761 WARN com.cloudera.cmf.BasicScmProxy: Exception while getting fetch configDefaults hash: nonejava.net.ConnectException: Connection refused at java.net.PlainSocketIm