Solved

we want to insert flowfile attributes(Filename,FileSize,Time) into Postgres Table Using NiFi

Exception in thread "main" java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:294)at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)Caused by: java.io.FileNotFoundException: Log directory specified does not exist: hdfs://nameservice1/user/spark/applicationHistoryat org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:268)at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:212)at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHisto

Hi ,We are currently upgrading from HDP 2.3.4 to HDP 2.6.5 following the following reference document as suggested in the following link.https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-upgrade/content/ambari_upgrade_guide.htmlOurs is an environment without internet connection so we downloaded and created the offline repository.We already upgraded the ambari from 2.4.2 to Ambari 2.6.2.2 successfully and performed all the prerequisites for HDP upgrade. We are currently got stuck in the "Register and Install Target Version" phase. We have registered the local repository for HDP 2.6.5 and from ambari>stacks and versions>manage version we are trying to install the new softwares which is failing with the following error message.This issue is only happening for only one of the hosts in cluster.Rest all other hosts the installation was successful."Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Noth

Hello!   Please help me about minimal hardware requirements for out small cluster.   We decided to make very small production cluster with high availability for archiving purposes based on Cloudera CDH 6.3.3 (community version) Storage size planning as about of 10-20 TiB Workload planning: - every 2 minutes ETL from external oracle to local parquet about of 500-1000 rows of data - periodically (very rarely) analytic queries to hive about search through all of parquets - periodically (very rarely) run spark ad-hoc tasks with goals same as above Components: - Cloudera manager - HDFS - Hive - Hue - Impala - Spark - Yarn with MR2 - ZooKeeper - Streamsets parcel (as a part of cloudera) We want to use only 3 hosts (not more) and disaster of any of this host must not crash all system. So we plan to place all of above components to all of the hosts. In another words, each component will be on each host. Is it normal and available or someone may advise another alignment? We a

Dear experts,I am running HDP 2.6.4. Recently i have installed Ranger and integrated Hive. When i am trying to change Ambari Cluster Name and Ranger Hive Service Name, i am receiving the below errors in the hiveserver2.log. Could you please help ? 'cluster1' is the original cluster name,I want to change the cluster name to a different name. ERROR [Thread-13]: util.PolicyRefresher(PolicyRefresher.java:loadPolicyfromPolicyAdmin(285)) - PolicyRefresher(serviceName=cluster1_hive): failed to find service. Will clean up local cache of policies (-1)org.apache.ranger.plugin.util.RangerServiceNotFoundException: cluster1_hive   

Hello.I need advice there I can see log(audit) of hdfs operations.For example(mkdir, putFromLocal, delete file,change file).I tried use command from cmd "hdfs dfs -mkdir /tmp/mydir" - and don't see this log.There can I find it? 

I upgraded Cloud Manager from 5.10.0 to 6.3.0, then i tried to run 'wget http://10.0.0.209:7180/api/v13/clusters/{clusterName}/services/{serviceName}/clientConfig', got a error 404. Checked the log file of cloudera-scm-server, got the following:2020-09-15 04:11:57,977 INFO scm-web-16747:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'.java.util.NoSuchElementException: Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'.at com.cloudera.api.dao.impl.ManagerDaoBase.findService(ManagerDaoBase.java:484)......but I checked the services via accessing API /clusters/{clusterName}/services/{serviceName}, actually the service 'yarn' is in this cluster, ref:[root@rhel76x64 ~]# curl http://<user>:<password>@10.0.0.209:7180/api/v13/clusters/platform-pod10-eason-cdh/services/yarn/{"healthChecks" : [ {"name" : "YARN_JOBHISTORY_HEALTH","summary" : "GOOD","suppressed" : false}, {"name"

CDP 7.0.3 version does it have any free version like cloudera express? do we have that here? also how many days is the trial version of it?

What are the limitations of a CDP 7.0.3 Trial Version aside on the 60-day trial period? is there any services that is not there in the CDP trial version?

Hello,We have got our Domain renamed from ABC to XYZ.com. Since then neither Cloudera Manager UI nor any other service UI's are accessible. As per my understanding, we would need to update the /etc/hosts file of each node in the cluster to reflect the new Domain name. however it would be great if anyone could advise on the below 2 things:1. Do we really need to get /etc/host/file updated, OR is it taken care by Cloudera2. Where all do we need to update the hostnames with new Domain name to have a fully functional cluster back to operational statusCurrently if I manually enter the complete address (ip+new doamin name) I can login to Cloudera Manager UI but from Cloudera Manager when I try to open any service UI's like NN UI it fails. Any help is much appreciated RegardsWert

How can I do a cloudera upgrade version without having any internet in cluster? May I know any way on how to do it?

Hi all! What I get:JSON with an arbitrary number of entries beneath "table". { "head": { "dnr": "122" }, "table": [ { "zn": 1, "datum": "20200211", <=== yyyyMMdd CET "uhrzvon": "10:40" <=== HH:mm CET }, { "zn": 2, "datum": "20200211", "uhrzvon": "10:47" }, { "zn": 3, "datum": "20200212", "uhrzvon": "08:11" } ]} What I need:Each entry contains "datum" and "uhrzvon" (CET) which I need to join and convert to timestamp (UTC). Because of a later JOLT-Transformation, I thought I can change the original entry of "datum" to use this in JOLT.  { "head": { "dnr": "122" }, "table": [ { "zn": 1, "datum": "2020-02-11 09:40:00.000", <=== yyyy-MM-dd HH:mm:ss.SSS UTC "uhrzvon": "10:40" }, { "zn": 2, "datum": "2020-02-11 09:47:00.000", "uhrzvon": "10:47" },

+ export HADOOP_IDENT_STRING=hdfs+ HADOOP_IDENT_STRING=hdfs+ '[' -n '' ']'+ '[' mkdir '!=' namenode ']'+ acquire_kerberos_tgt hdfs.keytab+ '[' -z hdfs.keytab ']'+ KERBEROS_PRINCIPAL=+ '[' '!' -z '' ']'+ '[' -n '' ']'+ '[' validate-writable-empty-dirs = namenode ']'+ '[' file-operation = namenode ']'+ '[' bootstrap = namenode ']'+ '[' failover = namenode ']'+ '[' transition-to-active = namenode ']'+ '[' initializeSharedEdits = namenode ']'+ '[' initialize-znode = namenode ']'+ '[' format-namenode = namenode ']'+ '[' monitor-decommission = namenode ']'+ '[' jnSyncWait = namenode ']'+ '[' nnRpcWait = namenode ']'+ '[' -safemode = '' -a get = '' ']'+ '[' monitor-upgrade = namenode ']'+ '[' finalize-upgrade = namenode ']'+ '[' rolling-upgrade-prepare = namenode ']'+ '[' rolling-upgrade-finalize = namenode ']'+ '[' nnDnLiveWait = namenode ']'+ '[' monitor-offline = namenode ']'+ '[' refresh-datanode = namenode ']'+ '[' put = namenode ']'+ '[' mkdir = namenode ']'+ '[' setfacl = namenode ']'+

Hello.I'm trying to use my own certs from CA(I converted them to right format)I'm already configurated TLS for cloudera manager server (when I signed on https://{MASTER_IP}:7183 - I see it.File cm_init.txt setsettings AGENT_TLS truesetsettings WEB_TLS truesetsettings NEED_AGENTS_VALIDATION truesetsettings AUTO_TLS_TYPE NONEsetsettings KEYSTORE_PATH /opt/agentcerts/MYKEYSTORE.jkssetsettings KEYSTORE_PASSWORD MYPASSWORDsetsettings TRUSTSTORE_PATH /opt/agentcerts/MYTRUSTSTORE.jkssetsettings TRUSTSTORE_PASSWORD MYPASSWORDAlso I used them for HUE web UI and Hue Load Balancer - its also work too.But, I need to configurate certs to agents too, because I didn't see status of my agents in (hosts ->all hosts tab).I changed file /etc/cloudera-scm-agent/config.ini, there main changes on him: use_tls=1verify_cert_file=/opt/agentcerts/MYCERT.pemclient_key_file=/opt/agentcerts/MYKEY.pemclient_keypw_file=/opt/agentcerts/KEYPASS.pwclient_cert_file=/opt/agentcerts/MYCERT.pemverify_cert_dir=

Hello,How to update DNS part in host name, example from 192.168.0.1.test-1.pl to 192.168.0.1.test.co.pl so that we can open CM GUI via 192.168.0.1.test.co.pl:7180 and other web gui (hue, impala etc) with the new dns name test.co.pl.  RegardsAnm

I have a shell action node that is able to send notifications to a chat application and can be used as a transition route if another action node in the workflow fails. What I'd like to be able to do, is use wf:user() and wf:lastErrorNode() in that notification. My idea was to pass these values to my shell script as arguments, but when I place 'wf:X()' as the argument to my shell action node, it doesn't resolve to anything and just gives me the raw string as the argument. Is there a way to use these in a shell action node? 

Rather than using the automatically generated names that are given to action nodes i.e. shell-12ba, I would like to be able to set the name myself. Is this possible? If so how? 

I keep getting the follwoing health error message:   The filesystem checkpoint is 22 hour(s), 40 minute(s) old. This is 2,267.75% of the configured checkpoint period of 1 hour(s). Critical threshold: 400.00%. 10,775 transactions have occurred since the last filesystem checkpoint. This is 1.08% of the configured checkpoint transaction target of 1,000,000.   What is causing this and how can I get it to stop.   Logs:   Number of transactions: 8 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 6 SyncTimes(ms): 132 Number of transactions: 8 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 7 SyncTimes(ms): 155 Finalizing edits file /dfs/nn/current/edits_inprogress_0000000000000021523 -> /dfs/nn/current/edits_0000000000000021523-0000000000000021530 Starting log segment at 21531 Rescanning after 30000 milliseconds Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). list c

Hello,I am puzzelling for some time if there is possibility to refer the both attributes and content of flow file and transform the content by combining both.Example:Assume a flow file with uuid=2c8a7845-8567-4eed-97ee-4dcd1c18668e and the json content{   "data":"some business data"} The result of the transformation should be a content like{   "nifi-filename": "2c8a7845-8567-4eed-97ee-4dcd1c18668e",   "nifi-content": {       "data": "some business data"   }} How can I reach that.? Best regardsJaro

Hello,I am trying to configure the StandardSSLContextService controller to work with either the PutElasticsearchHttpRecord or PutElasticsearchRecord processors. The Elasticsearch cluster has been secured by Opendistro for ElasticsearchI am able to do the following below successfully:openssl s_client -connect atlas201.cs:9200 -debug -state -cert esnode.pem -key esnode-key.pem -CAfile root-ca.pem But when I try those keys and certs in the StandardSSLContextService I am unable to get it to work. I am not too familiar with Opendistro as this was already a pre-configured cluster that I am working on.Could someone tell me what the following parameters need to be for me to successfully connect and ingest data into the ES cluster? 

I am using MiNiFi(0.5.0) to pick and transfer files from a Linux machine to NiFi(1.9.1). In few cases i can see duplicate files being transferred to NiFi.Flow is setup as belowGetFile ->LogAttribute -> PutFile(archive) -> RemoteProcessGroupLog 1:minifi-app.log:2020-09-13 06:09:00,760 INFO [Timer-Driven Process Thread-4] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237]] (237 bytes) to nifi://nifi1.myorg.com:7074 in 48 milliseconds at a rate of 4.74 KB/sec Log 2 :minifi-app.log:2020-09-13 06:09:01,910 INFO [Timer-Driven Process Thread-5] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.

How can I get credentials to acces Ambari repo ?

We have HDP cluster version `2.6.5` and ambari `2.6.1` version Cluster include 3 masters machines , and 211 data-nodes machines ( workers machines ) , all machines are `rhel 7.2` version Example master1.sys77.com , master2.sys77.com , master3.sys77.com … And data nodes machines as worker01.sys77.com , worker02.sys77.com ----> worker211.sys77.com Now we want to change the domain name to `bigdata.com` instead of `sys77.com` What is the procedure to replace the `domain name` (`sys77.com`) for Hadoop cluster ? ( HDP cluster with ambari )

Dear experts, I have installed apache kafka 2.4 on one node. I am having SSL and SASL(Kerberos) enabled for kafka broker and now enabled SASL for zookeeper. However when starting the broker , i am getting the below error, could you please help on this ? --error log--- [2020-09-07 14:11:09,761] DEBUG Client principal is "kafka/broker0@KAFKA.SECURE". (org.apache.zookeeper.Login)[2020-09-07 14:11:09,761] DEBUG Server principal is "krbtgt/KAFKA.SECURE@KAFKA.SECURE". (org.apache.zookeeper.Login)[2020-09-07 14:11:09,764] INFO TGT valid starting at: Mon Sep 07 14:11:09 IST 2020 (org.apache.zookeeper.Login)[2020-09-07 14:11:09,765] INFO TGT expires: Tue Sep 08 14:11:09 IST 2020 (org.apache.zookeeper.Login)[2020-09-07 14:11:09,765] INFO TGT refresh sleeping until: Tue Sep 08 09:30:58 IST 2020 (org.apache.zookeeper.Login)[2020-09-07 14:11:09,765] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)[2020-09-07 14:11:09,766] DEBUG creating

Hello, Would like some assistance/guidance on Kerberos. Our domain name has changed and since then our applications are unable to connect to Hadoop cluster. We are using MIT Kerberos.   Regards Wert