Does anyone have any update to this? Does anyone use NiFi to read data from OPC DA, or OPC UA?   There was talk about Hortonworks having a connector for those data sources three years ago. Does anyone know if that is supported by Cloudera now?    I am thinking that the fact that OPC DA has to be on Windows and uses COM has been the sticking point for us. It may be that moving to OPC UA is possible.  ... View more

BOTHER!   It seems like my problem was an SELinux problem. httpd could work in certain situations, but not others....   https://www.cloudera.com/documentation/enterprise/latest/topics/install_cdh_disable_selinux.html   is the official documentation I needed.   ... View more

I still have this problem if I delete the load balancer on that machine and instead deploy it to any other machine. ... View more

And before anyone asks - here are the permissions on the process directory [root@fleet02 httpd]# ls -la /run/cloudera-scm-agent/process total 0 drwxr-x--x. 31 root root 620 Aug 17 11:41 . drwxr-xr-x. 6 cloudera-scm cloudera-scm 160 Aug 16 13:58 .. drwxr-x--x. 3 zookeeper zookeeper 320 Aug 16 14:50 257-zookeeper-server drwxr-x--x. 3 hdfs hdfs 380 Aug 16 15:00 264-hdfs-SECONDARYNAMENODE drwxr-x--x. 6 oozie oozie 320 Aug 16 15:04 270-oozie-OOZIE_SERVER drwxr-x--x. 4 hive hive 340 Aug 16 15:07 272-hive-WEBHCAT drwxr-x--x. 4 hive hive 340 Aug 16 15:07 273-hive-HIVEMETASTORE drwxr-x--x. 4 hive hive 400 Aug 16 15:07 274-hive-HIVESERVER2 drwxr-x--x. 4 hue hue 280 Aug 16 15:31 275-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 480 Aug 16 15:31 276-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:34 282-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 460 Aug 16 15:31 283-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:50 284-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:51 285-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:54 286-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:55 287-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:57 288-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:59 289-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 16:43 290-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:19 291-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:29 292-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:31 293-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:37 294-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:38 295-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:41 296-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 260 Aug 17 11:41 297-hue-HUE_LOAD_BALANCER drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_3711304878547591218 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-5308202058122869851 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hive-conf_etchiveconf.cloudera.hive_1582453642590273763 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_solr-conf_etcsolrconf.cloudera.solr_1630522684046020626 drwxr-xr-x. 8 root root 180 Aug 16 15:06 ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_379702062030725595 ... View more

I'd like to second this problem. I have a pristene Cloudera cluster installed on CentOS 7.3 (CDH 5.12) Cloudera manager did seem to have other problems with permissions in /var/run on other services - but once that was sorted out everything else is running apart from the Load Balancer - I am getting the same error in the same place every time.   I have temporarily modified the httpd.sh script to run "whoami" and do an ls on the file it says permission denied I have even looked at every file mentioned within that config file and can see no problems with that.     + whoami hue + ls -la /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf -rw-r-----. 1 hue hue 8255 Aug 17 11:41 /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf + perl -pi -e 's#{{CLOUDERA_HTTPD_SSL_SCRIPT_DIR}}#/usr/lib64/cmf/service/common#g' /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/hue.conf + /usr/sbin/httpd -f /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf -t httpd: Could not open configuration file /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf: Permission denied + '[' 1 '!=' 0 ']' + echo '/run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid' /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid + exit 1     Any ideas? ... View more

It is really helpful to know it wasn't me messing up. I can switch off foreign key restraints temporarily as you suggest. Thanks! ... View more

Any news on this? I am wondering about installing the whole of HDF3.0 including the Scheme Registry and SAM as well as NiFi. ... View more

We now have HDF3. It is still not clear whether we can or should use Cloudbreak. Any suggestions? ... View more

Thanks. That is really helpful. I have set myself the task to duplicate your trucking app with the HDF 3.0 sandbox and SAM+Scheme registry. I see there are some videos which should help too. ... View more

Would it be possible to update for HDF 3.0? Thanks ... View more

Still worth answering if anyone knows the answer... ... View more

Is this article still valid for HDF version 3.0 which was released recently? Are there easier ways of deploying to Amazon? ... View more

That's really helpful - and explains a big misconception I had. ... View more

I am not sure I follow your pipeline. I get the impression that Flume is being effectively deprecated in favour of (the more complicated) NiFi/HDF. Kafka is only worthwhile if you really have ALOT of events being generated at once and I am not sure why you have added it in. Is it just because you know how to do Kafka->SparkStreaming. HDFS is a file system for distributed storage. Elastic Search is quite literally a search engine. It is great for doing things like "tell me how many million records look like this", "now if I restrict it by this other criteria how many do I get now". Neither is "for batch processing". It sounds like you are trying to build a lambda architecture system. Have you found any reference architectures for that? ... View more

Thanks! I would be interested to learn more when you are ready to announce it. ... View more

I got stung by this as well. I could not figure out the solution until I found this page. I am not keen on this change to the sandbox. ... View more

Thanks, yes there was a duplicate column (the key in both tables) and I did not realise that was preventing me from using CTAS. ... View more

Wow, excellent. Thanks. I did not realise the problem was my key had the same name in both tables. Using java regexes as you described solved my problem. Thanks! ... View more

You use the phrase "Create a Local Ambari Repository". that is not what you are doing. You are merely pointing you machines to look at an external repository. It would be great if you did create a local ambari repository - but that would require a lot more explanation. I had a quick look at Nexus for this, but found it was not trivial to set up for my purposes. ... View more

To partly answer my own question... Hadoop doesn't need DNS if all the machines are already in the hosts file. So at some point in the above the /etc/hosts file was populated with 192.168.0.11 ambari1.mycluster ambari1 192.168.0.12 master1.mycluster master1 192.168.0.21 slave1.mycluster slave1 192.168.0.22 slave2.mycluster slave2 ... View more

This is great. It will save me lots of time. I am trying this on an Ubuntu host, not a Mac, and everything is fine until I get down to trying to access ambari1 through the web. I don't have ambari1 in DNS anywhere. I can "vagrant ssh ambari1" and find the IP address, but presumably that wont let me install without FQDNs. Any ideas? Thanks again. ... View more

Thanks. That is helpful ... View more

Thanks people. That is very helpful. It sounds like I have some learning to do 🙂 ... View more