Member since
06-24-2014
45
Posts
9
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
487 | 06-27-2016 08:57 PM |
01-23-2020
05:25 AM
Does anyone have any update to this? Does anyone use NiFi to read data from OPC DA, or OPC UA? There was talk about Hortonworks having a connector for those data sources three years ago. Does anyone know if that is supported by Cloudera now? I am thinking that the fact that OPC DA has to be on Windows and uses COM has been the sticking point for us. It may be that moving to OPC UA is possible.
... View more
11-02-2017
04:16 AM
My Cloudera Manager initial cluster installation wizard failed fairly early on - it has installed CM on the hosts, but failed to install any Hadoop or other roles. So I needed to restart my browser - but now I cannot get the cluster install wizard back. Any suggestions on how to do this? PS I have tried the Hosts->Run Upgrade Wizard option and that doesn't really add any new roles to hosts. Shall I just do without the initial installation wizard and set up all the hosts and roles manually through the Cloudera Manager interface? Thanks
... View more
Labels:
- Labels:
-
Cloudera Manager
08-17-2017
05:33 AM
1 Kudo
BOTHER! It seems like my problem was an SELinux problem. httpd could work in certain situations, but not others.... https://www.cloudera.com/documentation/enterprise/latest/topics/install_cdh_disable_selinux.html is the official documentation I needed.
... View more
08-17-2017
05:00 AM
I still have this problem if I delete the load balancer on that machine and instead deploy it to any other machine.
... View more
08-17-2017
04:03 AM
And before anyone asks - here are the permissions on the process directory [root@fleet02 httpd]# ls -la /run/cloudera-scm-agent/process total 0 drwxr-x--x. 31 root root 620 Aug 17 11:41 . drwxr-xr-x. 6 cloudera-scm cloudera-scm 160 Aug 16 13:58 .. drwxr-x--x. 3 zookeeper zookeeper 320 Aug 16 14:50 257-zookeeper-server drwxr-x--x. 3 hdfs hdfs 380 Aug 16 15:00 264-hdfs-SECONDARYNAMENODE drwxr-x--x. 6 oozie oozie 320 Aug 16 15:04 270-oozie-OOZIE_SERVER drwxr-x--x. 4 hive hive 340 Aug 16 15:07 272-hive-WEBHCAT drwxr-x--x. 4 hive hive 340 Aug 16 15:07 273-hive-HIVEMETASTORE drwxr-x--x. 4 hive hive 400 Aug 16 15:07 274-hive-HIVESERVER2 drwxr-x--x. 4 hue hue 280 Aug 16 15:31 275-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 480 Aug 16 15:31 276-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:34 282-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 460 Aug 16 15:31 283-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:50 284-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:51 285-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:54 286-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:55 287-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:57 288-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:59 289-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 16:43 290-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:19 291-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:29 292-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:31 293-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:37 294-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:38 295-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:41 296-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 260 Aug 17 11:41 297-hue-HUE_LOAD_BALANCER drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_3711304878547591218 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-5308202058122869851 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hive-conf_etchiveconf.cloudera.hive_1582453642590273763 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_solr-conf_etcsolrconf.cloudera.solr_1630522684046020626 drwxr-xr-x. 8 root root 180 Aug 16 15:06 ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_379702062030725595
... View more
08-17-2017
04:02 AM
I'd like to second this problem. I have a pristene Cloudera cluster installed on CentOS 7.3 (CDH 5.12) Cloudera manager did seem to have other problems with permissions in /var/run on other services - but once that was sorted out everything else is running apart from the Load Balancer - I am getting the same error in the same place every time. I have temporarily modified the httpd.sh script to run "whoami" and do an ls on the file it says permission denied I have even looked at every file mentioned within that config file and can see no problems with that. + whoami
hue
+ ls -la /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf
-rw-r-----. 1 hue hue 8255 Aug 17 11:41 /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf
+ perl -pi -e 's#{{CLOUDERA_HTTPD_SSL_SCRIPT_DIR}}#/usr/lib64/cmf/service/common#g' /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/hue.conf
+ /usr/sbin/httpd -f /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf -t
httpd: Could not open configuration file /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf: Permission denied
+ '[' 1 '!=' 0 ']'
+ echo '/run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid'
/run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid
+ exit 1 Any ideas?
... View more
07-14-2017
10:46 PM
It is really helpful to know it wasn't me messing up. I can switch off foreign key restraints temporarily as you suggest. Thanks!
... View more
07-14-2017
06:15 PM
1 Kudo
Any news on this? I am wondering about installing the whole of HDF3.0 including the Scheme Registry and SAM as well as NiFi.
... View more
07-14-2017
06:13 PM
1 Kudo
I want to install HDF3 in Amazon AWS. It seems to me that I could set things up manually or I could use Cloudbreak. However all the documentation I have seen tells me that Cloudbreak does HDP. Would I be right in thinking that it should be able to install HDF3.0 fine if it has the right Ambari Blueprints? What are the options right now? My fallback position is to 1) Use Cloudbreak to install HDP in AWS 2a) install HDF3 on the existing (newly created) HDP cluster. or 2b) use Ambari to remove all services, remove ambari, and then install HDF3 on the created VMs as if they were new However I do not know if I should use an AMI already in Amazon, or build the Cloudbreak machine myself. Has anyone done this?
... View more
- Tags:
- Cloudbreak
- hdf3
Labels:
07-14-2017
06:12 PM
We now have HDF3. It is still not clear whether we can or should use Cloudbreak. Any suggestions?
... View more
07-14-2017
03:03 PM
1 Kudo
Hi, I am following the tutorial 830 which implements the trucking example app on the HDF 3.0 sandbox. https://hortonworks.com/tutorial/real-time-event-processing-in-nifi-sam-schema-registry-and-superset/ Problem One The tutorial itself does not have a commenting system associated with it. It suggests we use this platform. Except that the "Find Answers" link is not correct and it recommends using the tags "tutorial-830 and hdf-3.0.0" which I cannot create. Problem Two I am getting a foreign key constraint problem when importing the example SAM application from the supplied JSON. Can anyone confirm if this is a problem with the tutorial or whether I have done something wrong? Thanks... The exception on screen seems to be as follows:
An exception with message [com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (`streamline`.`topology`, CONSTRAINT `topology_ibfk_1` FOREIGN KEY (`versionId`) REFERENCES `topology_version` (`id`))] was thrown while processing request.
... View more
Labels:
- Labels:
-
Cloudera DataFlow (CDF)
07-13-2017
11:08 AM
Thanks. That is really helpful. I have set myself the task to duplicate your trucking app with the HDF 3.0 sandbox and SAM+Scheme registry. I see there are some videos which should help too.
... View more
07-10-2017
05:11 PM
Is this article still valid for HDF version 3.0 which was released recently? Are there easier ways of deploying to Amazon?
... View more
07-05-2017
03:45 PM
That's really helpful - and explains a big misconception I had.
... View more
07-05-2017
09:29 AM
I'm a bit confused regarding using NiFi/HDF for low latency streaming. Is it suitable when we want a fast response (eg < 0.5s) I understand that with the Streaming Analytics Manager we can do some kind of stream processing within NiFi - is that realistic? I assume the alternative is to go to Storm for stream processing, Storm or Flink, or Spark Streaming - if I am ok with the microbatch hit. If I go for Storm should I use it to fetch my events ASAP since latency is important, or still use NiFi to get the events from their source to Storm? (I know there is a question looking at part of this https://community.hortonworks.com/questions/35133/spark-streaming-20-is-it-suitable-for-low-latency.html )
... View more
Labels:
- Labels:
-
Apache NiFi
05-10-2017
01:35 PM
I am not sure I follow your pipeline. I get the impression that Flume is being effectively deprecated in favour of (the more complicated) NiFi/HDF. Kafka is only worthwhile if you really have ALOT of events being generated at once and I am not sure why you have added it in. Is it just because you know how to do Kafka->SparkStreaming. HDFS is a file system for distributed storage. Elastic Search is quite literally a search engine. It is great for doing things like "tell me how many million records look like this", "now if I restrict it by this other criteria how many do I get now". Neither is "for batch processing". It sounds like you are trying to build a lambda architecture system. Have you found any reference architectures for that?
... View more
05-10-2017
01:24 PM
Thanks! I would be interested to learn more when you are ready to announce it.
... View more
09-23-2016
08:43 AM
I got stung by this as well. I could not figure out the solution until I found this page. I am not keen on this change to the sandbox.
... View more
07-06-2016
11:55 AM
Thanks, yes there was a duplicate column (the key in both tables) and I did not realise that was preventing me from using CTAS.
... View more
07-06-2016
11:54 AM
Wow, excellent. Thanks. I did not realise the problem was my key had the same name in both tables. Using java regexes as you described solved my problem. Thanks!
... View more
07-05-2016
03:34 PM
1 Kudo
I am trying to join two wide tables and save the result to a new table. They happen to be CSV on disk... Now because the tables are very wide I don't really want to manually create all the columns myself - but never fear - they are just the union of the old columns. So I expect to be able to do a Create Table As Select command like this CREATE TABLE new_table AS select * from source but it is more like this CREATE TABLE combined_table AS SELECT * FROM `default.left_table` A1, `default.right_table` B1
where A1.the_key = B1.the_key or CREATE TABLE combined_table AS SELECT * FROM (SELECT * `default.left_table` A1, `default.right_table` B1 where A1.the_key = B1.the_key); Neither work for me. Is this supposed to work in Hive? If I am saving the results of my query to a new table do I always have to know the columns in that result set and create the table with those columns first? (I am using HDP 2.4.2, so a fairly recent Hive) Thanks
... View more
Labels:
- Labels:
-
Apache Hive
07-01-2016
10:01 AM
You use the phrase "Create a Local Ambari Repository". that is not what you are doing. You are merely pointing you machines to look at an external repository. It would be great if you did create a local ambari repository - but that would require a lot more explanation. I had a quick look at Nexus for this, but found it was not trivial to set up for my purposes.
... View more
06-28-2016
07:01 PM
To partly answer my own question... Hadoop doesn't need DNS if all the machines are already in the hosts file. So at some point in the above the /etc/hosts file was populated with 192.168.0.11 ambari1.mycluster ambari1 192.168.0.12 master1.mycluster master1 192.168.0.21 slave1.mycluster slave1 192.168.0.22 slave2.mycluster slave2
... View more
06-28-2016
06:49 PM
This is great. It will save me lots of time. I am trying this on an Ubuntu host, not a Mac, and everything is fine until I get down to trying to access ambari1 through the web. I don't have ambari1 in DNS anywhere. I can "vagrant ssh ambari1" and find the IP address, but presumably that wont let me install without FQDNs. Any ideas? Thanks again.
... View more
06-27-2016
08:57 PM
OK, sorry I solved this. I discovered that Sqoop client was on those two machines - and had not been restarted. I may have forgotten it when doing it manually.
... View more
06-27-2016
08:54 PM
[EDIT: Solved - but I am leaving the question in case anyone else hits the strange error message] I am upgrading a HDP cluster from 2.2 to 2.3 using a manual upgrade process. I have reached the end and perform the final step but ambari tells me that two of my machines are not upgraded. I can't see what the problem is, nor can I fix it. Any ideas? [root@hdp01 hdp]# ambari-server set-current --cluster-name=owalhdp --version-display-name=HDP-2.3.0.0 Using python /usr/bin/python2.6 Setting current version... Enter Ambari Admin login: admin Enter Ambari Admin password:
ERROR: Exiting with exit code 1.
REASON: Error during setting current version. Http status code - 500.
{
"status" : 500,
"message" : "org.apache.ambari.server.controller.spi.SystemException: Finalization failed. More details: \nSTDOUT: Begin finalizing the upgrade of cluster owalhdp to version 2.3.0.0-2557\nThe following 2 host(s) have not been upgraded to version 2.3.0.0-2557. Please install and upgrade the Stack Version on those hosts and try again.\nHosts: hdp01.FQDN, hdp04.FQDN\n\nSTDERR: The following 2 host(s) have not been upgraded to version 2.3.0.0-2557. Please install and upgrade the Stack Version on those hosts and try again.\nHosts: hdp01.FQDN, hdp04.FQDN\n"
} Now nothing in my manual process really tells me how to install and upgrade the Stack Version on individual hosts so I am at a loss for what to do.
... View more
Labels:
04-28-2016
02:26 PM
Thanks people. That is very helpful. It sounds like I have some learning to do 🙂
... View more