Re: " Please install and upgrade the Stack Version...

alexmc · ‎01-23-2020

Does anyone have any update to this? Does anyone use NiFi to read data from OPC DA, or OPC UA? There was talk about Hortonworks having a connector for those data sources three years ago. Does anyone know if that is supported by Cloudera now? I am thinking that the fact that OPC DA has to be on Windows and uses COM has been the sticking point for us. It may be that moving to OPC UA is possible.

alexmc · ‎11-02-2017

My Cloudera Manager initial cluster installation wizard failed fairly early on - it has installed CM on the hosts, but failed to install any Hadoop or other roles. So I needed to restart my browser - but now I cannot get the cluster install wizard back. Any suggestions on how to do this? PS I have tried the Hosts->Run Upgrade Wizard option and that doesn't really add any new roles to hosts. Shall I just do without the initial installation wizard and set up all the hosts and roles manually through the Cloudera Manager interface? Thanks

alexmc · ‎08-17-2017

BOTHER! It seems like my problem was an SELinux problem. httpd could work in certain situations, but not others.... https://www.cloudera.com/documentation/enterprise/latest/topics/install_cdh_disable_selinux.html is the official documentation I needed.

alexmc · ‎08-17-2017

I still have this problem if I delete the load balancer on that machine and instead deploy it to any other machine.

alexmc · ‎08-17-2017

And before anyone asks - here are the permissions on the process directory [root@fleet02 httpd]# ls -la /run/cloudera-scm-agent/process total 0 drwxr-x--x. 31 root root 620 Aug 17 11:41 . drwxr-xr-x. 6 cloudera-scm cloudera-scm 160 Aug 16 13:58 .. drwxr-x--x. 3 zookeeper zookeeper 320 Aug 16 14:50 257-zookeeper-server drwxr-x--x. 3 hdfs hdfs 380 Aug 16 15:00 264-hdfs-SECONDARYNAMENODE drwxr-x--x. 6 oozie oozie 320 Aug 16 15:04 270-oozie-OOZIE_SERVER drwxr-x--x. 4 hive hive 340 Aug 16 15:07 272-hive-WEBHCAT drwxr-x--x. 4 hive hive 340 Aug 16 15:07 273-hive-HIVEMETASTORE drwxr-x--x. 4 hive hive 400 Aug 16 15:07 274-hive-HIVESERVER2 drwxr-x--x. 4 hue hue 280 Aug 16 15:31 275-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 480 Aug 16 15:31 276-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:34 282-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 460 Aug 16 15:31 283-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:50 284-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:51 285-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:54 286-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:55 287-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:57 288-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:59 289-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 16:43 290-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:19 291-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:29 292-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:31 293-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:37 294-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:38 295-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:41 296-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 260 Aug 17 11:41 297-hue-HUE_LOAD_BALANCER drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_3711304878547591218 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-5308202058122869851 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hive-conf_etchiveconf.cloudera.hive_1582453642590273763 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_solr-conf_etcsolrconf.cloudera.solr_1630522684046020626 drwxr-xr-x. 8 root root 180 Aug 16 15:06 ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_379702062030725595

alexmc · ‎08-17-2017

I'd like to second this problem. I have a pristene Cloudera cluster installed on CentOS 7.3 (CDH 5.12) Cloudera manager did seem to have other problems with permissions in /var/run on other services - but once that was sorted out everything else is running apart from the Load Balancer - I am getting the same error in the same place every time. I have temporarily modified the httpd.sh script to run "whoami" and do an ls on the file it says permission denied I have even looked at every file mentioned within that config file and can see no problems with that. + whoami hue + ls -la /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf -rw-r-----. 1 hue hue 8255 Aug 17 11:41 /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf + perl -pi -e 's#{{CLOUDERA_HTTPD_SSL_SCRIPT_DIR}}#/usr/lib64/cmf/service/common#g' /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/hue.conf + /usr/sbin/httpd -f /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf -t httpd: Could not open configuration file /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf: Permission denied + '[' 1 '!=' 0 ']' + echo '/run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid' /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid + exit 1 Any ideas?

alexmc · ‎07-14-2017

It is really helpful to know it wasn't me messing up. I can switch off foreign key restraints temporarily as you suggest. Thanks!

alexmc · ‎07-14-2017

Any news on this? I am wondering about installing the whole of HDF3.0 including the Scheme Registry and SAM as well as NiFi.

alexmc · ‎07-14-2017

I want to install HDF3 in Amazon AWS. It seems to me that I could set things up manually or I could use Cloudbreak. However all the documentation I have seen tells me that Cloudbreak does HDP. Would I be right in thinking that it should be able to install HDF3.0 fine if it has the right Ambari Blueprints? What are the options right now? My fallback position is to 1) Use Cloudbreak to install HDP in AWS 2a) install HDF3 on the existing (newly created) HDP cluster. or 2b) use Ambari to remove all services, remove ambari, and then install HDF3 on the created VMs as if they were new However I do not know if I should use an AMI already in Amazon, or build the Cloudbreak machine myself. Has anyone done this?

alexmc · ‎07-14-2017

We now have HDF3. It is still not clear whether we can or should use Cloudbreak. Any suggestions?

alexmc · ‎07-14-2017

Hi, I am following the tutorial 830 which implements the trucking example app on the HDF 3.0 sandbox. https://hortonworks.com/tutorial/real-time-event-processing-in-nifi-sam-schema-registry-and-superset/ Problem One The tutorial itself does not have a commenting system associated with it. It suggests we use this platform. Except that the "Find Answers" link is not correct and it recommends using the tags "tutorial-830 and hdf-3.0.0" which I cannot create. Problem Two I am getting a foreign key constraint problem when importing the example SAM application from the supplied JSON. Can anyone confirm if this is a problem with the tutorial or whether I have done something wrong? Thanks... The exception on screen seems to be as follows: An exception with message [com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (`streamline`.`topology`, CONSTRAINT `topology_ibfk_1` FOREIGN KEY (`versionId`) REFERENCES `topology_version` (`id`))] was thrown while processing request.

alexmc · ‎07-13-2017

Thanks. That is really helpful. I have set myself the task to duplicate your trucking app with the HDF 3.0 sandbox and SAM+Scheme registry. I see there are some videos which should help too.

alexmc · ‎07-10-2017

Would it be possible to update for HDF 3.0? Thanks

alexmc · ‎07-10-2017

Still worth answering if anyone knows the answer...

alexmc · ‎07-10-2017

Is this article still valid for HDF version 3.0 which was released recently? Are there easier ways of deploying to Amazon?

alexmc · ‎07-05-2017

That's really helpful - and explains a big misconception I had.

alexmc · ‎07-05-2017

I'm a bit confused regarding using NiFi/HDF for low latency streaming. Is it suitable when we want a fast response (eg < 0.5s) I understand that with the Streaming Analytics Manager we can do some kind of stream processing within NiFi - is that realistic? I assume the alternative is to go to Storm for stream processing, Storm or Flink, or Spark Streaming - if I am ok with the microbatch hit. If I go for Storm should I use it to fetch my events ASAP since latency is important, or still use NiFi to get the events from their source to Storm? (I know there is a question looking at part of this https://community.hortonworks.com/questions/35133/spark-streaming-20-is-it-suitable-for-low-latency.html )

alexmc · ‎05-10-2017

I am not sure I follow your pipeline. I get the impression that Flume is being effectively deprecated in favour of (the more complicated) NiFi/HDF. Kafka is only worthwhile if you really have ALOT of events being generated at once and I am not sure why you have added it in. Is it just because you know how to do Kafka->SparkStreaming. HDFS is a file system for distributed storage. Elastic Search is quite literally a search engine. It is great for doing things like "tell me how many million records look like this", "now if I restrict it by this other criteria how many do I get now". Neither is "for batch processing". It sounds like you are trying to build a lambda architecture system. Have you found any reference architectures for that?

alexmc · ‎05-10-2017

Thanks! I would be interested to learn more when you are ready to announce it.

alexmc · ‎09-23-2016

I got stung by this as well. I could not figure out the solution until I found this page. I am not keen on this change to the sandbox.

alexmc · ‎07-06-2016

Thanks, yes there was a duplicate column (the key in both tables) and I did not realise that was preventing me from using CTAS.

alexmc · ‎07-06-2016

Wow, excellent. Thanks. I did not realise the problem was my key had the same name in both tables. Using java regexes as you described solved my problem. Thanks!

alexmc · ‎07-05-2016

I am trying to join two wide tables and save the result to a new table. They happen to be CSV on disk... Now because the tables are very wide I don't really want to manually create all the columns myself - but never fear - they are just the union of the old columns. So I expect to be able to do a Create Table As Select command like this CREATE TABLE new_table AS select * from source but it is more like this CREATE TABLE combined_table AS SELECT * FROM `default.left_table` A1, `default.right_table` B1 where A1.the_key = B1.the_key or CREATE TABLE combined_table AS SELECT * FROM (SELECT * `default.left_table` A1, `default.right_table` B1 where A1.the_key = B1.the_key); Neither work for me. Is this supposed to work in Hive? If I am saving the results of my query to a new table do I always have to know the columns in that result set and create the table with those columns first? (I am using HDP 2.4.2, so a fairly recent Hive) Thanks

alexmc · ‎07-01-2016

You use the phrase "Create a Local Ambari Repository". that is not what you are doing. You are merely pointing you machines to look at an external repository. It would be great if you did create a local ambari repository - but that would require a lot more explanation. I had a quick look at Nexus for this, but found it was not trivial to set up for my purposes.

alexmc · ‎06-28-2016

To partly answer my own question... Hadoop doesn't need DNS if all the machines are already in the hosts file. So at some point in the above the /etc/hosts file was populated with 192.168.0.11 ambari1.mycluster ambari1 192.168.0.12 master1.mycluster master1 192.168.0.21 slave1.mycluster slave1 192.168.0.22 slave2.mycluster slave2

alexmc · ‎06-28-2016

This is great. It will save me lots of time. I am trying this on an Ubuntu host, not a Mac, and everything is fine until I get down to trying to access ambari1 through the web. I don't have ambari1 in DNS anywhere. I can "vagrant ssh ambari1" and find the IP address, but presumably that wont let me install without FQDNs. Any ideas? Thanks again.

alexmc · ‎06-27-2016

OK, sorry I solved this. I discovered that Sqoop client was on those two machines - and had not been restarted. I may have forgotten it when doing it manually.

alexmc · ‎06-27-2016

[EDIT: Solved - but I am leaving the question in case anyone else hits the strange error message] I am upgrading a HDP cluster from 2.2 to 2.3 using a manual upgrade process. I have reached the end and perform the final step but ambari tells me that two of my machines are not upgraded. I can't see what the problem is, nor can I fix it. Any ideas? [root@hdp01 hdp]# ambari-server set-current --cluster-name=owalhdp --version-display-name=HDP-2.3.0.0 Using python /usr/bin/python2.6 Setting current version... Enter Ambari Admin login: admin Enter Ambari Admin password: ERROR: Exiting with exit code 1. REASON: Error during setting current version. Http status code - 500. { "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: Finalization failed. More details: \nSTDOUT: Begin finalizing the upgrade of cluster owalhdp to version 2.3.0.0-2557\nThe following 2 host(s) have not been upgraded to version 2.3.0.0-2557. Please install and upgrade the Stack Version on those hosts and try again.\nHosts: hdp01.FQDN, hdp04.FQDN\n\nSTDERR: The following 2 host(s) have not been upgraded to version 2.3.0.0-2557. Please install and upgrade the Stack Version on those hosts and try again.\nHosts: hdp01.FQDN, hdp04.FQDN\n" } Now nothing in my manual process really tells me how to install and upgrade the Stack Version on individual hosts so I am at a loss for what to do.

alexmc · ‎04-28-2016

Thanks. That is helpful

alexmc · ‎04-28-2016

Thanks people. That is very helpful. It sounds like I have some learning to do 🙂

Online	Offline
Last Visited	‎01-23-2020 10:52 AM

Date Registered	‎06-24-2014 12:47 AM
Last Visited	‎01-23-2020 10:52 AM
Total Messages Posted	45
Total Kudos Received	9

Re: " Please install and upgrade the Stack Version...

Re: Can NiFi connect to OPC DA and HDA..??

How to rerun the Install Wizard?

Re: Hue Load Balancing Error

Re: Hue Load Balancing Error

Re: Hue Load Balancing Error

Re: Hue Load Balancing Error

Re: Tutorial 830 foreign key constraint fails

Re: Deploy NiFi with Cloudbreak

Cloudbreak and HDF3 - can you do it?

Re: Can we deploy HDF with CloudBreak?

Tutorial 830 foreign key constraint fails

Re: HDF Reference Application

Re: HDF Reference Application

Re: Installing HDF on AWS

Re: HDF installation on EC2

Re: Is NiFi suitable for Low-Latency real time res...

Is NiFi suitable for Low-Latency real time respons...

Re: Big Data architecture for real time and batch ...

Re: Hortonworks certified Big Data Architect ?

Re: HDP 2.5 Sandbox Virtualbox Error: "vboxnet0 n...

Re: Can I do a Hive CTAS on the results of a join?

Re: Can I do a Hive CTAS on the results of a join?

Can I do a Hive CTAS on the results of a join?

Re: Setup Hortonworks Data Platform using Vagrant,...

Re: Setup Hortonworks Data Platform using Vagrant,...

Re: Setup Hortonworks Data Platform using Vagrant,...

Re: " Please install and upgrade the Stack Version...

" Please install and upgrade the Stack Version " d...

Re: HDF be used to feed Logstash?

Re: HDF be used to feed Logstash?