About alexmc

alexmc · ‎01-23-2020

Does anyone have any update to this? Does anyone use NiFi to read data from OPC DA, or OPC UA? There was talk about Hortonworks having a connector for those data sources three years ago. Does anyone know if that is supported by Cloudera now? I am thinking that the fact that OPC DA has to be on Windows and uses COM has been the sticking point for us. It may be that moving to OPC UA is possible.

alexmc · ‎11-02-2017

My Cloudera Manager initial cluster installation wizard failed fairly early on - it has installed CM on the hosts, but failed to install any Hadoop or other roles. So I needed to restart my browser - but now I cannot get the cluster install wizard back. Any suggestions on how to do this? PS I have tried the Hosts->Run Upgrade Wizard option and that doesn't really add any new roles to hosts. Shall I just do without the initial installation wizard and set up all the hosts and roles manually through the Cloudera Manager interface? Thanks

alexmc · ‎08-17-2017

BOTHER! It seems like my problem was an SELinux problem. httpd could work in certain situations, but not others.... https://www.cloudera.com/documentation/enterprise/latest/topics/install_cdh_disable_selinux.html is the official documentation I needed.

alexmc · ‎08-17-2017

I still have this problem if I delete the load balancer on that machine and instead deploy it to any other machine.

alexmc · ‎08-17-2017

And before anyone asks - here are the permissions on the process directory [root@fleet02 httpd]# ls -la /run/cloudera-scm-agent/process total 0 drwxr-x--x. 31 root root 620 Aug 17 11:41 . drwxr-xr-x. 6 cloudera-scm cloudera-scm 160 Aug 16 13:58 .. drwxr-x--x. 3 zookeeper zookeeper 320 Aug 16 14:50 257-zookeeper-server drwxr-x--x. 3 hdfs hdfs 380 Aug 16 15:00 264-hdfs-SECONDARYNAMENODE drwxr-x--x. 6 oozie oozie 320 Aug 16 15:04 270-oozie-OOZIE_SERVER drwxr-x--x. 4 hive hive 340 Aug 16 15:07 272-hive-WEBHCAT drwxr-x--x. 4 hive hive 340 Aug 16 15:07 273-hive-HIVEMETASTORE drwxr-x--x. 4 hive hive 400 Aug 16 15:07 274-hive-HIVESERVER2 drwxr-x--x. 4 hue hue 280 Aug 16 15:31 275-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 480 Aug 16 15:31 276-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:34 282-hue-HUE_LOAD_BALANCER drwxr-x--x. 9 hue hue 460 Aug 16 15:31 283-hue-HUE_SERVER drwxr-x--x. 4 hue hue 280 Aug 16 15:50 284-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:51 285-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:54 286-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:55 287-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:57 288-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 15:59 289-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 16 16:43 290-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:19 291-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:29 292-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:31 293-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 280 Aug 17 11:37 294-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:38 295-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 300 Aug 17 11:41 296-hue-HUE_LOAD_BALANCER drwxr-x--x. 4 hue hue 260 Aug 17 11:41 297-hue-HUE_LOAD_BALANCER drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_3711304878547591218 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-5308202058122869851 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_hive-conf_etchiveconf.cloudera.hive_1582453642590273763 drwxr-xr-x. 4 root root 100 Aug 16 15:06 ccdeploy_solr-conf_etcsolrconf.cloudera.solr_1630522684046020626 drwxr-xr-x. 8 root root 180 Aug 16 15:06 ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_379702062030725595

alexmc · ‎08-17-2017

I'd like to second this problem. I have a pristene Cloudera cluster installed on CentOS 7.3 (CDH 5.12) Cloudera manager did seem to have other problems with permissions in /var/run on other services - but once that was sorted out everything else is running apart from the Load Balancer - I am getting the same error in the same place every time. I have temporarily modified the httpd.sh script to run "whoami" and do an ls on the file it says permission denied I have even looked at every file mentioned within that config file and can see no problems with that. + whoami hue + ls -la /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf -rw-r-----. 1 hue hue 8255 Aug 17 11:41 /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf + perl -pi -e 's#{{CLOUDERA_HTTPD_SSL_SCRIPT_DIR}}#/usr/lib64/cmf/service/common#g' /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/hue.conf + /usr/sbin/httpd -f /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf -t httpd: Could not open configuration file /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf: Permission denied + '[' 1 '!=' 0 ']' + echo '/run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid' /run/cloudera-scm-agent/process/297-hue-HUE_LOAD_BALANCER/httpd.conf is invalid + exit 1 Any ideas?

alexmc · ‎07-14-2017

It is really helpful to know it wasn't me messing up. I can switch off foreign key restraints temporarily as you suggest. Thanks!

alexmc · ‎07-14-2017

Any news on this? I am wondering about installing the whole of HDF3.0 including the Scheme Registry and SAM as well as NiFi.

alexmc · ‎07-14-2017

I want to install HDF3 in Amazon AWS. It seems to me that I could set things up manually or I could use Cloudbreak. However all the documentation I have seen tells me that Cloudbreak does HDP. Would I be right in thinking that it should be able to install HDF3.0 fine if it has the right Ambari Blueprints? What are the options right now? My fallback position is to 1) Use Cloudbreak to install HDP in AWS 2a) install HDF3 on the existing (newly created) HDP cluster. or 2b) use Ambari to remove all services, remove ambari, and then install HDF3 on the created VMs as if they were new However I do not know if I should use an AMI already in Amazon, or build the Cloudbreak machine myself. Has anyone done this?

alexmc · ‎07-14-2017

We now have HDF3. It is still not clear whether we can or should use Cloudbreak. Any suggestions?

alexmc · ‎07-14-2017

Hi, I am following the tutorial 830 which implements the trucking example app on the HDF 3.0 sandbox. https://hortonworks.com/tutorial/real-time-event-processing-in-nifi-sam-schema-registry-and-superset/ Problem One The tutorial itself does not have a commenting system associated with it. It suggests we use this platform. Except that the "Find Answers" link is not correct and it recommends using the tags "tutorial-830 and hdf-3.0.0" which I cannot create. Problem Two I am getting a foreign key constraint problem when importing the example SAM application from the supplied JSON. Can anyone confirm if this is a problem with the tutorial or whether I have done something wrong? Thanks... The exception on screen seems to be as follows: An exception with message [com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (`streamline`.`topology`, CONSTRAINT `topology_ibfk_1` FOREIGN KEY (`versionId`) REFERENCES `topology_version` (`id`))] was thrown while processing request.

alexmc · ‎07-13-2017

Thanks. That is really helpful. I have set myself the task to duplicate your trucking app with the HDF 3.0 sandbox and SAM+Scheme registry. I see there are some videos which should help too.

alexmc · ‎07-13-2017

By the way - there is a guide to the HDF sandbox at https://hortonworks.com/tutorial/hortonworks-sandbox-guide/section/2/ but unfortunately it has not yet been updated for HDF 3.0 - Hopefully it will be soon.

alexmc · ‎07-10-2017

Are there sandboxes which I can deploy to Amazon AWS? I want to be able to show a client what it can do rather than carry my laptop over to him.

alexmc · ‎07-10-2017

Would it be possible to update for HDF 3.0? Thanks

alexmc · ‎07-10-2017

Still worth answering if anyone knows the answer...

alexmc · ‎07-10-2017

Is this article still valid for HDF version 3.0 which was released recently? Are there easier ways of deploying to Amazon?

alexmc · ‎07-05-2017

That's really helpful - and explains a big misconception I had.

alexmc · ‎07-05-2017

I'm a bit confused regarding using NiFi/HDF for low latency streaming. Is it suitable when we want a fast response (eg < 0.5s) I understand that with the Streaming Analytics Manager we can do some kind of stream processing within NiFi - is that realistic? I assume the alternative is to go to Storm for stream processing, Storm or Flink, or Spark Streaming - if I am ok with the microbatch hit. If I go for Storm should I use it to fetch my events ASAP since latency is important, or still use NiFi to get the events from their source to Storm? (I know there is a question looking at part of this https://community.hortonworks.com/questions/35133/spark-streaming-20-is-it-suitable-for-low-latency.html )

alexmc · ‎05-10-2017

I am not sure I follow your pipeline. I get the impression that Flume is being effectively deprecated in favour of (the more complicated) NiFi/HDF. Kafka is only worthwhile if you really have ALOT of events being generated at once and I am not sure why you have added it in. Is it just because you know how to do Kafka->SparkStreaming. HDFS is a file system for distributed storage. Elastic Search is quite literally a search engine. It is great for doing things like "tell me how many million records look like this", "now if I restrict it by this other criteria how many do I get now". Neither is "for batch processing". It sounds like you are trying to build a lambda architecture system. Have you found any reference architectures for that?

alexmc · ‎05-10-2017

Thanks! I would be interested to learn more when you are ready to announce it.

alexmc · ‎02-15-2017

I have been asked if I can give a Hadoop Architectures talk at a conference. My problem now is what to include or not. I can cover basic Hadoop without too much trouble - but what about the extra non-Hadoop stuff which is typically included in the ecosystem? What do you think is vital to be mentioned? I am thinking: some kind of text database for a searchable front end (eg SolR or ElasticSearch) Some kind of notebook (eg iPython, Zeppelin) Spark running on YARN, Some kind of ingestion (sqoop, spark streaming, flume) Streaming options (Spark Streaming, Storm, Flink) workflow (oozie) If anyone is willing to chat to me about their architecture - and perhaps send me a "back of an envelope" architecture diagram then that would be absolutely great. Comment here or message me.

alexmc · ‎09-23-2016

I got stung by this as well. I could not figure out the solution until I found this page. I am not keen on this change to the sandbox.

alexmc · ‎07-11-2016

I was going to say that the bug sounded familiar - and I see that I commented on that yarn-3227 over a year ago. Glad to hear that bug has been fixed now. But I also see that the original posted is using Yarn 2.7.1 suggesting it is a different problem. 😞

alexmc · ‎07-06-2016

Thanks, yes there was a duplicate column (the key in both tables) and I did not realise that was preventing me from using CTAS.

alexmc · ‎07-06-2016

Wow, excellent. Thanks. I did not realise the problem was my key had the same name in both tables. Using java regexes as you described solved my problem. Thanks!

alexmc · ‎07-05-2016

I am trying to join two wide tables and save the result to a new table. They happen to be CSV on disk... Now because the tables are very wide I don't really want to manually create all the columns myself - but never fear - they are just the union of the old columns. So I expect to be able to do a Create Table As Select command like this CREATE TABLE new_table AS select * from source but it is more like this CREATE TABLE combined_table AS SELECT * FROM `default.left_table` A1, `default.right_table` B1 where A1.the_key = B1.the_key or CREATE TABLE combined_table AS SELECT * FROM (SELECT * `default.left_table` A1, `default.right_table` B1 where A1.the_key = B1.the_key); Neither work for me. Is this supposed to work in Hive? If I am saving the results of my query to a new table do I always have to know the columns in that result set and create the table with those columns first? (I am using HDP 2.4.2, so a fairly recent Hive) Thanks

alexmc · ‎07-01-2016

You use the phrase "Create a Local Ambari Repository". that is not what you are doing. You are merely pointing you machines to look at an external repository. It would be great if you did create a local ambari repository - but that would require a lot more explanation. I had a quick look at Nexus for this, but found it was not trivial to set up for my purposes.

alexmc · ‎06-28-2016

To partly answer my own question... Hadoop doesn't need DNS if all the machines are already in the hosts file. So at some point in the above the /etc/hosts file was populated with 192.168.0.11 ambari1.mycluster ambari1 192.168.0.12 master1.mycluster master1 192.168.0.21 slave1.mycluster slave1 192.168.0.22 slave2.mycluster slave2

alexmc · ‎06-28-2016

This is great. It will save me lots of time. I am trying this on an Ubuntu host, not a Mac, and everything is fine until I get down to trying to access ambari1 through the web. I don't have ambari1 in DNS anywhere. I can "vagrant ssh ambari1" and find the IP address, but presumably that wont let me install without FQDNs. Any ideas? Thanks again.

Online	Offline
Last Visited	‎01-23-2020 10:52 AM

Date Registered	‎06-24-2014 12:47 AM
Last Visited	‎01-23-2020 10:52 AM
Total Messages Posted	45
Total Kudos Received	9

Re: " Please install and upgrade the Stack Version...

Re: Can NiFi connect to OPC DA and HDA..??

How to rerun the Install Wizard?

Re: Hue Load Balancing Error

Re: Hue Load Balancing Error

Re: Hue Load Balancing Error

Re: Hue Load Balancing Error

Re: Tutorial 830 foreign key constraint fails

Re: Deploy NiFi with Cloudbreak

Cloudbreak and HDF3 - can you do it?

Re: Can we deploy HDF with CloudBreak?

Tutorial 830 foreign key constraint fails

Re: HDF Reference Application

Re: Sandbox for HDF

Re: Sandbox for HDF

Re: HDF Reference Application

Re: Installing HDF on AWS

Re: HDF installation on EC2

Re: Is NiFi suitable for Low-Latency real time res...

Is NiFi suitable for Low-Latency real time respons...

Re: Big Data architecture for real time and batch ...

Re: Hortonworks certified Big Data Architect ?

What should go into a Hadoop Architectures talk?

Re: HDP 2.5 Sandbox Virtualbox Error: "vboxnet0 n...

Re: We are seeing an issue in our kerberized env t...

Re: Can I do a Hive CTAS on the results of a join?

Re: Can I do a Hive CTAS on the results of a join?

Can I do a Hive CTAS on the results of a join?

Re: Setup Hortonworks Data Platform using Vagrant,...

Re: Setup Hortonworks Data Platform using Vagrant,...

Re: Setup Hortonworks Data Platform using Vagrant,...