I started using lxml, but now I am using the Selenium package of Python. That note might help me out in the future, thanks for that! ... View more

I decided to stick to regular Python, there wasn't really a need for Spark. As I had to get results, I didn't even use Scrapy or Nutch, but I certainly will have a look at it. It looks very interesting! ... View more

And would you recommend to run Scrapy in a PySpark environment? I will have a look at the Pydoop API, thanks for the recommendation. ... View more

1) First one confirmed (works on both hosts) hduser@ubuntuVM:~$ ssh ubuntuvm2.com Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64) * Documentation: <a href="https://help.ubuntu.com/">https://help.ubuntu.com/</a> System information disabled due to load higher than 1.0 New release '16.04.1 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Fri Aug 12 08:28:46 2016 from ubuntuvm.com hduser@ubuntuvm2:~$ 2) After a while, the connection is automatically interrupted ("closed by foreign host"). hduser@ubuntuVM2:~$ telnet ubuntuvm.com 8440 Trying 192.168.10.131... Connected to ubuntuVM.com. Escape character is '^]'. Connection closed by foreign host. hduser@ubuntuVM2:~$ ... View more

Note that I excluded some lines beginnen with "at ..." (these were just lists of services). So this is probably a problem due to a wrong hostname? INFO:root:BootStrapping hosts ['ubuntuvm1.com', 'ubuntuvm2.com'] using /usr/lib/python2.6/site-packages/ambari_server cluster primary OS: ubuntu14 with user 'hduser' sshKey File /var/run/ambari-server/bootstrap/6/sshKey password File null using tmp dir /var/run/ambari-server/bootstrap/6 ambari: ubuntuvm.com; server_port: 8080; ambari version: 2.2.2.0; user__run_as: root INFO:root:Executing parallel bootstrap Bootstrap process timed out. It was destroyed. 11 Aug 2016 18:25:16,717 INFO [pool-15-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/6 11 Aug 2016 18:25:16,717 INFO [pool-15-thread-1] BSHostStatusCollector:62 - HostList for polling on [ubuntuvm1.com, ubuntuvm2.com] 11 Aug 2016 18:25:17,509 ERROR [qtp-ambari-client-181] AbstractResourceProvider:280 - Caught AmbariException when creating a resource org.apache.ambari.server.HostNotFoundException: Host not found, hostname= at .. 11 Aug 2016 18:25:17,511 ERROR [qtp-ambari-client-181] BaseManagementHandler:57 - Caught a system exception while attempting to create a resource: An internal system exception occurred: Host not found, hostname = org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Host not found, hostname= at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:282) ... Caused by: org.apache.ambari.server.HostNotFoundException: Host not found, hostname= at .. 11 Aug 2016 18:25:22,292 INFO [qtp-ambari-agent-314] HeartBeatHandler:309 - HeartBeatHandler.sendCommands: sending ExecutionCommand for host ubuntuvm.com, role check_host, roleCommand ACTIONEXECUTE, and command ID 88-0, task ID 646 Thanks for your quick reply! ... View more

I am only testing out Hadoop, using 2 seperate laptops running virtual machines. Code: INFO:root:BootStrapping hosts ['ubuntuvm1.com', 'ubuntuvm2.com'] using /usr/lib/python2.6/site-packages/ambari_server cluster primary OS: ubuntu14 with user 'hduser' sshKey File /var/run/ambari-server/bootstrap/6/sshKey password File null using tmp dir /var/run/ambari-server/bootstrap/6 ambari: ubuntuvm.com; server_port: 8080; ambari version: 2.2.2.0; user__run_as: root INFO:root:Executing parallel bootstrap Bootstrap process timed out. It was destroyed. 11 Aug 2016 18:25:16,717 INFO [pool-15-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/6 11 Aug 2016 18:25:16,717 INFO [pool-15-thread-1] BSHostStatusCollector:62 - HostList for polling on [ubuntuvm1.com, ubuntuvm2.com] 11 Aug 2016 18:25:17,509 ERROR [qtp-ambari-client-181] AbstractResourceProvider:280 - Caught AmbariException when creating a resource org.apache.ambari.server.HostNotFoundException: Host not found, hostname= at .. 11 Aug 2016 18:25:17,511 ERROR [qtp-ambari-client-181] BaseManagementHandler:57 - Caught a system exception while attempting to create a resource: An internal system exception occurred: Host not found, hostname = org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Host not found, hostname= at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:282) ... Caused by: org.apache.ambari.server.HostNotFoundException: Host not found, hostname= at .. 11 Aug 2016 18:25:22,292 INFO [qtp-ambari-agent-314] HeartBeatHandler:309 - HeartBeatHandler.sendCommands: sending ExecutionCommand for host ubuntuvm.com, role check_host, roleCommand ACTIONEXECUTE, and command ID 88-0, task ID 646 Thanks for your quick reply! ... View more