You have to do a sequence of StructTypes, a java list of structTypes, or a Sequence of struct Types. You could have passed in a Sequence of Structfields. Thats one the constructors (fields:Seq[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType So... val customSchema =StructType(Seq(StructField("year",IntegerType,true),StructField("make",StringType,true),StructField("model",StringType,true),StructField("comment",StringType,true),StructField("blank",StringType,true))) ... View more

That is only helpful if the value is not encrypted ... View more

That did the trick thanks a lot! ... View more

Ok I will try the initialize Solr.  Thanks for the idea Ben!  I will try it as son as I get to another office.  I can't do it from the current client as their firewalls are pretty tight and I have it in GCE, but I'm excited to try this. ... View more

Ben, the services are on the same host, and yes I already verified those things.   ... View more

Hey man thanks for link but that's for CDH 5.7.  Can you tell me if that will have a problem with CDH 5.9?  I don't have CDH 5.7, and I'm not going to try and revert to an old version.  ... View more

Thanks Ben,    I checked the Zookeeper logs there is not much help there    16-12-26 20:13:44,298 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1593c8f09d800f5 with negotiated timeout 30000 for client /10.128.0.2:60980 2016-12-26 20:13:44,314 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1593c8f09d800f5 type:create cxid:0x2 zxid:0x1b2e8 txntype:-1 reqpath:n/a Error Path:/solr Error:KeeperErrorCode = NoNode for /solr 2016-12-26 20:13:44,636 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1593c8f09d800f5, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2016-12-26 20:13:44,637 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.128.0.2:60980 which had sessionid 0x1593c8f09d800f5 2016-12-26 20:13:48,298 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.128.0.2:60986 2016-12-26 20:13:48,302 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.128.0.2:60986 2016-12-26 20:13:48,304 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1593c8f09d800f6 with negotiated timeout 30000 for client /10.128.0.2:60986 2016-12-26 20:13:48,322 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1593c8f09d800f6 type:create cxid:0x2 zxid:0x1b2ea txntype:-1 reqpath:n/a Error Path:/solr Error:KeeperErrorCode = NoNode for /solr 2016-12-26 20:13:48,643 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1593c8f09d800f6, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2016-12-26 20:13:48,644 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.128.0.2:60986 which had sessionid 0x1593c8f09d800f6 2016-12-26 20:14:10,000 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1593c8f09d800f3, timeout of 30000ms exceeded 2016-12-26 20:14:10,000 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1593c8f09d800f3 2016-12-26 20:14:12,000 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1593c8f09d800f4, timeout of 30000ms exceeded 2016-12-26 20:14:12,000 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1593c8f09d800f4 2016-12-26 20:14:16,000 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1593c8f09d800f5, timeout of 30000ms exceeded 2016-12-26 20:14:16,000 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1593c8f09d800f5 2016-12-26 20:14:20,000 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1593c8f09d800f6, timeout of 30000ms exceeded 2016-12-26 20:14:20,000 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1593c8f09d800f6 2016-12-26 20:14:40,993 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.128.0.2:32886 2016-12-26 20:14:40,993 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.128.0.2:32886 2016-12-26 20:14:40,995 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1593c8f09d800f7 with negotiated timeout 30000 for client /10.128.0.2:32886 2016-12-26 20:14:41,019 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.128.0.2:32892 2016-12-26 20:14:41,020 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.128.0.2:32892   Zookeeper is running, name-node-1 resolves to an IP and zookeeper and solr-server are on the same node anyway.  What could be going on?   ... View more

There isn't even a working link that takes me to the "Spark 2.0 Beta" either. http://www.cloudera.com/downloads/beta/spark2/2-0-0.html ... View more

Hi I used this helpful page to do it.  Sorry I didn't repond earlier.  Do you want the exact commands?   https://cloud.google.com/compute/docs/networking ... View more