Member since
09-24-2015
527
Posts
136
Kudos Received
19
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1699 | 06-30-2017 03:15 PM | |
2352 | 10-14-2016 10:08 AM | |
7643 | 09-07-2016 06:04 AM | |
9244 | 08-26-2016 11:27 AM | |
1129 | 08-23-2016 02:09 PM |
09-12-2016
06:54 PM
@Roberto Sancho I tried to use the solution suggested but I am getting following error. I ran pyspark like this. pyspark --jars ./spark-csv_2.11-1.4.0.jar --jars ./commons-csv-1.4.jar --jars ./univocity-parsers-2.2.1.jar Error: ===== Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/hdp/2.4.2.0-258/spark/python/pyspark/sql/readwriter.py", line 137, in load
return self._df(self._jreader.load(path))
File "/usr/hdp/2.4.2.0-258/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
File "/usr/hdp/2.4.2.0-258/spark/python/pyspark/sql/utils.py", line 45, in deco
return f(*a, **kw)
File "/usr/hdp/2.4.2.0-258/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o44.load.
: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.databricks.spark.csv.DefaultSource
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62)
at scala.util.Try.orElse(Try.scala:82)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:62)
... 14 more
>>>
... View more
08-25-2016
06:29 PM
Make sure to use Ambari-2.2.2. A related issue was fixed in 2.2.2 https://issues.apache.org/jira/browse/AMBARI-15774
... View more
08-24-2016
04:38 PM
2 Kudos
Look like the keyless ssh is not setup properly, check the document at http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_set_up_password-less_ssh.html Or install Ambari agent manually
... View more
08-23-2016
02:09 PM
Hi: The /etc/hosts had a wrong ip, now is working
... View more
08-16-2016
12:20 PM
I added a comment to Lester's response above yesterday... it looks like you can't see it unless you click a link to show more details. At any rate, to answer your question, in this scenario pig scripts are working fine in the grunt shell. It appears something may have gotten corrupted in the Ambari view, as recreating it with the same settings (that were set up following the Ambari Views documentation) did give a view that is now working fine. This issue is resolved for me. It may be worth noting, just for interest sake, that this is the first time this Pig view has worked on this cluster. Originally when it was set up with Ambari 2.1.0 there was an error that, upon researching, looked like it was related to a bug that had been fixed in Ambari 2.2.0. After upgrading Ambari I got the error above. Now that it has been recreated it is working fine.
... View more
07-14-2016
06:31 AM
@Roberto Sancho Check this https://community.hortonworks.com/questions/40121/about-hue-access-hdfs-ha.html
... View more
07-09-2016
06:30 PM
1 Kudo
@Roberto Sancho Following two articles will give little more detailed steps to accomplish the same task. [1] MySQL setup for Ambari https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/content/_using_ambari_with_mysql.html [2] How do I change an existing Ambari DB Postgres to MySQL? http://www.hadoopadmin.co.in/bigdata/how-do-i-change-an-existing-ambari-db-postgres-to-mysql/
... View more
07-11-2016
07:47 AM
2 Kudos
Is the notebook server running on a different machine? Or in a virtual machine? localhost means 'this computer', so the default settings require it to be running on the same machine as the browser. Just check if you are using chrome. Settings-> Enable guest browsing.
... View more