I got it to work with the following in my repo I linked earlier hdfs dfs -put drivers/* /tmp/udfs beeline !connect jdbc:hive2://localhost:10000 “” ”” add jar hdfs://sandbox.hortonworks.com:8020/tmp/udfs/mongo-hadoop-hive-1.5.0-SNAPSHOT.jar; add jar hdfs://sandbox.hortonworks.com:8020/tmp/udfs/mongo-hadoop-core-1.5.0-SNAPSHOT.jar; add jar hdfs://sandbox.hortonworks.com:8020/tmp/udfs/mongodb-driver-3.0.4.jar; DROP TABLE IF EXISTS bars; CREATE EXTERNAL TABLE bars ( objectid STRING, Symbol STRING, TS STRING, Day INT, Open DOUBLE, High DOUBLE, Low DOUBLE, Close DOUBLE, Volume INT ) STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES('mongo.columns.mapping'='{"objectid":"_id", "Symbol":"Symbol", "TS":"Timestamp", "Day":"Day", "Open":"Open", "High":"High", "Low":"Low", "Close":"Close", "Volume":"Volume"}') TBLPROPERTIES('mongo.uri'='mongodb://localhost:27017/marketdata.minibars'); ... View more

Actually, I checked that ExportSnapshot includes the Tools interface, hence it does support -files and -libjars so I think the approach is still good. Sorry, false panic. ... View more

Hi Artem, I'm currently stuck in a particular use case where in I'm trying to access Hive Table data using spark.read.jdbc as shown below: export SPARK_MAJOR_VERSION=2 spark-shell import org.apache.spark.sql.{DataFrame, Row,SparkSession} val connectionProperties = new java.util.Properties() val hiveQuery = "(SELECT * from hive_table limit 10) tmp" val hiveResult = spark.read.jdbc("jdbc:hive2://hiveServerHostname:10000/hiveDBName;user=hive;password=hive", hiveQuery, connectionProperties).collect() But when I check for the results in hiveResult it's just empty. Could you please suggest what's going on here? I know we can access Hive tables using HiveSesssion and I've successfully tried that but is it possible to run hive queries and access Hive data using the above method? ... View more

Hi all, Has anyone a workaround for this problem ? I have exactly the same case. I have similar issues on the Sandbox 2.5 (VirtualBox-5.1.12-112440-Win - HDP_2.5_virtualbox). I killed the jobs with putty as root : yarn application -kill application_1482410373661_0002 but they are still visible on Ambari. [root@sandbox ~]# yarn application -kill application_1482410373661_0002 16/12/24 12:26:40 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 16/12/24 12:26:40 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/172.17.0.2:8050 16/12/24 12:26:40 INFO client.AHSProxy: Connecting to Application History server at sandbox.hortonworks.com/172.17.0.2:10200 16/12/24 12:26:44 WARN retry.RetryInvocationHandler: Exception while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over null. Not retrying because try once and fail. org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1482410373661_0002' doesn't exist in RM. I've found an issue corresponding : Tez client keeps trying to talk to RM even if RM does not know about the application https://issues.apache.org/jira/browse/TEZ-3156 This patch should be included as it was fixed for version 0.7.1 In the log (Ambary query) I can read 993 time : INFO : Map 1: 0/1 Reducer 2: 0/2 The query is the proposed in the tutorial : ( http://fr.hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#section_4) <code>SELECT truckid, avg(mpg) avgmpg FROM truck_mileage GROUP BY truckid; Any idea how to clear the history and restart without the running state ? Thanks in advance ... View more

This error can also occur if the Atlas service is turned off. From the exception stack trace: at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.io.IOException: java.net.ConnectException: Connection refused at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:107) at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:99) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.atlas.security.SecureClientUtils$1.getHttpURLConnection(SecureClientUtils.java:99) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:159) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 26 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) See the link here for more information: https://community.hortonworks.com/questions/22396/ranger-dependency-on-atlas.html ... View more