Member since
02-02-2016
583
Posts
518
Kudos Received
98
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4189 | 09-16-2016 11:56 AM | |
| 1749 | 09-13-2016 08:47 PM | |
| 6943 | 09-06-2016 11:00 AM | |
| 4174 | 08-05-2016 11:51 AM | |
| 6246 | 08-03-2016 02:58 PM |
05-26-2016
03:10 PM
1 Kudo
@Sunile Manjee Can you please try this? sqlContext.setConf("mapred.input.dir.recursive","true") OR sqlContext.setConf("mapreduce.input.fileinputformat.input.dir.recursive","true")
... View more
05-26-2016
01:38 PM
Hi @Kirk Haslbeck, don't know which version you are using but if you didn't see then take a look at below Jira it might help. https://issues.apache.org/jira/browse/SPARK-7442
... View more
05-26-2016
11:30 AM
1 Kudo
@Tajinderpal Singh Quick checks. 1. whether you have enough memory to run hive shell on that node. free -m 2. how many hive shell sessions running? ps -aef|grep hive EDITED: 3. Are you running hive on tez? if yes then check if you have available resources on Resource manager UI.
... View more
05-25-2016
08:26 PM
@vagrawalSetup YARN queue:
(Optional) You can setup/configure a YARN queue to customize what portion of the cluster the Spark job should use. To do this follow the two steps below: i. Open the Yarn Queue Manager view to setup a queue for Spark with below capacities:
Capacity: 50% Max Capacity: 90% (on sandbox, do not reduce below this or the Spark jobs will not run) screen-shot-2016-05-25-at-92342-pm.png
... View more
05-25-2016
07:35 PM
2 Kudos
@Mamta Chawla There are 3 ways. 1. hadoop fs -cat /tmp/test.sh|exec sh 2. You can install HDP NFS and mount the hdfs directory on local file system from where you can execute your script. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs-nfs-instructions.html 3. You can write an oozie shell workflow and call your .sh HDFS file inside the workflow. http://rogerhosto.com/apache-oozie-shell-script-example/
... View more
05-25-2016
07:18 PM
@Timothy Spann Currently Sqoop doesn't direct ORC hive table import, please refer below Jira. https://issues.apache.org/jira/browse/SQOOP-2192 As a workaround, we need to import the data in a temp table with text format through sqoop and further copy the data from tmp table to ORC format table.
... View more
05-25-2016
04:37 PM
4 Kudos
@Albert Domingo Labernia CreateHBaseTable.java import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.HBaseAdmin;
public class CreateHBaseTable
{
public static void main(String[] args) throws IOException
{
HBaseConfiguration hconfig = new HBaseConfiguration(new Configuration());
HTableDescriptor htable = new HTableDescriptor("Test");
htable.addFamily( new HColumnDescriptor("id"));
htable.addFamily( new HColumnDescriptor("Name"));
System.out.println( "Connecting..." );
HBaseAdmin hbase_admin = new HBaseAdmin( hconfig );
System.out.println( "Creating Table..." );
hbase_admin.createTable( htable );
System.out.println("Done!");
}
} export HADOOP_CLASSPATH= $HADOOP_PATH:`hbase classpath` javac -cp `hbase classpath` CreateHBaseTable.java
java -cp `hbase classpath` CreateHBaseTable
... View more
05-25-2016
04:17 PM
@wsalazar
Can you please try with below imports? import org.apache.spark._
import org.apache.spark.rdd.NewHadoopRDD
import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.mapreduce.TableInputFormat
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HColumnDescriptor
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.HTable;
Also if possible please share your code and command.
... View more
05-25-2016
01:21 PM
Do you have any hive metastore running in cluster or you want to connect to hive embedded metastore?
... View more