I have a fixed width file which i am trying to load in hive. But the thing is that i am getting a '\n' character in one of the line in file which is causing record to split and thereby causing regex to fail. I am creating table using below mentioned approach in hive. create external table test.abc1_ext(a STRING,b STRING, c STRING, d STRING, e STRING, f STRING, g STRING, h STRING, i STRING, j STRING, k STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = "(.{12})(.{1})(.{50})(.{30})(.{5})(.{30})(.{4})(.{26})(.{10})(.{10})(.{8})") LOCATION '/abc/'; Column d contains '\n' causing record to split. is there a way to handle that in hive? Regards Rahul ... View more

I am trying to run a spark application which is reading data from hive tables into dataframes and joining them. When i try to run the dataframe individually in spark shell then all joins works fine and i am able to persist data in ORC format in HDFS. But when i run it as an application using spark submit i am getting below mentioned error. Missing an output location for shuffle 2 I did a research on this and found this to be related to Memory issue. I am not getting that why this error is not coming in spark shell even with the same configuration and i am able to persist everything. Command i am using to run application is mentioned below spark-submit --master yarn-client --driver-memory 10g --num-executors 3 --executor-memory 10g --executor-cores 2 --class main.scala.test.Cences --files /usr/hdp/current/spark-client/conf/hive-site.xml --jars /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar /home/talend/test_2.11-0.0.1.jar My cluster configuration is 2 Master Nodes, 3 slave nodes(4 cores and 28 GB each) and 1 Edge Node. Hive tables from which i am reading data are of around 150 MB (very less) in size which is very less as compared to the memory i am giving to spark programs. I am calling following dataframes functions i.e. saveAsTable(), write.format(), persist() in between in application. Any suggestions would really be helpful? ... View more

I have 2 dataframes in spark as mentioned below. val test = hivecontext.sql("select max(test_dt) as test_dt from abc"); test: org.apache.spark.sql.DataFrame = [test_dt: string] val test1 = hivecontext.table("testing"); where test1 has columns like id,name,age,audit_dt I want to compare these 2 dataframes and filter rows from test1 where audit_dt > test_dt. Somehow I am not able to do that. I am able to compare audit_dt with literal date using lit function but i am not able to compare it with another dataframe column. I am able to compare literal date using lit function as mentioned below val output = test1.filter(to_date(test1("audit_date")).gt(lit("2017-03-23"))) Can anyone suggest as way to compare it with column of dataframe test? Thanks Rahul ... View more

I have installed Ranger KMS on 6 node cluster. I am trying to create encryption zone now. For that i am following below mentioned link http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_admin_tools/content/hdfs-encr-appendix.html In this link in step no 5 it is mentioned that add newly created group to dfs.permissions.superusergroup property in Ambari. So i added it as hdfs,cdp in above mentioned property in Ambari and restarted HDFS. But i am not able to run hdfs dfsadmin -report command using user name "mgr" in group "cdp". I wanted to check if we can put 2 different values for same property in Ambari? Or do we need to keep only newly created group in dfs.permissions.superusergroup property? If yes then will removing hdfs have any implications? We have created few HDFS directories using hdfs user and there is data in those. Or is there is way to provide both values as superuser? ... View more