I have a 6 node HDP 2.5 cluster(1 Edge, 1 Master, 1 Secondary Master, 3 Data Nodes) running on Azure VM's. I am looking to install Kafka on on that cluster. I wanted to have Kafka Connect API, Kafka Streams API alongwith that. I went on to "add service" in Ambari and i shows Kafka 0.10 with HDP 2.5. What would the best way to install kafka in my case on HDP 2.5? How many brokers should i be installing and where to run brokers and zookeeper? Any help would be appreciated. Thanks in Advance. ... View more

yarn-running-applications.pngHi, I am facing lot of trouble in dealing with slowness/unresponsiveness of hive cli. Initially i thought it might be due to lack to resources in cluster but when i went and check running applications I went and see yarn running applications i found that there are lot of resources being left in cluster. I face the same issue when i run spark application which acquires around 50-60% of the cluster. Please note that i have not set up queuing in yarn. All my applications goes to default queue. I am not able to understand why opening hive cli gets stuck even after resource availability in cluster. Could anyone from the community help me in resolving this? Do i need to setup queuing. I am also attaching screenshot for the running applications in yarn when i try to open hive shell ... View more

I need to ingest large amount of data into one of the hive orc table using talend.(around 48 million). I am also decrypting and normalizing/denormalizing few of the columns before ingesting the data. Due to memory issues i cannot ingest data at once so i need to ingest it in chunks(by diving on basis on some columns). I tried using limit but was not sure if limit will give new rows on each run of job? Or is there any other hive analytics function which fits this scenario? I want to break the data amongs 3-4 runs of the job so that it didnot cause out of memory error? Example : I ran query based on grouping of relation and it showed results as mentioned below. hive> select count(1),relation from refinedlayer.customerpref group by relation; OK 1719076 NULL 2523 CHILD 33522 OTHER 121394 PARTNER 3282312 SELF I would like to break records of SELF relation into 5-6 runs. Can i use any analytics function on top of it? ... View more

I have a fixed width file which i am trying to load in hive. But the thing is that i am getting a '\n' character in one of the line in file which is causing record to split and thereby causing regex to fail. I am creating table using below mentioned approach in hive. create external table test.abc1_ext(a STRING,b STRING, c STRING, d STRING, e STRING, f STRING, g STRING, h STRING, i STRING, j STRING, k STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = "(.{12})(.{1})(.{50})(.{30})(.{5})(.{30})(.{4})(.{26})(.{10})(.{10})(.{8})") LOCATION '/abc/'; Column d contains '\n' causing record to split. is there a way to handle that in hive? Regards Rahul ... View more

I have a fixed width file which i am trying to load in hive. But the thing is that i am getting a '\n' character in one of the line in file which is causing record to split and thereby causing regex to fail. I am creating table using below mentioned approach in hive. create external table test.abc1_ext(a STRING,b STRING, c STRING, d STRING, e STRING, f STRING, g STRING, h STRING, i STRING, j STRING, k STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = "(.{12})(.{1})(.{50})(.{30})(.{5})(.{30})(.{4})(.{26})(.{10})(.{10})(.{8})") LOCATION '/abc/'; Column d contains '\n' causing record to split. is there a way to handle that in hive? Regards Rahul ... View more

I am trying to run a spark application which is reading data from hive tables into dataframes and joining them. When i try to run the dataframe individually in spark shell then all joins works fine and i am able to persist data in ORC format in HDFS. But when i run it as an application using spark submit i am getting below mentioned error. Missing an output location for shuffle 2 I did a research on this and found this to be related to Memory issue. I am not getting that why this error is not coming in spark shell even with the same configuration and i am able to persist everything. Command i am using to run application is mentioned below spark-submit --master yarn-client --driver-memory 10g --num-executors 3 --executor-memory 10g --executor-cores 2 --class main.scala.test.Cences --files /usr/hdp/current/spark-client/conf/hive-site.xml --jars /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar /home/talend/test_2.11-0.0.1.jar My cluster configuration is 2 Master Nodes, 3 slave nodes(4 cores and 28 GB each) and 1 Edge Node. Hive tables from which i am reading data are of around 150 MB (very less) in size which is very less as compared to the memory i am giving to spark programs. I am calling following dataframes functions i.e. saveAsTable(), write.format(), persist() in between in application. Any suggestions would really be helpful? ... View more

I have 2 dataframes in spark as mentioned below. val test = hivecontext.sql("select max(test_dt) as test_dt from abc"); test: org.apache.spark.sql.DataFrame = [test_dt: string] val test1 = hivecontext.table("testing"); where test1 has columns like id,name,age,audit_dt I want to compare these 2 dataframes and filter rows from test1 where audit_dt > test_dt. Somehow I am not able to do that. I am able to compare audit_dt with literal date using lit function but i am not able to compare it with another dataframe column. I am able to compare literal date using lit function as mentioned below val output = test1.filter(to_date(test1("audit_date")).gt(lit("2017-03-23"))) Can anyone suggest as way to compare it with column of dataframe test? Thanks Rahul ... View more

I am trying to covert string column in dataframe to date/time. I am loading dataframe from hive tables and i have tried below mentioned function in converting string to date/time. But it is not giving me the correct output as it is converting all values to null. (unix_timestamp($"BIRTHDT","MM-dd-yyyy").cast("date")) && (to_date(($"BIRTHDT","MM-dd-yyyy").cast("date")) Values in my birthdt columns are as follows 20061202 20061203 20061205 20061206 20061208 Am i missing something? Thanks Rahul ... View more

I have created a hivecontext in spark and i am reading hive ORC tables from hivecontext into spark dataframes. I have saved that dataframe into temp table. I am looking for how to specify left outer join when running sql queries on that temporary table? Any help would be appreciated. Code that i am running is mentioned below. import org.apache.spark.sql._ val hivecontext = new org.apache.spark.sql.hive.HiveContext(sc) val a = hivecontext.table("customer.a_orc") val b = hivecontext.table("customer.b_orc") a.registerTempTable("a") b.registerTempTable("b") val output = hivecontext.sql("select a.*,b.* from a,b where a left outer join b on (a.id=b.id)) The output dataframe shown above is giving error. Is there a way to specify left outer join like this or do i have to create separate dataframes? ... View more

I have installed Ranger KMS on 6 node cluster. I am trying to create encryption zone now. For that i am following below mentioned link http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_admin_tools/content/hdfs-encr-appendix.html In this link in step no 5 it is mentioned that add newly created group to dfs.permissions.superusergroup property in Ambari. So i added it as hdfs,cdp in above mentioned property in Ambari and restarted HDFS. But i am not able to run hdfs dfsadmin -report command using user name "mgr" in group "cdp". I wanted to check if we can put 2 different values for same property in Ambari? Or do we need to keep only newly created group in dfs.permissions.superusergroup property? If yes then will removing hdfs have any implications? We have created few HDFS directories using hdfs user and there is data in those. Or is there is way to provide both values as superuser? ... View more

I have installed Ranger KMS on 6 node cluster. I am trying to create encryption zone now. For that i am following below mentioned link http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_admin_tools/content/hdfs-encr-appendix.html In this link in step no 5 it is mentioned that add newly created group to dfs.permissions.superusergroup property in Ambari. So i added it as hdfs,cdp in above mentioned property in Ambari and restarted HDFS. But i am not able to run hdfs dfsadmin -report command using user name "mgr" in group "cdp". I wanted to check if we can put 2 different values for same property in Ambari? Or do we need to keep only newly created group in dfs.permissions.superusergroup property? If yes then will removing hdfs have any implications? We have created few HDFS directories using hdfs user and there is data in those. Or is there is way to provide both values as superuser? ... View more

I have installed Ranger KMS on 6 node cluster. I am trying to create encryption zone now. For that i am following below mentioned link http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_admin_tools/content/hdfs-encr-appendix.html In this link in step no 5 it is mentioned that add newly created group to dfs.permissions.superusergroup property in Ambari. So i added it as hdfs,cdp in above mentioned property in Ambari and restarted HDFS. But i am not able to run hdfs dfsadmin -report command using user name "mgr" in group "cdp". I wanted to check if we can put 2 different values for same property in Ambari? Or do we need to keep only newly created group in dfs.permissions.superusergroup property? If yes then will removing hdfs have any implications? We have created few HDFS directories using hdfs user and there is data in those. Or is there is way to provide both values as superuser? ... View more

Are there any steps to install kudu on Quickstart cloudera VM and link it to Impala to create tables? I have installed Kudu using yum on cloudera CDH 5.8 quickstart and able to create tables using Impala. But when I query that table I get below mentioned error as "kudu features are diabled by startup flag --disable-kudu"   Please suggest? ... View more