About rreddy

rreddy · ‎10-02-2017

Exclusivity is possible in yarn using node labels. Use the property spark spark.yarn.am.nodeLabelExpression to restrict application master to a set of nodes while running spark on yarn. Add the node labels to whichever nodes you want to use for application masters (which I believe will launch driver program for spark). Enabling YARN Node Labels In your case, if temporary nodes are handful compared to static nodes, it will be worth to explore non exclusive node labels that will prevent AM from being created on temporary nodes.

rreddy · ‎10-02-2017

Are you running Spark on YARN? If so, as explained in SPARK-4253 you cannot set spark.driver.host . It will ignore this config item in yarn-cluster mode.

rreddy · ‎10-02-2017

What are you trying to compare exactly? Hive insert query performance in LLAP vs HS2 mode? It will be helpful to know if: Queue, user limits, memory settings are all the same? and if its batch insert or single row? how many llap containers are running in cluster vs total configured yarn

rreddy · ‎10-02-2017

useful for maven based builds...thanks

rreddy · ‎06-22-2017

We want to avoid non-local reads of data as much as possible for best performance. Details here: http://ercoppa.github.io/HadoopInternals/AnatomyMapReduceJob.html#maptask-launch

rreddy · ‎06-22-2017

Have you confirmed if there are containers being run on this node(and non local reads) thats causing job to be slow? If thats the case I would recommend to install only 'datanode' process first and once cluster is balanced (maybe after day) add 'nodemanager' process to run containers on the node.

rreddy · ‎06-22-2017

1) If you are HDP customer, please open support ticket to get exact answer depending on your current version on Ambari and metastore type. Features are added and deprecated in every release so its always recommended to get official response from support depending on your installation. 2) https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_system-admin-guide/content/sysadminguides_ha_chap3.html

rreddy · ‎06-22-2017

From what location are you invoking hplsql shell? If you have hplsql-site.xml in current directory, it should take precedence over default hplsql-site.xml file under lib folder. As an alternative you can try using SET command once you are inside shell to set parameters.

rreddy · ‎06-22-2017

You probably are using Hive on Tez. There is user-level explain for Hive on Tez users. Apply below setting and then run 'explain' query to see much more clearly readable tree of operations. This is also available for Hive on Spark and setting is called 'hive.spark.explain.user' set hive.explain.user=true

rreddy · ‎06-22-2017

If you are doing this on single node in cluster then yes delete the original copied data files and Namenode will take care of recreating missing data files.

rreddy · ‎06-22-2017

AFAIK there are options to discover and 'repair' corrupted files that are stored in HDFS. Most common reasons for file file corruption are associated to hdfs blocks missing or corrupted. HDFS may automatically act to fix such corrupt files periodically depending on cause like missing block or checksum mismatch etc. But in your case the file itself is still open and not considered 'complete' or 'closed' by hdfs so unless you have way to recreate entire file from source system by 'reprocessing' such files cant be 'fixed'.

rreddy · ‎03-27-2017

One option is to delete existing external table and create new table that includes new column. Since this is Hive metadata operation, your data files wont be touched. Downside is that you will have to execute alter table command to redefine partitions on new table.

rreddy · ‎03-27-2017

Can you post xml version of your falcon job?

rreddy · ‎03-26-2017

In production type cluster (with 10's-100+ nodes) with Namenode HA enabled, best practice is to have 2 Namenodes (1 active and 1 standby) and 3 Journal nodes.

rreddy · ‎09-16-2016

One option in Spark is to perform a broadcast join (aka map-side join in hadoop world). With broadcast join, you can very effectively join a large table (fact) with relatively small tables (dimensions) by avoiding sending all data of the large table over the network. You can use broadcast function to mark a dataset to be broadcast when used in a join operator. It uses spark.sql.autoBroadcastJoinThreshold setting to control the size of a table that will be broadcast to all worker nodes when performing a join.

rreddy · ‎08-23-2016

Error is due to not being able to open a hive meta store connection. I would check 1) If hive metastore is up and running, test using beeline-cli 2) If step1 is working, then spark app is not picking up hive-site.xml config file. As an alternate way, can you try to have a copy of hive-site.xml in your Oozie workflow lib folder and access it as shown below: <file>lib/hive-site.xml</file> Have you also considered using spark action in Oozie 4.2.0 that is included in HDP 2.4?

rreddy · ‎08-17-2016

Spark Standalone mode is Spark’s own built-in clustered environment. Standalone-Master is the resource manager for the Spark Standalone cluster.Standalone-Worker is the worker in the Spark Standalone cluster. To install Spark Standalone mode, you simply place a compiled version of Spark on each node on the cluster.You can launch standalone cluster either manually, by starting a master and workers by hand, or use launch scripts. In most enterprises, you already have Hadoop cluster that is running YARN and want to leverage it for resource management instead of additionally running Spark Standalone mode. If using YARN, spark applications will run its spark-master and spark-workers within containers of YARN. Irrespective of your deployment mode, Spark application will consume same resources it requires to process the data. In case of YARN you have to be aware of what other workloads will be running on cluster (like MR, Tez etc) at same time spark application is executing and size your machines accordingly.

rreddy · ‎08-15-2016

You think filtering on skewed value and splitting skewed dataset into two different dataframes i.e one with all skewed rows and other with remaining rows,(and repartition if necessary) then outer join with second dataset will help especially in performance? We can later join result datasets and get same end result

rreddy · ‎08-15-2016

Can you share full details of error message?You are probably missing some libraries. If you are reading data from hdfs and running in yarn-cluster mode your parallelism by default will be equal to number of hdfs blocks. As best practice you should avoid doing collect operation unless its small test dataset and instead use saveAsTextFile method to write result dataset to hdfs or local file.

rreddy · ‎08-10-2016

What version of spark and hdp? Can you list out all jar under SPARK_HOME directory from worker machine in cluster?

rreddy · ‎08-08-2016

In what mode are you running spark application? spark shell or yarn client etc?

rreddy · ‎08-02-2016

We want to troubleshoot Sqoop export action that is failing randomly (like once in 10 days) because mapper task is exceeding its memory limit. The job runs successfully 99% of times with same config. Interestingly if failed job is rerun without any changes it always runs successfully. What we looked at so far: Container and mapper minimum set to 3GB at cluster level and job uses this settings Rerun of failed job without any changes to data or config is successful (always) Analysis of successful job counters shows that all mappers (set to 20 in job config) process between 900K-1M records Pepperdata shows that avg memory consumption of each mapper task for successful runs is always < 1GB Sqoop action dosent use any primary key or split-by flags The number of rows processed by Sqoop is approx same for all days (< 1000 rows delta) We understand that fix is to increase mapper memory limit but: What is causing mapper in failed job to consume >3GB and when rerun it finishes successfully with <1GB If there is a way to do determine memory requirements of Sqoop mapper task(using any profiling tools) so they can avoid such random failures

rreddy · ‎07-18-2016

We explicitly listed out FQDN's of all hosts in both the clusters under [domain_realm] section of krb5.conf file. We have to update this file everytime we add node to our clusters and our clusters are currently less than <100 nodes and this solution is manageable but for large clusters this may be challenge.

rreddy · ‎07-07-2016

I'm assuming you are referring to /tmp/ directory in hdfs. You can use below command to clean it up and cron it to run every week. hadoop fs -rm -r /tmp/*

rreddy · ‎07-07-2016

We have 1 Active Directory KDC names AD.COM which is shared by all environments (active directory users obtain tickets from AD.COM) then we have separate CORP.COM kerberos KDC's (same realm name ) in each layer Dev,TST,AT,PROD. Existing krb5.conf file config(part of it) in all enviroments is : AD.COM = { kdc = ad-kdc.com admin_server = ad-kdc.com } [domain_realm] .company.com = CORP.COM company.com = CORP.COM .ad.com = AD.COM ad.com = AD.COM [capaths] AD.COM = { CORP.COM = . } Now we want to 1)add DR.CORP.COM kerberos KDC for DR cluster and 2) also set up cross realm trust with PROD cluster to be able to use distcp . What should be krb5.conf file to set up cross realm trust for DR cluster? Nodes in both DR and PROD cluster have same '.company.com' domain so we are not sure how to set up krb5.conf for cross realm trust? We wanted to try below but not sure if clients in DR cluster can access PROD name node as domain name is same for all nodes in all clusters ---- FOR DR CLUSTER [domain_realm] .company.com = DR.CORP.COM company.com = DR.CORP.COM .AD.com = AD.COM AD.com = AD.COM [capaths] AD.COM = { DR.CORP.COM = . CORP.COM = DR.CORP.COM } ---- FOR PROD CLUSTER [domain_realm] .company.com = .CORP.COM company.com = .CORP.COM .AD.com = AD.COM AD.com = AD.COM [capaths] AD.COM = { .CORP.COM = . DR.CORP.COM = .CORP.COM } Looking for some best practices or help with config above.

rreddy · ‎07-06-2016

Our concern is that we do not have time based hdfs directory structure ...like what you mentioned in example: <location type="data" path="/user/ambari-qa/data/in/${YEAR}/${MONTH}/${DAY}/${HOUR}"/><location type="stats" path="/none"/> Does falcon have ability to have ability to look at file created timestamp of file and apply retention policy??

rreddy · ‎07-06-2016

We want to implement retention policy for different hdfs directories and are evaluating Falcon to achieve this. In our case, we have random hdfs directory names without any standard naming conventions like /data/incoming/customer1/, /data/outgoing/customer2/, /data/temp/process1-n/ etc. Can we use falcon in such case to implement data retention? If yes, what are high level steps?

rreddy · ‎07-01-2016

Thanks for reply Ravi. In the second approach, will there be any performance issues with jobs in DR cluster since they need to talk to kdc-master in PROD cluster everytime for a ticket? We have .5 gb connectivity between two clusters and we will be running falcon based replication.

rreddy · ‎07-01-2016

As what user are you accessing ambari? Does user have admin access to amabri?

rreddy · ‎06-30-2016

@ScipioTheYounger Setting up falcon in kerberos cluster can be found here and hdp-falcon-replication-tutorial has sample cluster and hdfs replication code.

Online	Offline
Last Visited	‎06-06-2018 05:47 PM

Date Registered	‎02-08-2016 02:27 PM
Last Visited	‎06-06-2018 05:47 PM
Total Messages Posted	39
Total Kudos Received	28

Re: Move data gives Found duplicated storage UUID ...

Re: minimum number of journal nodes needed.

Re: KDC Cross Realm Trust Setup- krb5.conf config?...

Re: Best way to merge multi part file into single ...

Re: Kafka 0.9 new-producer on kerberized HDP 2.4

Re: dynamic spark.driver.host allocation

Re: dynamic spark.driver.host allocation

Re: Does Hive LLAP support acid transaction tables

Re: Unable to connect to kerborised hiveserver2 fr...

Re: Jobs running slow after a added a datanode

Re: Jobs running slow after a added a datanode

Re: HA for backend databases Ambari, Hive and Rang...

Re: Override hplsql-site.xml location

Re: Hive Explain plan Interpretation

Re: Move data gives Found duplicated storage UUID ...

Re: decompress snappy file if it was left open in ...

Re: how to add partition to existing table having ...

Re: Falcon job running on source instead of target

Re: minimum number of journal nodes needed.

Re: Joining Large data-set - Spark parquet

Re: HDP 2.4 oozie workflow shell based action call...

Re: Spark Standalone Cluster.

Re: DataFrames/Sets: Simple join is skewed due to ...

Re: Fetch distinct values of a column in Dataframe...

Re: Spark application fails on slaves when launchi...

Re: Spark driver memory keeps growing

How to Determine Sqoop Task Memory Requirements??

Re: KDC Cross Realm Trust Setup- krb5.conf config?...

Re: Should i need to cleaning up of tmp space in h...

KDC Cross Realm Trust Setup- krb5.conf config??

Re: Use Falcon only to implement hdfs data retenti...

Use Falcon only to implement hdfs data retention i...

Re: KDC Setup between PROD and DR cluster?

Re: Why isn't Ambari creating hash or archive for ...

Re: Falcon simple feed example in secured cluster ...