Hi,   I understand that you have a spark java code, Which is taking 2 hours to process 4MB of data and you like to improve the performance of this application.   I recommend you to check the below documents, Which helps in performance tuning both in code and configuration level.   https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-1/ https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/   Thanks Jerry ... View more

Hi Dhiraj,   Assign the application to a queue named after the primary group (usually the first group) to which the user belongs.   Reference Link: https://blog.cloudera.com/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basics/#crayon-5bb5e2c83a686341712232   Thanks Jerry ... View more

Hi, Are you trying to run the sqoop command in the worker node? Can you try redeploying the sqoop client configuration and restarting Cloudera manager agent on the node. Which can help in reconfiguring the symlinks Thanks Jerry ... View more

Hi , Can you try "INVALIDATE METADATA [[db_name.]table_name]". usually we run this command if table is not created by Impala to reload the metadata. Link: https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_invalidate_metadata.html Thanks Jerry ... View more

Hi Harish, You can create a hive context and can access the hive table. Example Program: from pyspark.sql import HiveContext hive_context = HiveContext(sc) sample = hive_context.table("default.<tablename>") sample.show() Reference Link: https://stackoverflow.com/questions/36051091/query-hive-table-in-pyspark ... View more

Hi, Can you try to execute a sample spark application. Let us know the results spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --keytab <location>/<filename>.keytab --principal <principle name> /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 1 1 Thanks Jerry ... View more

Hi Dean, Gateway role is just for the client configuration, As it is not a service the state is denoted as N/A and it is expected behavior. Do you facing this issue, Even after adding the Gateway node/ Deploying client configuration. Thanks Jerry ... View more

Hi Based on the error message you have shared. ... TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module' This error corresponds to bug JIRA SPARK-19019 [1]. This bug relates to a compatibility issue between Spark and Python 3.6 Spark 1.6 requires Python 2.6+ as per the Document[2] [1] https://issues.apache.org/jira/browse/SPARK-19019 [2] https://spark.apache.org/docs/1.6.0/#downloading [3] https://www.cloudera.com/documentation/enterprise/5-14-x/topics/spark_python.html#spark_python__section_ark_lkn_25 ... View more

Hi, If it is a External table we need to delete the directory manually. ALTER TABLE table_name DROP [IF EXISTS] PARTITION <>; hadoop fs -rm -r <partition file path> Thanks Jerry ... View more

Hi Naveen, If you have limited number of ports available. You can assign port for each application. --conf "spark.driver.port=4050" —conf "spark.executor.port=51001" --conf "spark.ui.port=4005" Hope it helps Thanks Jerry ... View more

Hi , Try to check which process is using the port 9083. netstat -plant | grep 9083 If required kill and wait if it is restarting the process automatically but any script. Once confirmed restart and check HMS and HS2 Thanks Jerry ... View more

Hi Renuka, Sqoop 2 is not supported in CDH 6.x. To upgrade to CDH 6.x, you must delete the Sqoop 2 service before the upgrade Check the below link for sqoop2 metastore backup https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_cdh_upgrade_backup.html#concept_yfz_ltv_hs Hope it helps Thanks Jerry ... View more

Hi, From the shared exception it seems you are hitting bug "SQOOP-2999". A change to sqoop (SQOOP-2737) introduced a new class OracleUtils which has a dependency on org.apache.commons.lang3.StringUtils. Unfortunately the jar that fulfills this dependency is not on the classpath that Sqoop passes to the mappers. As a workaround add the jar manually sqoop import -libjars /opt/cloudera/parcels/CDH/jars/commons-lang3-3.1.jar Let us know if any questions Thanks Jerry ... View more

Hi,   If there is any changes in the Hive metadata. Please try to run msck repair table <tablename> to get it in sync   Reference Link: https://www.cloudera.com/documentation/enterprise/5-13-x/topics/cdh_ig_hive_troubleshooting.html   Thanks Jerry ... View more

Hi Vinod, 32K partition is huge to handle, We can define bucket instead of partition to avoid too many small files. Can you share the type of query you are trying on this partitions. Thanks Jerry ... View more

Hi Fenton, Thanks for the information, Join condition could create cross product which result in the multiplication of records(As shown in the screen shot). Could you please share the query plan and join condition to check further. explain <query> Thanks Jerry ... View more

Hi, We could specifies the maximum size of each Parquet data file produced by Impala INSERT statements. By specifying "set PARQUET_FILE_SIZE=<size>" Reference Link: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_parquet_file_size.html Impala partitioning and hive bucketing will also help in managing data Link: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_partitioning.html Hope it helps. Let us know if any questions. Thanks Jerry ... View more

Hi, We understand your query is failing with RPCTIMEOUT. It could be many reason which can cause timeout. 1. Could you able to check at which point query starts to fail (From Impala daemon log using query profile id) 2. Share the Query profile. Impala will create a optimised query plan only when it is completely aware about the table stats. Please make sure you have ran "compute stats db.tablename" on all the tables used in that query. Hope it helps. Let me know if you have any questions Thanks Jerry ... View more

Hi , yarn application can go to pending state is the resource are unavailable. Could you please share the below information to check further. 1. Resource manager schedular screen shot 2. yarn application console output Let us know if you have any questions. Thanks Jerry ... View more

Hi,   yarn.scheduler.maximum-allocation-mb is specified as 20 GB which means the largest amount of physical memory, that can be requested for a container and yarn.scheduler.minimum-allocation-mb will be the least amount of physical memory, that can be requested for a container.   When we submit a MR job requested container memory will be assigned “mapreduce.map.memory.mb” which is by default 1 GB. If it is not specified then we will be given container of 1GB.(Same for reducer)   This can be verified in the yarn logs -:   mapreduce.map.memory.mb - requested container memory 1GB INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>   mapreduce.map.java.opts - Which is 80% of container memory by default org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m     1 GB is the default and it is quite low. I recommend reading the below link. It provides a good understanding of YARN and MR memory setting, how they relate, and how to set some baseline settings based on the cluster node size (disk, memory, and cores).   https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html     Hope it helps. Let us know if you have any questions   Thanks Jerry ... View more