Hi,   I understand that you have a spark java code, Which is taking 2 hours to process 4MB of data and you like to improve the performance of this application.   I recommend you to check the below documents, Which helps in performance tuning both in code and configuration level.   https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-1/ https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/   Thanks Jerry ... View more

Hi Manus,   I could see you have submitted the job in client mode. Could you please let us know If you are facing any exception during the execution.   spark-submit --class ey.profiler.sparkProfilerClient --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jar --deploy-mode client DataProfiling.jar    For cluster mode, it should be: --deploy-mode cluster   The result of the job will be printed as part of the driver log. In the case of cluster mode, it should be available in the Application master container logs.   Could you please share the actual output to validate further   Thanks Jerry   ... View more

Hi Dhiraj,   Assign the application to a queue named after the primary group (usually the first group) to which the user belongs.   Reference Link: https://blog.cloudera.com/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basics/#crayon-5bb5e2c83a686341712232   Thanks Jerry ... View more

Hi, Are you trying to run the sqoop command in the worker node? Can you try redeploying the sqoop client configuration and restarting Cloudera manager agent on the node. Which can help in reconfiguring the symlinks Thanks Jerry ... View more

Hi , Can you try "INVALIDATE METADATA [[db_name.]table_name]". usually we run this command if table is not created by Impala to reload the metadata. Link: https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_invalidate_metadata.html Thanks Jerry ... View more

Hi Harish, You can create a hive context and can access the hive table. Example Program: from pyspark.sql import HiveContext hive_context = HiveContext(sc) sample = hive_context.table("default.<tablename>") sample.show() Reference Link: https://stackoverflow.com/questions/36051091/query-hive-table-in-pyspark ... View more

Hi, Can you try to execute a sample spark application. Let us know the results spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --keytab <location>/<filename>.keytab --principal <principle name> /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 1 1 Thanks Jerry ... View more

Hi Dean, Gateway role is just for the client configuration, As it is not a service the state is denoted as N/A and it is expected behavior. Do you facing this issue, Even after adding the Gateway node/ Deploying client configuration. Thanks Jerry ... View more

Hi Based on the error message you have shared. ... TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module' This error corresponds to bug JIRA SPARK-19019 [1]. This bug relates to a compatibility issue between Spark and Python 3.6 Spark 1.6 requires Python 2.6+ as per the Document[2] [1] https://issues.apache.org/jira/browse/SPARK-19019 [2] https://spark.apache.org/docs/1.6.0/#downloading [3] https://www.cloudera.com/documentation/enterprise/5-14-x/topics/spark_python.html#spark_python__section_ark_lkn_25 ... View more

Hi, If it is a External table we need to delete the directory manually. ALTER TABLE table_name DROP [IF EXISTS] PARTITION <>; hadoop fs -rm -r <partition file path> Thanks Jerry ... View more

Hi Naveen, If you have limited number of ports available. You can assign port for each application. --conf "spark.driver.port=4050" —conf "spark.executor.port=51001" --conf "spark.ui.port=4005" Hope it helps Thanks Jerry ... View more

Hi , Try to check which process is using the port 9083. netstat -plant | grep 9083 If required kill and wait if it is restarting the process automatically but any script. Once confirmed restart and check HMS and HS2 Thanks Jerry ... View more

Hi Renuka, Sqoop 2 is not supported in CDH 6.x. To upgrade to CDH 6.x, you must delete the Sqoop 2 service before the upgrade Check the below link for sqoop2 metastore backup https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_cdh_upgrade_backup.html#concept_yfz_ltv_hs Hope it helps Thanks Jerry ... View more

Hi, From the shared exception it seems you are hitting bug "SQOOP-2999". A change to sqoop (SQOOP-2737) introduced a new class OracleUtils which has a dependency on org.apache.commons.lang3.StringUtils. Unfortunately the jar that fulfills this dependency is not on the classpath that Sqoop passes to the mappers. As a workaround add the jar manually sqoop import -libjars /opt/cloudera/parcels/CDH/jars/commons-lang3-3.1.jar Let us know if any questions Thanks Jerry ... View more

Hi,   If there is any changes in the Hive metadata. Please try to run msck repair table <tablename> to get it in sync   Reference Link: https://www.cloudera.com/documentation/enterprise/5-13-x/topics/cdh_ig_hive_troubleshooting.html   Thanks Jerry ... View more

Hi Vinod, 32K partition is huge to handle, We can define bucket instead of partition to avoid too many small files. Can you share the type of query you are trying on this partitions. Thanks Jerry ... View more

Hi Fenton, Thanks for the information, Join condition could create cross product which result in the multiplication of records(As shown in the screen shot). Could you please share the query plan and join condition to check further. explain <query> Thanks Jerry ... View more

Hi, We could specifies the maximum size of each Parquet data file produced by Impala INSERT statements. By specifying "set PARQUET_FILE_SIZE=<size>" Reference Link: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_parquet_file_size.html Impala partitioning and hive bucketing will also help in managing data Link: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_partitioning.html Hope it helps. Let us know if any questions. Thanks Jerry ... View more

Hi, We understand your query is failing with RPCTIMEOUT. It could be many reason which can cause timeout. 1. Could you able to check at which point query starts to fail (From Impala daemon log using query profile id) 2. Share the Query profile. Impala will create a optimised query plan only when it is completely aware about the table stats. Please make sure you have ran "compute stats db.tablename" on all the tables used in that query. Hope it helps. Let me know if you have any questions Thanks Jerry ... View more

Hi , yarn application can go to pending state is the resource are unavailable. Could you please share the below information to check further. 1. Resource manager schedular screen shot 2. yarn application console output Let us know if you have any questions. Thanks Jerry ... View more