Step -1 : read data from cassandra STPE-2 : read data from aws s3. step -3 : cache() data from step -1 and step -2 . STEP-4 : find difference between step 1 and step 2 and cache data and run count() on difference data. STEP- 5: write difference data into aws s3 bucket . STEP -6 : write step -1 data to step -2 aws s3 bucket . ( This is basically overwrite step) STEP-7 : write difference data ( STEP-5) into database . ( this step is getting empty data though same data in STPE-4 is available in memory but after STEP- 6 same data is empty .) ... View more

I am reading data from cassandra table ( table size 7 GB ) and writing into aws s3 in parquet file . Job is taking 20 min and it writes 3 GB data in aws s3 in parquet format, below are configurations: total executors  in the cluster -  28 vcores total memory in the cluster - 52 GB number of nodes in the cluster - 3  per node capacity - 8 vcores and 15 GB ram executors per core - 2 memory per executor - 2G driver cores - 4 driver memory - 4G   spark cassandra read : data = spark_session.read \ .format('org.apache.spark.sql.cassandra') \ .options(table=table, keyspace=keyspace) \ .load()   data write into aws s3: data .write.parquet(aws_path,mode="overwrite")     ... View more

I have 4 python scripts and one configuration file of .txt . out of 4 python files , one file has entry point for spark application and also importing functions from other python files . But configuration file is imported in some other python file that is not entry point for spark application . I want to write spark submit command in pyspark , but I am not sure how to provide multiple files along configuration file with spark submit command when configuration file is not python file but text file or ini file. for demonstration: 4 python files : file1.py , file2.py , file3.py . file4.py 1 configuration file : conf.txt file1.py : this file has spark session and calling to all other python files . file3.py : this python file is reading conf.txt . I want to provide all these files with spark submit but not sure about command . One solution I have identified is : spark - submit -- master local -- driver - memory 2g -- executor - memory 2g -- py - files s3_path\file2 . py , s3_path\file3 . py , s3_path\file4 . py s3_path\file1 . py but with above spark submit I am not sure how to pass conf.txt . ... View more

I am getting hive metastore connection , I am using oracle for metastore storage . below is error : 2019-06-19 08:23:11,473 INFO [pool-5-thread-200]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(340)) - ugi=ambari-qa-Servername ip=/<IP-ADDRESS> cmd=get_all_databases 2019-06-19 08:23:11,476 ERROR [pool-5-thread-200]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(155)) - Retrying HMSHandler after 2000 ms (attempt 2 of 10) with error: javax.jdo.JDOException: Exception thrown when executing query We have oracle as metastorage , and this issue is observed once or twice in month . ... View more

I am using Keyprefixregionsplitpolicy for the hbase table , and i have too many regions getting created even with data size in Kb though . As per hortonworks documentation it mentions " KeyPrefixRegionSplitPolicy is a curious addition to the HBase arsenal. You can configure the length of the prefix for your row keys for grouping them, and this split policy ensures that the regions are not split in the middle of a group of rows having the same prefix. If you have set prefixes for your keys, then you can use this split policy to ensure that rows having the same rowkey prefix always end up in the same region. " but I have doubt that does same rowkey prefix lands into one region and other types of prefix rows lands in different region . I mean does it maps one to one mapping on basis of prefix rows like for every prefix rows it maps to region by region , e.g : prefix1_rows1 ---region1 prefix2_rows2 --- region2 and so on reference : https://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/ Kind Regards Anurag Mishra ... View more

In ambari hbase avg load is high , it is 360 regions per regionserver .....what are the parameters in general I have to tune related to hbase so that hbase avg load could be less than 250 regions per regionserver. ... View more

I have more than 350 regions per region server and as result it has to created one alert in ambari . There is way to increase regionsize and reduce number of regions but I am not sure about performance after the increment of size of region. What are optimal ways to reduce number of regions per regionserver. ... View more

I am getting below error in hiveserver2 : java.lang.UnsupportedOperationException: getDelegationToken() can be called only in thrift (non local) mode . what does thrift (non local ) mode mean here ? Hiverserversdoas is true in the cluster. Regards Anurag Mishra ... View more

I have first reduced table versions from 20 to 3 , as number of regions allocated to the table were too high and i suspect as versions configured were 20 hence this much of data got stored in hbase , also table is frequently getting updated . my approach to reduce regions is to alter version from 20 to 3 for all the column families for the table and then run major compaction for the table from hbase cli . But major compaction is taking so much of time what is the reason for this ? Kind Regards Anurag Mishra ... View more

i do not see spark service when i click on Actions and Add to service . is this case spark installation from ambari in hdp 2.2 and ambari 1.7 is not supported ...or there is other way around to do the same ? Regards Anurag ... View more

I have added few kafka servers in the cloudera cluster but now while pushing messages I am getting below errors : WARN clients.NetworkClient: Error while fetching metadata with correlation id 5493 : {l=LEADER_NOT_AVAILABLE} I have checked Advertised Host advertised.host.name and it is set as default kafka group and kafka broker group1. Thanks ... View more

kakfa leader not available I have added 1 more broker earlier kafka has 3 brokers , producer is throwing an error ; WARN clients.NetworkClient: Error while fetching metadata with correlation id 5493 : {amoll=LEADER_NOT_AVAILABLE}   I have checked  advertised.host.name and I see kafka broker default group and kafka broker1 group there . Kind Regards Amol ... View more

I want to setup kerberos with AD/LDAP . I have some queries , I have enabled kerberos MIT KDC . Enabling kerberos with MIT kdc requires to install kdc server , util , libs etc . if i want to enable kerberos with AD what is the procedure ? I am listing below points as per my understanding . 1. need ldap url . 2. need kdc hosts details ( kerberos server is installed ) is it imp to keep AD server and kerberos server seperate. If I want kerberos server on different host , AD on different server how is the procedure to integrate kerberos with AD/ldap . while enabling kerberos from ambari I always keep kdc hosts and AD on same server . Kind Regards Anurag ... View more

how does sentry differ from ranger ? what are the things We can not achieve with using sentry and that is achievable from ranger and vice-versa. ... View more

I am getting exceeding 1500 connections error , I see zookeeper maxclient value is set to 1500 . I want to understand what are cons if we keep value zero that implies any number of connections to zookeeper . Kind Regards Anurag ... View more

I want to load data into dynamically partitioned table in hive using pyspark , table is already created in hive only data load has to be done with pyspark . I am using below code to do above requirement but need more suggestions : spark = SparkSession \ .builder \ .appName("Python Spark SQL Hive integration example") \ .config("spark.sql.warehouse.dir", warehouse_location) \ .enableHiveSupport() \ .getOrCreate() # spark is an existing SparkSession spark.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src") 1. INPATH path should be dynamic --- how to pass in spark.sql Kind Regards Anurag ... View more

I have huge data and that has been migrated from old cluster to new cluster with distcp what are the mechanisms available to verify the data if there is any data loss or corruption happened . ... View more

i am submitting spark2 sample job with spark-submit command but it is throwing an errror that there is permission issue over /anaconda3/bin/python though permission is there . The issue is happening when job is submitted in cluster mode or client mode but when job is being submitted in local mode there is no issue . Please refer below points : 1. We are able to submit job from spark submit job using anaconda (python3) in local mode. 2. We are not able to submit job in either yarn-client or yarn-cluster mode. 3. We have python3 installed in only edge node from there We are trying to submit job. 4. is it necessary to install python3 in all other nodes , or with some configurations changes We can make this available . spark-submit --verbose /usr/hdp/current/spark2-client/examples/src/main/python/pi.py ---> This succeeds spark-submit --master yarn --deploy-mode client --verbose /usr/hdp/current/spark2-client/examples/src/main/python/pi.py ----> fails spark-submit --master yarn --deploy-mode cluster --verbose /usr/hdp/current/spark2-client/examples/src/main/python/pi.py ----> fails Kind Regards ... View more

i am submitting spark2 sample job with spark-submit command but it is throwing an errror that there is permission issue over /anaconda3/bin/python though permission is there . The issue is happening when job is submitted in cluster mode or client mode but when job is being submitted in local mode there is no issue . Please refer below points : 1. We are able to submit job from spark submit job using anaconda (python3) in local mode. 2. We are not able to submit job in either yarn-client or yarn-cluster mode. 3. We have python3 installed in only edge node from there We are trying to submit job. 4. is it necessary to install python3 in all other nodes , or with some configurations changes We can make this available . spark-submit --verbose /usr/hdp/current/spark2-client/examples/src/main/python/pi.py ---> This succeeds spark-submit --master yarn --deploy-mode client --verbose /usr/hdp/current/spark2-client/examples/src/main/python/pi.py ----> fails spark-submit --master yarn --deploy-mode cluster --verbose /usr/hdp/current/spark2-client/examples/src/main/python/pi.py ----> fails Kind Regards ... View more

I am trying to fetch results data stored in hive orc table , but when I run the query it gives results fast but When I run the with limit clause it is taking too much of time . select * from tablename where year="2019" and month="jan-mar" ; - Runs perfectly within 30-40secs select * from tablename limit 10; - Gets stuck ... View more

I want to migrate data from source cluster to destination cluster , and i want to create script for that , but when namenode switch happens script stops working as active namenode address is hard coded , I want to use nameuris but seems to be not working when running the job in different cluster . how to use nameuris to copy data using distcp in HA enabled cluster . Thanks Anurag ... View more

I am going through cluster resource management with help of yarn capacity scheduler , I read this post in the hortonworks docs but I have few doubts : " have uploaded the image in which it is mentioned that there is 3 nodes cluster with long running applications of 20 GB containers. The LLAP daemons require 90 GB of cluster resources, but preemption does not occur because the available queues are under capacity with lower relative resource usage. With only 80 GB available on any of the nodes, LLAP must wait for the long-running applications to finish before it can access cluster resources." My query is why LLAP application can not submit on all three nodes taking up 30 GB from each node or any combinations which satisfy its resource need . ... View more

I have newly built cluster , but I am not able to start oozie server please find below stack trace : 2018-11-22 06:05:04,444 - Unexpected Oozie WAR preparation output setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-server/conf} setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-server/oozie-server} setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie} setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 setting JRE_HOME=${JAVA_HOME} setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m" setting OOZIE_LOG=/var/log/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting OOZIE_DATA=/hadoop/oozie/data setting OOZIE_HTTP_PORT=11000 setting OOZIE_ADMIN_PORT=11001 setting JAVA_LIBRARY_PATH=/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64 setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 " setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-server/conf} setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-server/oozie-server} setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie} setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 setting JRE_HOME=${JAVA_HOME} setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m" setting OOZIE_LOG=/var/log/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting OOZIE_DATA=/hadoop/oozie/data setting OOZIE_HTTP_PORT=11000 setting OOZIE_ADMIN_PORT=11001 setting JAVA_LIBRARY_PATH=/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64 setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 " INFO: Adding extension: /usr/hdp/current/oozie-server/libext/falcon-oozie-el-extension-0.10.0.2.6.5.0-292.jar INFO: Adding extension: /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar INFO: Oozie webconsole disabled, ExtJS library not specified File/Dir does no exist: /usr/hdp/current/oozie-server/oozie-server/conf/ssl/server.xml Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 150, in <module> OozieServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 978, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 76, in start self.configure(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 70, in configure oozie(is_server=True, upgrade_type=upgrade_type) File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py", line 199, in oozie oozie_server_specific(upgrade_type) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py", line 314, in oozie_server_specific prepare_war(params) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/oozie_prepare_war.py", line 87, in prepare_war raise Fail(message) resource_management.core.exceptions.Fail: Unexpected Oozie WAR preparation output setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-server/conf} setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-server/oozie-server} setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie} setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 setting JRE_HOME=${JAVA_HOME} setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m" setting OOZIE_LOG=/var/log/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting OOZIE_DATA=/hadoop/oozie/data setting OOZIE_HTTP_PORT=11000 setting OOZIE_ADMIN_PORT=11001 setting JAVA_LIBRARY_PATH=/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64 setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 " setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-server/conf} setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-server/oozie-server} setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie} setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 setting JRE_HOME=${JAVA_HOME} setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m" setting OOZIE_LOG=/var/log/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting OOZIE_DATA=/hadoop/oozie/data setting OOZIE_HTTP_PORT=11000 setting OOZIE_ADMIN_PORT=11001 setting JAVA_LIBRARY_PATH=/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64 setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 " INFO: Adding extension: /usr/hdp/current/oozie-server/libext/falcon-oozie-el-extension-0.10.0.2.6.5.0-292.jar INFO: Adding extension: /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar INFO: Oozie webconsole disabled, ExtJS library not specified ... View more

what are the limitations of hive in its latest version , as Hive has also added ACID support . What are limitations there where hive is not suitable . ... View more

I want to add JohnSnowLabs:spark-nlp:1.7.3 so that i could use this package with spark shell as well with pyspark . I don't have internet connectivity , hence package should be available offline , what are they ways available . spark-shell --packages JohnSnowLabs:spark-nlp:1.7.3 pyspark --packages JohnSnowLabs:spark-nlp:1.7.3 spark-submit --packages JohnSnowLabs:spark-nlp:1.7.3 ... View more

I am getting below error when I am running the query : at sqlline.SqlLine.main(SqlLine.java:292) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794) at org.apache.phoenix.schema.stats.StatisticsUtil.readStatistics(StatisticsUtil.java:160) at org.apache.phoenix.query.TableStatsCache$StatsLoader.load(TableStatsCache.java:92) at org.apache.phoenix.query.TableStatsCache$StatsLoader.load(TableStatsCache.java:83) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 21 more Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208) at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211) at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1259) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ... 33 more Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) ... View more

I am getting below error when I am running the query : at sqlline.SqlLine.main(SqlLine.java:292) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794) at org.apache.phoenix.schema.stats.StatisticsUtil.readStatistics(StatisticsUtil.java:160) at org.apache.phoenix.query.TableStatsCache$StatsLoader.load(TableStatsCache.java:92) at org.apache.phoenix.query.TableStatsCache$StatsLoader.load(TableStatsCache.java:83) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 21 more Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208) at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211) at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1259) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ... 33 more Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) ... View more

I have HDP 2.6.5 installed in the cluster but when i try to run teradata import sqoop job it is failing with below error : 18/10/30 15:16:15 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 18/10/30 15:16:15 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 18/10/30 15:16:15 ERROR sqoop.ConnFactory: Sqoop could not found specified connection manager class o.apache.sqoop.teradata.TeradataConnManager. Please check that you've specified the class correctly. 18/10/30 15:16:15 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: java.lang.ClassNotFoundException: o.apache.sqoop.teradata.TeradataConnManager at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:166) at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:266) at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:51) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.main(Sqoop.java:243) Caused by: java.lang.ClassNotFoundException: o.apache.sqoop.teradata.TeradataConnManager at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.jar However teratajars are available under sqoop lib : tdgssconfig.jar teradata-connector-1.5.4-hadoop2.jar terajdbc4.jar hortonworks-teradata-connector-1.5.4.2.6.5.0-292.jar jar tvf hortonworks-teradata-connector-1.5.4.2.6.5.0-292.jar | grep org.apache.sqoop.teradata.TeradataConnManager org/apache/sqoop/teradata/TeradataConnManager.class ... View more

I want to run distcp job copying huge data from source cluster to destination cluster , how can i increase the performance or speed of the distcp job ? ... View more