About Arun-

Arun- · ‎11-01-2018

Below are some FAQ's which helps you to quickly identify some important info for DPS-DLM deployment. Pre-req's for DPS and DLM DB version- postgres 9.3 to 9.6 OS - RHEL 7.0 and above Ambari - 2.6.2 HDP 2.6.5 Distcp should work b/w source and target clusters. Beacon user should be created in AD there is no choice of using custom user in this DLM 1.1 release the onboarded service user for your application should exists in AD and needs to be resolved(id <username> on both source and target clusters . docker version Required ports needs to be open b/w source and target clusters and also to access DPS UI where to install DLM and DPS software components? DLM engine needs to be installed as m-pack on both clusters using Ambari server DLM app is dockerized container needs to be installed on DPS host Which URL needs to be given to register cluster in DPS UI the Ambari URL integrated with knox http://<>:8443 I'm unable to see the DLM icon in DPS UI after enabling DLM component in DPS User needs to be part of Infra-admin role Verify DLM Engine install Verify that Beacon was added as a user to the HDFS superuser group. hdfs groups beacon The output should display HDFS (or value of the dfs.permissions.superusergroup config) as one of the groups. Beacon user should be part of ranger policies https://docs.hortonworks.com/HDPDocuments/DLM1/DLM-1.2.0/installation/content/dlm_verify_the_dlm_engine_installation.html Mostly used commands for for troubleshooting On DPS host use below commands docker ps -- check ports,containers and uptime docker images docker exec -it <docker-name> docker exec -it 029ec380bb3d /bin/ls -alrt /usr/dp-app/ docker logs --follow dp-app docker exec -it d6390b6c0c50 /bin/ls -alrt /usr/dp-app/ Required Machine config for DPS and DLM DPS runs on separate machine which will run all docker containers. <br>Master Node config is recommended for this host with atleast 64 GB of memory if you are using external database for same host consider more memory and CPU For hive replication the in target cluster beacon is auto creating deny policy in ranger ..is this expected behavior or bug in DLM 1.1? This is to prevent any writes from happening outside of replication to the target database the deny policy is only on the replication target database For Hive replication can we schedule job per table basis? No ,in this current DLM 1.1 release only database level is supported. Please upvote if its helpful.

Arun- · ‎08-27-2018

Q1.Does Hive LLAP supports stored procedures? UDF’s https://community.hortonworks.com/articles/117833/creating-custom-udf-and-adding-udf-jar-to-hive-lla.html Question on handling small files problem If ACID tables are not used then how to handle small files problem in Hive?. Is there any archival process to follow like creating HAR files? Alter Table/Partition Concatenate Version information In Hive release 0.8.0 RCFile added support for fast block level merging of small RCFiles using concatenate command. In Hive release 0.14.0ORC files added support fast stripe level merging of small ORC files using concatenate command. ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] CONCATENATE; If the table or partition contains many small RCFiles or ORC files, then the above command will merge them into larger files. In case of RCFile the merge happens at block level whereas for ORC files the merge happens at stripe level thereby avoiding the overhead of decompressing and decoding the data. Question on Mutations So if we need to apply a thousand mutations, this would be a thousand operations, rather than one bulk operation. Please refer for the lock section - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_data-access/content/lock-manager.html Question on cache eviction: Can LLAP be used to read more data than can fit into memory? Yes , it has eviction policy and stored the data in compressed format. Question on data transfer: Specific example, if a “select *” is performed on a very large table, can the application receive that data as a “stream” or does some component (LLAP or HiveServer2, etc) need to hold the entire dataset in memory? All the results are streamed to HDFS and the results are streamed from there . NO memory constraint Question on query result: Related, does LLAP send results back as they become available (like Hbase scanresults) or only once the query completes? – Returns the results once SQL completes Question on compaction: We may benefit from Hive’s ACID feature to handle “deltas”. Advantages seem to be: •it would allow the updated data to be available in queries before a compaction has taken place. “You can update the data. compaction should be transparent” •compaction implementation already exists, no need for bespoke implementation – Hive has inbuilt compaction technique [Major and Minor] Question on spark and hive llap integration: •Can LLAP be leveraged to serve data to Spark jobs efficiently? I.e., can LLAP inform Spark on the partitioning of the data it will provide? Or is it very course, plain jdbc, interface? LLAP Spark Context is in Tech Preview(TP). Question on cache eviction algorithm: Seems Hive Metastore does not cache much data. Which means each query for Metadata, which would include statistics, goes through the “datanucleus” ORM layer. Is this correct? LLAP has a metadata cache. Caching The daemon caches metadata for input files, as well as the data. The metadata and index information can be cached even for data that is not currently cached. Metadata is stored in process in Java objects; cached data is stored in the format described in the I/O section, and kept off-heap (see Resource management). Eviction policy. The eviction policy is tuned for analytical workloads with frequent (partial) table-scans. Initially, a simple policy like LRFU is used. The policy is pluggable. Caching granularity. Column-chunks are the unit of data in the cache. This achieves a compromise between low-overhead processing and storage efficiency. The granularity of the chunks depends on the particular file format and execution engine (Vectorized Row Batch size, ORC stripe, etc.). A bloom filter is automatically created to provide Dynamic Runtime Filtering. Question Hive LLAP on specific Nodes In Ambari, how to specify where to run llap daemon on specific node. Running Hive LLAP on specific Nodes using YARN Node Labels https://community.hortonworks.com/content/kbentry/170868/running-llap-on-specific-nodes-using-yarn-node-lab.html How fast I will know when LLAP query execution will fail?. If this execution mode is sethive. llap.execution.mode=only will fail immediately before submitting to LLAP How to cancel LLAP queries which are in RUNNING State? you should check yarn and see ifllaphas enough containers allocated. 1.Yarn top 2.Yarn application -kill <appid> How to modify LLAP Log Options There is a flag in Ambari-Hive config section UI

Arun- · ‎08-27-2018

HIVE LLAP - a one-page architecture overview https://community.hortonworks.com/articles/149894/llap-a-one-page-architecture-overview.html Hive - Understanding concurrent sessions + queue allocation + preemption https://community.hortonworks.com/articles/56636/hive-understanding-concurrent-sessions-queue-alloc.html Hive LLAP Dashboards https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/grafana_hive_llap_dashboards.html Hive LLAP Logs info https://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.html Monitoring LLAP metrics http://www.kartikramalingam.com/hive-llap/ Debugging Hive LLAP Query https://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.html Question on Hive LLAP benchmarks Please share if any Hive LLAP benchmarks? https://hortonworks.com/blog/3x-faster-interactive-query-hive-llap/ LLAP Tuning Here is an excellent article on LLAP tuning. https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html

Arun- · ‎10-12-2017

GetTCP - Connects over TCP to the provided endpoint(s). Received data will be written as content to the FlowFile ListenTCP- Listens for incoming TCP connections and reads data from each connection using a line separator as the message demarcator. Ref:- https://nifi.apache.org/docs.html

Arun- · ‎09-30-2017

Thanks to @Matt Clarke for resolving this Major issue. In a typical customer environment there is a challenge while deploying HDF Cluster & enabling LDAPS Authentication because of Username case. In Active directory userid exists as (Ex for Empid:- X1122) but When I have imported users in Ranger by setting lowercase=true all imported users are displayed like this in lower case (x1122) . I have created all required policies for kafka and nifi .verified smoke tests for Kafka and they are PASSED. But smoke tests for NiFi are FAILED because because NiFi respects only AD value(X1122) and there is no inbuilt intelligence todo a case conversion. All the NiFI ranger policies has userid as (x112233).So Ranger Nifi policies are not applicable in this scenario and ranger nifi plugin authorization is not working correctly. So,NiFi Ranger Authorization has Failed to access View NiFI UI under /flow ranger policy. NiFi does not have a option to change case sensitive of returned results, but with the ldap-provider there are two configuration options for "identity Strategy": 1. (default) USE_DN --> This strategy will use the users complete DN returned by LDAP upon successful authentication for authorization.<br> 2. USE_USERNAME --> This strategy will use the username as typed in the login screen for authorization upon successful authentication with LDAP. No matter what method of authentication is used, the value used above based on configuration is passed through and identity mapping patterns configured in NiFi and the result sent to the configured authorizer. That authorizer in your case is Ranger. We resolve this issue by using "USE_USERNAME" So as long as user logs in as all lowercase, it will work We also changed user search filter to: <property name="User Search Filter">(&(sAMAccountName={0})(memberOf=CN=hwx,OU=Groups,OU=Global,OU=XX,DC=XX,DC=XX)) </property> and proper search base needed to be: <property name="User Search Base">OU=Users,OU=XX,DC=XX,DC=XX</property>

Arun- · ‎09-16-2017

HIVE Beeline: ============ Binary mode > !connect 'jdbc:hive2://prod07.app.hwx.com:10000/;transportMode=binary' http mode beeline -u 'jdbc:hive2://prod07.app.hwx.com:10001/;transportMode=http;httpPath=cliservice' In HS2 HA Environment with zookeeper out auto-discovery mode !connect jdbc:hive2://prod09.app.hwx.com:2181,prod10.app.hwx.com:2181,prod11.app.hwx.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 In Kerberos Environment - Hive Beeline command !connect 'jdbc:hive2://prod07.app.hwx.com:10001/default;principal=hive/prod07.app.hwx.com@EXAMPLE.COM;transportMode=http;httpPath=cliservice' KNOX with Beeline !connect jdbc:hive2://knox101.app.hwx.com:8443/default;transportMode=http;httpPath=gateway/default/hive;ssl=true knox with webhdfs curl -iku raj_ops -X GET https://knox101.app.hwx.com:8443/gateway/default/webhdfs/v1/tmp?op=LISTSTATUS if this article helps you.please up vote it.

Arun- · ‎04-27-2017

I have provided below the implementation steps for integrating KNOX with Loadbalancer assuming once your Loadbalancer is ready. SSL connection should terminate on Knox servers at Loadbalancer side Sticky session should be enabled. Currently the SSL connection terminates at load balancer side and internally it encrypts and loops through one of the KNOX channels. JKS file creation OPEN in IE browser the load balancer URL Example:-https://hadoop-knox.dev.XXXXXX.com/ Click on lock symbol click on view certificates and Certificate path choose Root click on view certificate--details--copytofile--base 509 format --save as .pem file choose intermediateIssuer CA click on view certificate--details--copytofile--base 509 format --save as .pem file choose loadbalncercert and click on details--copytofile--base 509 format --save as .pem file copy these 3 files into knox edge node. I have copied to certfiles folder /tmp/knoxhacerts/new/certfiles/lb-rootca.pem /tmp/knoxhacerts/new/certfiles/lb-intermediate-issuer.pem /tmp/knoxhacerts/new/certfiles/hadoop-knox-dev-lb.pem create new JKS file as below cp /usr/hdp/current/knox-server/data/security/keystores/gateway.jks /tmp/knoxhacerts/dev-knox-test-1.jks keytool -storepasswd -keystore /tmp/knoxhacerts/dev-knox-test-1.jks enter current master secret password then change the password by using new password. Import all the PEM encoded files to these JKS file. keytool -import -alias rootca-lb -keystore dev-knox-test-1.jks -file /tmp/knoxhacerts/new/certfiles/lb-rootca.pem keytool -import -alias intca-lb -keystore dev-knox-test-1.jks -file /tmp/knoxhacerts/new/certfiles/lb-intermediate-issuer.pem keytool -import -alias dev-lb -keystore dev-knox-test-1.jks -file /tmp/knoxhacerts/new/certfiles/hadoop-knox-dev-lb.pem Ca-cert chain for ODBC: Copy the contents of all below files by opening in a notepad editor to one merge-chainfile(merge-cacertchain.crt) by /tmp/knoxhacerts/new/certfiles/lb-rootca.pem /tmp/knoxhacerts/new/certfiles/lb-intermediate-issuer.pem /tmp/knoxhacerts/new/certfiles/hadoop-knox-dev-lb.pem Verification Step: Use SSLPoke to verify connectivity. Try the Java class SSLPoke to see if your truststore contains the right certificates. This will let you connect to a SSL service, send a byte of input, and watch the output. Download SSLPoke.class (https://confluence.atlassian.com/kb/files/779355358/779355357/1/1441897666313/SSLPoke.class) compile javac SSLPoke.java Execute the class as per the below, changing the URL and port appropriately. <JAVA_HOME>/bin/java SSLPoke jira.example.com 443 Failed Scenario: A failed connection would produce the below: 1 /usr/bin/java SSLPoke jira.example.com 443 2 sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target HAPPY Path: devenap02.dev.abc.net# java -Djavax.net.ssl.trustStore=/tmp/knoxhacerts/new/dev-knox-test-1.jks SSLPoke hadoop-knox.dev.XXXXXX.com 443 Successfully connected Pls upvote if this article helps.

Arun- · ‎04-27-2017

if you want to verify the Certificate contents of KNOX Server execute below command openssl s_client -showcerts -connect 127.0.0.1:8443 if developers want to connect to KNOX with SSL enabled copy cert contents from above command to knox.crt file and import to a Keystore by executing below command keytool -import -keystore myLocalTrustStore.jks -file knox.crt Now developers use as below beeline> !connect "jdbc:hive2://hadoop-knox.dev.XXXX.com:8443/default;transportMode=http; httpPath=gateway/default/hive;ssl=true;sslTrustStore=/tmp/knoxhacerts/new/myLocalTrustStore.jks;trustStorePassword=knoxdev" Hive JDBC jdbc:hive2://{gateway-host}:{gateway-port}/; ssl=true; sslTrustStore={gateway-trust-store-path}; trustStorePassword={gateway-trust-store-password}; transportMode=http; httpPath={gateway-path}/{cluster-name}/hive If you want to list the imported certs in a JKS file execute below command. keytool -v -list -keystore gateway.jks command to create new truststore myNewTrustSTore.jks keytool -import -alias knox -keystore ./myNewTrustStore.jks -file ./knox-cert.pem knox-cert.pem is the cert you saved knox.crt certificate in pem format if you want to change SSL certificate for KNOX http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/knox_ca_signed_certificates_production.html Pls upvote if this article helps.

Arun- · ‎03-13-2017

@rama Host < HiveServer2 m/c FQDN>, Port: 10000 Hive Server Type: Hive Server 2, Mechanism: User Name and Password, User Name : give your name and ensure that user has access in that edge node. Paswd: XXXXX Refer: http://hortonworks.com/wp-content/uploads/2014/05/Product-Guide-HDP-2.1-v1.01.pdf if it helps don't forget to upvote

Arun- · ‎03-13-2017

@suresh krish Follow the steps mentioned here. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_optional_install_a_new_mit_kdc.html Please refer to this article http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_kerberos_overview.html

Arun- · ‎02-03-2017

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/ranger_rest_api_create_policy.html

Arun- · ‎01-30-2017

Please try to run this smoke test and provide complete command O/p if any error. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/validate_the_installation_pig.html

Arun- · ‎01-27-2017

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform#LanguageManualTransform-SQLStandardBasedAuthorizationDisallowsTRANSFORM Ref: https://issues.apache.org/jira/browse/HIVE-6415

Arun- · ‎01-27-2017

I have used above API calls and executed on my cluster it worked below is the procedure I have followed for one hdfs service and one policy. Objective: Export ranger policies from cluster1 to cluster2 variables: <clustername>=DEVLHDP <policy_name>=<clustername>_hadoop-1-20160615193010 To download all policies from cluster1 http://ranger1:6080/service/plugins/policies/download/<clustername>_hadoop get service from cluster1 ranger1 curl -iv -u admin:xxxx -H "Content-type:application/json" -X GET http://ranger1:6080/service/public/v2/api/service/name/<clustername>_hadoop copy the json output to ranger-service.json create service in new cluster2 ranger2 curl -iv -u admin:xxxxx -d @ranger-service.json -H "Content-Type: application/json" -X POST http://ranger2:6080/service/public/v2/api/service Getting a sample policy from cluster1 ranger1 curl -iv -u admin:xxxxx -H "Content-type:application/json" -X GET http://ranger1:6080/service/public/v2/api/service/<clustername>_hadoop/policy/<clustername>_hadoop-1-20160615193010 copy the json output to ranger_policy.json create policy in new cluster2 ranger2 curl -iv -u admin:xxxxx -d @ranger_policy.json -H "Content-Type: application/json" -X POST http://ranger2:6080/service/public/v2/api/policy Tested in HDP 2.4.2 which has Apache Ranger 0.5.2 Ref: https://issues.apache.org/jira/browse/RANGER-1214 If this helped, pls vote/accept answer.

Arun- · ‎01-26-2017

Its because of the credentials or file permissions on /tmp/krbcc.**..please provide complete command with output.

Arun- · ‎12-17-2016

The Issue seems like node got blacklisted and its a 1 node cluster.. I would recommend to cleanup the files. Identify the large size file in hdfs using hdfs dfs -ls -R / report.Delete it from /user/<username>(Ex;- hdfs,hive) /.Trash folder too***** Clean up the /tmp space in hdfs layer. Make sure all required services are running ..restart the same. Now monitor the storage space on that node using command "df -h" Verify a smoke test and then execute your application job.

Arun- · ‎11-11-2016

Is there any HDFS Balancer process is running?...

Arun- · ‎11-04-2016

when ever you initiate it will try to get min containers thats the reason it will be in hung up state.

Arun- · ‎10-31-2016

Hi, Please check ambari server log file and also verify the ports are open for MYSQL 3306 on that host.

Arun- · ‎08-24-2016

yarn.resourcemanager.ha.rm-ids- List of logical IDs for the RMs. e.g., "rm1,rm2" Here is the description with examples for each field value http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_hadoop-ha/content/ha-rm-config-failover.html https://hadoop.apache.org/docs/r2.5.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html. Let me know if you need more information.

Arun- · ‎08-04-2016

Can you please verify the BASE URL present in repo files is accessible from all the nodes?.Cross-verify the same in ambari server configuration file.

Arun- · ‎07-06-2016

The problem got fixed instead of passing <java-opts> -Dp1=v1 -Dp2=v2 </java-opts> we did in one-liner <java-opts>-Dp1=v1 -Dp2=v2 </java-opts>

Arun- · ‎07-05-2016

It's a similar issue: https://issues.apache.org/jira/browse/OOZIE-1794?jql=project%20%3D%20OOZIE%20AND%20text%20~%20java-opts https://issues.apache.org/jira/browse/OOZIE-861?jql=project%20%3D%20OOZIE%20AND%20text%20~%20java-opts

Arun- · ‎07-05-2016

A JVM runs with a number of system properties. You can configure system properties by using the -D option.In HDP 2.4.2 with oozie-4.2 in the workflow xml passing multiple java-opt as <java-opts> -Dp1=v1 -Dp2=v2 </java-opts> but encountered with exit code 127 container getting killed because the properties were not set correctly during container execution. But the same runs well using java program and also in previous oozie versions. Is there any limitation in OOZIE JAVA action for system property usage?.

Arun- · ‎07-05-2016

Here is the solution : hdfs dfs -put /usr/hdp/current/oozie-client/lib/*.jar /user/oozie/share/lib/*/oozie/ oozie admin -oozie http://x:11000/oozie -shareliblist oozie oozie admin -oozie http://x:11000/oozie -sharelibupdate oozie oozie admin -oozie http://x:11000/oozie -shareliblist oozie | grep client

Arun- · ‎06-30-2016

We are trying to create a Java application that creates an Oozie client to submit and run an Oozie job. Stack - HDP 2.4 hortonworks@hdputility02:~/deployments/kmart/lib$ yarn jar feed.jar com.hooklogic.tools.OozieFTPGetAction Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/oozie/client/OozieClientException at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:278) at org.apache.hadoop.util.RunJar.run(RunJar.java:214) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: org.apache.oozie.client.OozieClientException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 4 more

Arun- · ‎06-16-2016

Thanks for your responses.Problem is with Network using publicIp's instead of private Ip..updates source cluster private ip's in destination cluster.

Arun- · ‎06-15-2016

Hello team, Encountered errors during data migration from CDH4.2 cluster to HDP 2.4 cluster using DISTCP and below are the details. Please let me know your thoughts. CDH4.2 NON_HA+Non-secure NameNode a.a.a.a IP) castor-namenode-01 core Hadoop version 2.0.0 HDP2.4 HA + Non-secure Active NameNode hdpmasternode07 Standby NN hdpmasternode06 http://b.b.b.b:50070/dfshealth.html#tab-overview core Hadoop version : 2.7 Passed Test cases: ------------------ Copy empty dirs from CDH to HDP 2.4 cluster Accessing the HDP 2.4 hdfs from CDH cluster using webhdfs protocol and vice-versa. Command used ------------- hdfs@hdpmasternode07:/$ hadoop distcp hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz hdfs://b.b.b.b:8020/tmp Logs: ------ hdfs@hdpmasternode07:/$ hadoop distcp hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz hdfs://b.b.b.b:8020/tmp 16/06/15 19:27:45 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz], targetPath=hdfs://b.b.b.b:8020/tmp, targetPathExists=true, preserveRawXattrs=false} 16/06/15 19:27:45 INFO impl.TimelineClientImpl: Timeline service address: http://hdpmasternode06:8188/ws/v1/timeline/ 16/06/15 19:27:46 INFO impl.TimelineClientImpl: Timeline service address: http://hdpmasternode06:8188/ws/v1/timeline/ 16/06/15 19:27:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 16/06/15 19:27:46 INFO mapreduce.JobSubmitter: number of splits:1 16/06/15 19:27:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1466011429718_0013 16/06/15 19:27:47 INFO impl.YarnClientImpl: Submitted application application_1466011429718_0013 16/06/15 19:27:47 INFO mapreduce.Job: The url to track the job: http://hdpmasternode07:8088/proxy/application_1466011429718_0013/ 16/06/15 19:27:47 INFO tools.DistCp: DistCp job-id: job_1466011429718_0013 16/06/15 19:27:47 INFO mapreduce.Job: Running job: job_1466011429718_0013 16/06/15 19:27:53 INFO mapreduce.Job: Job job_1466011429718_0013 running in uber mode : false 16/06/15 19:27:53 INFO mapreduce.Job: map 0% reduce 0% 16/06/15 19:28:03 INFO mapreduce.Job: map 100% reduce 0% 16/06/15 19:31:04 INFO mapreduce.Job: Task Id : attempt_1466011429718_0013_m_000000_0, Status : FAILED Error: java.io.IOException: File copy failed: hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz --> hdfs://b.b.b.b:8020/tmp/retail-batch.inv261b.log.gz at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:285) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz to hdfs://b.b.b.b:8020/tmp/retail-batch.inv261b.log.gz at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.net.SocketTimeoutException: connect timed out at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:302) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:247) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:998) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:934) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:852) at sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2412) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1559) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:370) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:135) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:116) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.<init>(ByteRangeInputStream.java:101) at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:388) at org.apache.hadoop.hdfs.web.HftpFileSystem.open(HftpFileSystem.java:404) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:298) ... 16 more

Arun- · ‎01-08-2016

How can I Provide customized service accounts in Ambari Blueprint.Any references can help us.

Arun- · ‎10-25-2015

Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 138, in <module> OozieServiceCheck().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 61, in service_check OozieServiceCheckDefault.oozie_smoke_shell_file(smoke_test_file_name, prepare_hdfs_file_name) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 123, in oozie_smoke_shell_file logoutput=True File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 260, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin http://hdpmaster3.bigdataprod1.wh.xxxxxcorp.com:11000/oozie /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf /usr/hdp/current/hadoop-client/bin ambari-qa False' returned 1. source /usr/hdp/current/oozie-client/conf/oozie-env.sh ; /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job -oozie http://hdpmaster3.bigdataprod1.wh.xxxxcorp.com:11000/oozie -config /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run Job ID : 0000005-151024200514443-oozie-oozi-W ------------------------------------------------------------------------------------------------------------------------------------ Workflow Name : map-reduce-wf App Path : hdfs://WHPROD1NN/user/ambari-qa/examples/apps/map-reduce/workflow.xml Status : FAILED Run : 0 User : ambari-qa Group : - Created : 2015-10-25 03:15 GMT Started : 2015-10-25 03:15 GMT Last Modified : 2015-10-25 03:15 GMT Ended : 2015-10-25 03:15 GMT CoordAction ID: - Actions ------------------------------------------------------------------------------------------------------------------------------------ ID Status Ext ID Ext Status Err Code ------------------------------------------------------------------------------------------------------------------------------------ 0000005-151024200514443-oozie-oozi-W@:start: OK - OK - ------------------------------------------------------------------------------------------------------------------------------------ 0000005-151024200514443-oozie-oozi-W@mr-node FAILED - - EJ001 -------------

Online	Offline
Last Visited	‎03-20-2023 05:12 PM

Date Registered	‎07-24-2019 09:28 AM
Last Visited	‎03-20-2023 05:12 PM
Total Messages Posted	46
Total Kudos Received	31

Re: Error Trying to get Basic Pig Syntax Running

Re: Hive Query does not run

Re: How to Set JAVA System Properties in OOZIE JAV...

Re: ClassNotFoundException: org.apache.oozie.clien...

Re: DISTCP fails from CHD4.2(Non HA+Non-secure) t...

DPS DLM FAQ's

Common LLAP questions answered

LLAP Cheat Sheet

Re: not able to understand difference between GetT...

How to handle username case conversion while Integ...

Smoke Tests for all HIVE connection modes

Integrating KNOX with LoadBalancer

KNOX Troubleshooting

Re: Microsoft HIVE ODBC Driver data source connect...

Re: kerbores not started

Re: Is there a way to export ranger policies from ...

Re: Error Trying to get Basic Pig Syntax Running

Re: Hive Transform Error

Re: Is there a way to export ranger policies from ...

Re: I want to login has mysql user and do kinit fo...

Re: Hive Query does not run

Re: BlockMissingException hive query over tez or m...

Re: Nodemanager dont start and hive client is taki...

Re: Issue in adding Hive service from ambari in HD...

Re: enabling resourceManager HA

Re: Ambari Install, Start, and Test Errors

Re: How to Set JAVA System Properties in OOZIE JAV...

Re: How to Set JAVA System Properties in OOZIE JAV...

How to Set JAVA System Properties in OOZIE JAVA AC...

Re: ClassNotFoundException: org.apache.oozie.clien...

ClassNotFoundException: org.apache.oozie.client.Oo...

Re: DISTCP fails from CHD4.2(Non HA+Non-secure) t...

DISTCP fails from CHD4.2(Non HA+Non-secure) to HD...

Customized service accounts with Ambari BluePrint

OOzie smoke tests failed in Ambari UI