Member since
07-24-2019
46
Posts
31
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
734 | 01-30-2017 09:57 PM | |
5248 | 12-17-2016 12:11 AM | |
1691 | 07-06-2016 06:54 PM | |
1945 | 07-05-2016 05:41 PM | |
1822 | 06-16-2016 04:03 PM |
11-01-2018
09:02 PM
2 Kudos
Below are some FAQ's which helps you to quickly identify some important info for DPS-DLM deployment. Pre-req's for DPS and DLM DB version- postgres 9.3 to 9.6
OS - RHEL 7.0 and above
Ambari - 2.6.2
HDP 2.6.5
Distcp should work b/w source and target clusters.
Beacon user should be created in AD there is no choice of using custom user in this DLM 1.1 release
the onboarded service user for your application should exists in AD and needs
to be resolved(id <username> on both source and target clusters .
docker version
Required ports needs to be open b/w source and target clusters and also to access DPS UI
where to install DLM and DPS software components? DLM engine needs to be installed as m-pack on both clusters using Ambari server
DLM app is dockerized container needs to be installed on DPS host
Which URL needs to be given to register cluster in DPS UI the Ambari URL integrated with knox http://<>:8443
I'm unable to see the DLM icon in DPS UI after enabling DLM component in DPS User needs to be part of Infra-admin role Verify DLM Engine install Verify that Beacon was added as a user to the HDFS superuser group.
hdfs groups beacon
The output should display HDFS (or value of the dfs.permissions.superusergroup config) as one of the groups.
Beacon user should be part of ranger policies
https://docs.hortonworks.com/HDPDocuments/DLM1/DLM-1.2.0/installation/content/dlm_verify_the_dlm_engine_installation.html Mostly used commands for for troubleshooting On DPS host use below commands
docker ps -- check ports,containers and uptime
docker images
docker exec -it <docker-name>
docker exec -it 029ec380bb3d /bin/ls -alrt /usr/dp-app/
docker logs --follow dp-app
docker exec -it d6390b6c0c50 /bin/ls -alrt /usr/dp-app/ Required Machine config for DPS and DLM DPS runs on separate machine which will run all docker containers. <br>Master Node config is recommended for this host with atleast 64 GB of memory
if you are using external database for same host consider more memory and CPU For hive replication the in target cluster beacon is auto creating deny policy in ranger ..is this expected behavior or bug in DLM 1.1? This is to prevent any writes from happening outside of replication to the target database
the deny policy is only on the replication target database For Hive replication can we schedule job per table basis? No ,in this current DLM 1.1 release only database level is supported. Please upvote if its helpful.
... View more
08-27-2018
07:46 PM
6 Kudos
Q1.Does Hive LLAP supports stored procedures? UDF’s https://community.hortonworks.com/articles/117833/creating-custom-udf-and-adding-udf-jar-to-hive-lla.html Question on handling small files problem If ACID tables are not used then how to handle small files problem in Hive?. Is there any archival process to follow like creating HAR files? Alter Table/Partition Concatenate
Version information
In Hive release 0.8.0 RCFile added support for fast block level merging of small RCFiles using concatenate command. In Hive release 0.14.0ORC files added support fast stripe level merging of small ORC files using concatenate command. ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] CONCATENATE; If the table or partition contains many small RCFiles or ORC files, then the above command will merge them into larger files. In case of RCFile the merge happens at block level whereas for ORC files the merge happens at stripe level thereby avoiding the overhead of decompressing and decoding the data. Question on Mutations So if we need to apply a thousand mutations, this would be a thousand operations, rather than one bulk operation.
Please refer for the lock section - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_data-access/content/lock-manager.html Question on cache eviction: Can LLAP be used to read more data than can fit into memory?
Yes , it has eviction policy and stored the data in compressed format. Question on data transfer: Specific example, if a “select *” is performed on a very large table, can the application receive that
data as a “stream” or does some component (LLAP or HiveServer2, etc) need to hold the entire dataset in memory?
All the results are streamed to HDFS and the results are streamed from there . NO memory constraint Question on query result: Related, does LLAP send results back as they become available (like Hbase scanresults) or only once the query completes? –
Returns the results once SQL completes Question on compaction: We may benefit from Hive’s ACID feature to handle “deltas”. Advantages seem to be:
•it would allow the updated data to be available in queries before a compaction has taken place. “You can update the data. compaction should be transparent”
•compaction implementation already exists, no need for bespoke implementation – Hive has inbuilt compaction technique [Major and Minor] Question on spark and hive llap integration: •Can LLAP be leveraged to serve data to Spark jobs efficiently? I.e., can LLAP inform Spark on the partitioning of the data it will provide? Or is it very course, plain jdbc, interface?
LLAP Spark Context is in Tech Preview(TP). Question on cache eviction algorithm: Seems Hive Metastore does not cache much data. Which means each query for Metadata, which would include statistics, goes through the “datanucleus” ORM layer.
Is this correct?
LLAP has a metadata cache. Caching The daemon caches metadata for input files, as well as the data. The metadata and index information can be cached even for data that is not currently cached. Metadata is stored in process in Java objects; cached data is stored in the format described in the I/O section, and kept off-heap (see Resource management).
Eviction policy. The eviction policy is tuned for analytical workloads with frequent (partial) table-scans. Initially, a simple policy like LRFU is used. The policy is pluggable.
Caching granularity. Column-chunks are the unit of data in the cache. This achieves a compromise between low-overhead processing and storage efficiency. The granularity of the chunks depends on the particular file format and execution engine (Vectorized Row Batch size, ORC stripe, etc.).
A bloom filter is automatically created to provide Dynamic Runtime Filtering. Question Hive LLAP on specific Nodes In Ambari, how to specify where to run llap daemon on specific node. Running Hive LLAP on specific Nodes using YARN Node Labels
https://community.hortonworks.com/content/kbentry/170868/running-llap-on-specific-nodes-using-yarn-node-lab.html How fast I will know when LLAP query execution will fail?. If this execution mode is sethive. llap.execution.mode=only will fail
immediately before submitting to LLAP How to cancel LLAP queries which are in RUNNING State? you should check yarn and see ifllaphas enough containers allocated.
1.Yarn top
2.Yarn application -kill <appid> How to modify LLAP Log Options There is a flag in Ambari-Hive config section UI
... View more
08-27-2018
07:36 PM
4 Kudos
HIVE LLAP - a one-page architecture overview https://community.hortonworks.com/articles/149894/llap-a-one-page-architecture-overview.html Hive - Understanding concurrent sessions + queue
allocation + preemption https://community.hortonworks.com/articles/56636/hive-understanding-concurrent-sessions-queue-alloc.html Hive LLAP Dashboards https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/grafana_hive_llap_dashboards.html Hive LLAP Logs info https://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.html Monitoring LLAP metrics http://www.kartikramalingam.com/hive-llap/ Debugging Hive LLAP Query https://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.html Question
on Hive LLAP benchmarks Please share if any Hive
LLAP benchmarks? https://hortonworks.com/blog/3x-faster-interactive-query-hive-llap/ LLAP Tuning Here is an excellent article on
LLAP tuning. https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html
... View more
- Find more articles tagged with:
- Data Processing
- FAQ
- Hive
Labels:
10-12-2017
12:19 AM
1 Kudo
GetTCP - Connects over TCP to the provided endpoint(s). Received data will be written as content to the FlowFile ListenTCP- Listens for incoming TCP connections and reads data from each connection using a line separator as the message demarcator. Ref:- https://nifi.apache.org/docs.html
... View more
09-30-2017
10:18 AM
2 Kudos
Thanks to @Matt Clarke for resolving this Major issue. In a typical customer environment there is a challenge while deploying HDF Cluster & enabling LDAPS Authentication because of Username case. In Active directory userid exists as (Ex for Empid:- X1122)
but When I have imported users in Ranger by setting lowercase=true all imported users are displayed like this in lower case (x1122) . I have created all required policies for kafka and nifi .verified smoke tests for Kafka and they are PASSED.
But smoke tests for NiFi are FAILED because because NiFi respects only AD value(X1122) and there is no inbuilt intelligence todo a case conversion.
All the NiFI ranger policies has userid as (x112233).So Ranger Nifi policies are not applicable in this scenario and ranger nifi plugin authorization is not working correctly.
So,NiFi Ranger Authorization has Failed to access View NiFI UI under /flow ranger policy. NiFi does not have a option to change case sensitive of returned results, but with the ldap-provider there are two configuration options for "identity Strategy": 1. (default) USE_DN --> This strategy will use the users complete DN returned by LDAP upon successful authentication for authorization.<br>
2. USE_USERNAME --> This strategy will use the username as typed in the login screen for authorization upon successful authentication with LDAP. No matter what method of authentication is used, the value used above based on configuration is passed through and identity mapping patterns configured in NiFi and the result sent to the configured authorizer. That authorizer in your case is Ranger.
We resolve this issue by using "USE_USERNAME"
So as long as user logs in as all lowercase, it will work We also changed user search filter to:
<property name="User Search Filter">(&(sAMAccountName={0})(memberOf=CN=hwx,OU=Groups,OU=Global,OU=XX,DC=XX,DC=XX))
</property>
and proper search base needed to be:
<property name="User Search Base">OU=Users,OU=XX,DC=XX,DC=XX</property>
... View more
- Find more articles tagged with:
- Cloud & Operations
- Issue Resolution
- issue-resolution
- NiFi
- nifi-api
Labels:
09-16-2017
11:52 PM
2 Kudos
HIVE Beeline: ============ Binary mode > !connect 'jdbc:hive2://prod07.app.hwx.com:10000/;transportMode=binary'
http mode
beeline -u 'jdbc:hive2://prod07.app.hwx.com:10001/;transportMode=http;httpPath=cliservice' In HS2 HA Environment
with zookeeper out auto-discovery mode !connect jdbc:hive2://prod09.app.hwx.com:2181,prod10.app.hwx.com:2181,prod11.app.hwx.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 In Kerberos Environment - Hive Beeline command
!connect 'jdbc:hive2://prod07.app.hwx.com:10001/default;principal=hive/prod07.app.hwx.com@EXAMPLE.COM;transportMode=http;httpPath=cliservice' KNOX with Beeline
!connect jdbc:hive2://knox101.app.hwx.com:8443/default;transportMode=http;httpPath=gateway/default/hive;ssl=true
knox with webhdfs curl -iku raj_ops -X GET https://knox101.app.hwx.com:8443/gateway/default/webhdfs/v1/tmp?op=LISTSTATUS if this article helps you.please up vote it.
... View more
- Find more articles tagged with:
- How-ToTutorial
- Sandbox & Learning
- smoketests
04-27-2017
06:49 PM
1 Kudo
I have provided below the implementation steps for integrating KNOX with Loadbalancer assuming once your Loadbalancer is ready. SSL connection should terminate on Knox servers at Loadbalancer side Sticky session should be enabled. Currently the SSL connection terminates at load balancer side and internally it encrypts and loops through one of the KNOX channels.
JKS file creation
OPEN in IE browser the load balancer URL
Example:-https://hadoop-knox.dev.XXXXXX.com/
Click on lock symbol click on view certificates and Certificate path choose Root click on view certificate--details--copytofile--base 509 format --save as .pem file
choose intermediateIssuer CA click on view certificate--details--copytofile--base 509 format --save as .pem file
choose loadbalncercert and click on details--copytofile--base 509 format --save as .pem file copy these 3 files into knox edge node. I have copied to certfiles folder
/tmp/knoxhacerts/new/certfiles/lb-rootca.pem
/tmp/knoxhacerts/new/certfiles/lb-intermediate-issuer.pem
/tmp/knoxhacerts/new/certfiles/hadoop-knox-dev-lb.pem create new JKS file as below
cp /usr/hdp/current/knox-server/data/security/keystores/gateway.jks /tmp/knoxhacerts/dev-knox-test-1.jks keytool -storepasswd -keystore /tmp/knoxhacerts/dev-knox-test-1.jks enter current master secret password then change the password by using new password.
Import all the PEM encoded files to these JKS file.
keytool -import -alias rootca-lb -keystore dev-knox-test-1.jks -file /tmp/knoxhacerts/new/certfiles/lb-rootca.pem
keytool -import -alias intca-lb -keystore dev-knox-test-1.jks -file /tmp/knoxhacerts/new/certfiles/lb-intermediate-issuer.pem
keytool -import -alias dev-lb -keystore dev-knox-test-1.jks -file /tmp/knoxhacerts/new/certfiles/hadoop-knox-dev-lb.pem Ca-cert chain for ODBC: Copy the contents of all below files by opening in a notepad editor to one merge-chainfile(merge-cacertchain.crt) by
/tmp/knoxhacerts/new/certfiles/lb-rootca.pem
/tmp/knoxhacerts/new/certfiles/lb-intermediate-issuer.pem
/tmp/knoxhacerts/new/certfiles/hadoop-knox-dev-lb.pem Verification Step:
Use SSLPoke to verify connectivity. Try the Java class SSLPoke to see if your truststore contains the right certificates. This will let you connect to a SSL service, send a byte of input, and watch the output. Download SSLPoke.class (https://confluence.atlassian.com/kb/files/779355358/779355357/1/1441897666313/SSLPoke.class) compile javac SSLPoke.java Execute the class as per the below, changing the URL and port appropriately.
<JAVA_HOME>/bin/java SSLPoke jira.example.com 443 Failed Scenario: A failed connection would produce the below: 1 /usr/bin/java SSLPoke jira.example.com 443 2 sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
HAPPY Path: devenap02.dev.abc.net# java -Djavax.net.ssl.trustStore=/tmp/knoxhacerts/new/dev-knox-test-1.jks SSLPoke hadoop-knox.dev.XXXXXX.com 443 Successfully connected
Pls upvote if this article helps.
... View more
- Find more articles tagged with:
- How-ToTutorial
- Knox
- knox-gateway
- knox-namenode-ha
- Security
Labels:
04-27-2017
06:45 PM
1 Kudo
if you want to verify the Certificate contents of KNOX
Server execute below command openssl s_client -showcerts
-connect 127.0.0.1:8443 if developers want to connect to KNOX with SSL enabled
copy cert contents from above
command to knox.crt file and import to a Keystore by executing below command keytool -import -keystore
myLocalTrustStore.jks -file knox.crt Now developers use as below beeline> !connect "jdbc:hive2://hadoop-knox.dev.XXXX.com:8443/default;transportMode=http; httpPath=gateway/default/hive;ssl=true;sslTrustStore=/tmp/knoxhacerts/new/myLocalTrustStore.jks;trustStorePassword=knoxdev" Hive JDBC jdbc:hive2://{gateway-host}:{gateway-port}/; ssl=true; sslTrustStore={gateway-trust-store-path}; trustStorePassword={gateway-trust-store-password}; transportMode=http; httpPath={gateway-path}/{cluster-name}/hive If you want to list the imported certs in a JKS file
execute below command. keytool -v -list -keystore
gateway.jks command to create new truststore myNewTrustSTore.jks keytool -import -alias knox
-keystore ./myNewTrustStore.jks -file ./knox-cert.pem knox-cert.pem is the cert you
saved knox.crt certificate in pem format if you want to change SSL certificate for KNOX http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/knox_ca_signed_certificates_production.html Pls upvote if this article helps.
... View more
- Find more articles tagged with:
- Issue Resolution
- issue-resolution
- Knox
- knox-gateway
- knox-ldap
- Security
Labels:
03-13-2017
10:19 PM
@rama Host < HiveServer2 m/c FQDN>, Port: 10000 Hive Server Type: Hive Server 2, Mechanism: User Name and Password, User Name : give your name and ensure that user has access in that edge node. Paswd: XXXXX Refer: http://hortonworks.com/wp-content/uploads/2014/05/Product-Guide-HDP-2.1-v1.01.pdf if it helps don't forget to upvote
... View more
03-13-2017
09:59 PM
@suresh krish Follow the steps mentioned here. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_optional_install_a_new_mit_kdc.html Please refer to this article http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_kerberos_overview.html
... View more
02-03-2017
07:49 PM
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/ranger_rest_api_create_policy.html
... View more
01-30-2017
09:57 PM
1 Kudo
Please try to run this smoke test and provide complete command O/p if any error. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/validate_the_installation_pig.html
... View more
01-27-2017
07:17 PM
1 Kudo
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform#LanguageManualTransform-SQLStandardBasedAuthorizationDisallowsTRANSFORM Ref: https://issues.apache.org/jira/browse/HIVE-6415
... View more
01-27-2017
05:01 PM
4 Kudos
I have used above API calls and executed on my cluster it worked below is the procedure I have followed for one hdfs service and one policy. Objective: Export ranger policies from cluster1 to cluster2 variables:
<clustername>=DEVLHDP
<policy_name>=<clustername>_hadoop-1-20160615193010 To download all policies from cluster1 http://ranger1:6080/service/plugins/policies/download/<clustername>_hadoop get service from cluster1 ranger1 curl -iv -u admin:xxxx -H "Content-type:application/json" -X GET http://ranger1:6080/service/public/v2/api/service/name/<clustername>_hadoop copy the json output to ranger-service.json create service in new cluster2 ranger2 curl -iv -u admin:xxxxx -d @ranger-service.json -H "Content-Type: application/json" -X POST http://ranger2:6080/service/public/v2/api/service Getting a sample policy from cluster1 ranger1 curl -iv -u admin:xxxxx -H "Content-type:application/json" -X GET http://ranger1:6080/service/public/v2/api/service/<clustername>_hadoop/policy/<clustername>_hadoop-1-20160615193010 copy the json output to ranger_policy.json create policy in new cluster2 ranger2 curl -iv -u admin:xxxxx -d @ranger_policy.json -H "Content-Type: application/json" -X POST http://ranger2:6080/service/public/v2/api/policy Tested in HDP 2.4.2 which has Apache Ranger 0.5.2 Ref: https://issues.apache.org/jira/browse/RANGER-1214 If this helped, pls vote/accept answer.
... View more
01-26-2017
04:58 PM
Its because of the credentials or file permissions on /tmp/krbcc.**..please provide complete command with output.
... View more
12-17-2016
12:11 AM
The Issue seems like node got blacklisted and its a 1 node cluster.. I would recommend to cleanup the files. Identify the large size file in hdfs using hdfs dfs -ls -R / report.Delete it from /user/<username>(Ex;- hdfs,hive) /.Trash folder too***** Clean up the /tmp space in hdfs layer. Make sure all required services are running ..restart the same. Now monitor the storage space on that node using command "df -h" Verify a smoke test and then execute your application job.
... View more
11-11-2016
06:00 PM
Is there any HDFS Balancer process is running?...
... View more
11-04-2016
04:33 PM
when ever you initiate it will try to get min containers thats the reason it will be in hung up state.
... View more
10-31-2016
01:16 PM
Hi, Please check ambari server log file and also verify the ports are open for MYSQL 3306 on that host.
... View more
08-24-2016
04:12 PM
yarn.resourcemanager.ha.rm-ids- List of logical IDs for the RMs. e.g., "rm1,rm2" Here is the description with examples for each field value http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_hadoop-ha/content/ha-rm-config-failover.html https://hadoop.apache.org/docs/r2.5.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html. Let me know if you need more information.
... View more
08-04-2016
03:01 PM
Can you please verify the BASE URL present in repo files is accessible from all the nodes?.Cross-verify the same in ambari server configuration file.
... View more
07-06-2016
06:54 PM
The problem got fixed instead of passing <java-opts> -Dp1=v1 -Dp2=v2 </java-opts> we did in one-liner <java-opts>-Dp1=v1 -Dp2=v2 </java-opts>
... View more
07-05-2016
09:05 PM
It's a similar issue: https://issues.apache.org/jira/browse/OOZIE-1794?jql=project%20%3D%20OOZIE%20AND%20text%20~%20java-opts https://issues.apache.org/jira/browse/OOZIE-861?jql=project%20%3D%20OOZIE%20AND%20text%20~%20java-opts
... View more
07-05-2016
08:52 PM
A JVM runs with a number of system properties. You can configure system properties by using the -D option.In HDP 2.4.2 with oozie-4.2 in the workflow xml passing multiple java-opt as <java-opts> -Dp1=v1 -Dp2=v2 </java-opts> but encountered with exit code 127 container getting killed because the properties were not set correctly during container execution. But the same runs well using java program and also in previous oozie versions. Is there any limitation in OOZIE JAVA action for system property usage?.
... View more
Labels:
- Labels:
-
Apache Oozie
07-05-2016
05:41 PM
Here is the solution : hdfs dfs -put /usr/hdp/current/oozie-client/lib/*.jar /user/oozie/share/lib/*/oozie/ oozie admin -oozie http://x:11000/oozie -shareliblist oozie
oozie admin -oozie http://x:11000/oozie -sharelibupdate oozie
oozie admin -oozie http://x:11000/oozie -shareliblist oozie | grep client
... View more
06-30-2016
03:03 PM
We are trying to create a Java application that creates an Oozie client to submit and run an Oozie job. Stack - HDP 2.4 hortonworks@hdputility02:~/deployments/kmart/lib$ yarn jar feed.jar com.hooklogic.tools.OozieFTPGetAction
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/oozie/client/OozieClientException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: org.apache.oozie.client.OozieClientException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 4 more
... View more
Labels:
- Labels:
-
Apache Oozie
06-16-2016
04:03 PM
1 Kudo
Thanks for your responses.Problem is with Network using publicIp's instead of private Ip..updates source cluster private ip's in destination cluster.
... View more
06-15-2016
07:47 PM
Hello team,
Encountered errors during data migration from CDH4.2 cluster to HDP 2.4 cluster using DISTCP and below are the details.
Please let me know your thoughts.
CDH4.2 NON_HA+Non-secure
NameNode
a.a.a.a IP) castor-namenode-01
core Hadoop version 2.0.0
HDP2.4 HA + Non-secure
Active NameNode hdpmasternode07
Standby NN hdpmasternode06
http://b.b.b.b:50070/dfshealth.html#tab-overview
core Hadoop version : 2.7 Passed Test cases:
------------------ Copy empty dirs from CDH to HDP 2.4 cluster
Accessing the HDP 2.4 hdfs from CDH cluster using webhdfs protocol and vice-versa. Command used
------------- hdfs@hdpmasternode07:/$ hadoop distcp hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz hdfs://b.b.b.b:8020/tmp Logs:
------
hdfs@hdpmasternode07:/$ hadoop distcp hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz hdfs://b.b.b.b:8020/tmp
16/06/15 19:27:45 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz], targetPath=hdfs://b.b.b.b:8020/tmp, targetPathExists=true, preserveRawXattrs=false}
16/06/15 19:27:45 INFO impl.TimelineClientImpl: Timeline service address: http://hdpmasternode06:8188/ws/v1/timeline/
16/06/15 19:27:46 INFO impl.TimelineClientImpl: Timeline service address: http://hdpmasternode06:8188/ws/v1/timeline/
16/06/15 19:27:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
16/06/15 19:27:46 INFO mapreduce.JobSubmitter: number of splits:1
16/06/15 19:27:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1466011429718_0013
16/06/15 19:27:47 INFO impl.YarnClientImpl: Submitted application application_1466011429718_0013
16/06/15 19:27:47 INFO mapreduce.Job: The url to track the job: http://hdpmasternode07:8088/proxy/application_1466011429718_0013/
16/06/15 19:27:47 INFO tools.DistCp: DistCp job-id: job_1466011429718_0013
16/06/15 19:27:47 INFO mapreduce.Job: Running job: job_1466011429718_0013
16/06/15 19:27:53 INFO mapreduce.Job: Job job_1466011429718_0013 running in uber mode : false
16/06/15 19:27:53 INFO mapreduce.Job: map 0% reduce 0%
16/06/15 19:28:03 INFO mapreduce.Job: map 100% reduce 0%
16/06/15 19:31:04 INFO mapreduce.Job: Task Id : attempt_1466011429718_0013_m_000000_0, Status : FAILED
Error: java.io.IOException: File copy failed: hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz --> hdfs://b.b.b.b:8020/tmp/retail-batch.inv261b.log.gz
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:285)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hftp://a.a.a.a:50070/tmp/inv261/retail-batch.inv261b.log.gz to hdfs://b.b.b.b:8020/tmp/retail-batch.inv261b.log.gz
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
... 10 more
Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.net.SocketTimeoutException: connect timed out
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:302)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:247)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:998)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:934)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:852)
at sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2412)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1559)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:370)
at org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:135)
at org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:116)
at org.apache.hadoop.hdfs.web.ByteRangeInputStream.<init>(ByteRangeInputStream.java:101)
at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:383)
at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:388)
at org.apache.hadoop.hdfs.web.HftpFileSystem.open(HftpFileSystem.java:404)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:298)
... 16 more
... View more
Labels:
- Labels:
-
Apache Hadoop
01-08-2016
07:32 PM
How can I Provide customized service accounts in Ambari Blueprint.Any references can help us.
... View more
Labels:
- Labels:
-
Apache Ambari
10-25-2015
03:32 AM
1 Kudo
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 138, in <module>
OozieServiceCheck().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 61, in service_check
OozieServiceCheckDefault.oozie_smoke_shell_file(smoke_test_file_name, prepare_hdfs_file_name)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 123, in oozie_smoke_shell_file
logoutput=True
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 260, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin http://hdpmaster3.bigdataprod1.wh.xxxxxcorp.com:11000/oozie /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf /usr/hdp/current/hadoop-client/bin ambari-qa False' returned 1. source /usr/hdp/current/oozie-client/conf/oozie-env.sh ; /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job -oozie http://hdpmaster3.bigdataprod1.wh.xxxxcorp.com:11000/oozie -config /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
Job ID : 0000005-151024200514443-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : map-reduce-wf
App Path : hdfs://WHPROD1NN/user/ambari-qa/examples/apps/map-reduce/workflow.xml
Status : FAILED
Run : 0
User : ambari-qa
Group : -
Created : 2015-10-25 03:15 GMT
Started : 2015-10-25 03:15 GMT
Last Modified : 2015-10-25 03:15 GMT
Ended : 2015-10-25 03:15 GMT
CoordAction ID: -
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000005-151024200514443-oozie-oozi-W@:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0000005-151024200514443-oozie-oozi-W@mr-node FAILED - - EJ001 -------------
... View more
Labels: