Member since
04-03-2019
962
Posts
1743
Kudos Received
146
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 17735 | 03-08-2019 06:33 PM | |
| 7164 | 02-15-2019 08:47 PM |
12-22-2017
04:55 AM
i did same thing on HDP2.5 but still showing below error 534-5.7.14 IvciS8yws8APKDFHijkrFoJ92wCiJPAO_hwu5n84JrRGIqSyumJ05MoXUq94ogjYKFwfS5
534-5.7.14 iP5EQUiY-JheLLz82D33Qp2yF8wuQkoQ6BAwjeCpThA9GoTte7WBFbZnRByREw7MnpQ3Am
534-5.7.14 qRMD4_bLAgQqiJIzCgxU1ppqCt9qc> Please log in via your web browser and
534-5.7.14 then try again.
534-5.7.14 Learn more at
534 5.7.14 https://support.google.com/mail/answer/78754 e8sm22695829pga.72 - gsmtp
at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:892)
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:814)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:728)
at javax.mail.Service.connect(Service.java:386)
at javax.mail.Service.connect(Service.java:245)
at javax.mail.Service.connect(Service.java:194)
at javax.mail.Transport.send0(Transport.java:253)
at javax.mail.Transport.send(Transport.java:124)
at org.apache.ambari.server.notifications.dispatchers.EmailDispatcher.dispatch(EmailDispatcher.java:160)
at org.apache.ambari.server.notifications.DispatchRunnable.run(DispatchRunnable.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
... View more
05-03-2017
04:21 PM
Very good article Rahul. Quick question: Does the table have to be partitioned? I'm trying to replicate a non-partitioned table with UI and I'm getting an exception. default/FalconWebException:FalconException:java.net.URISyntaxException:Partition Details are missing. How can I replicate this table using the UI?
... View more
08-07-2017
08:25 PM
Hi, Thanks for this guide i followed exacty how you explained but i still getting this error Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, URL: http://loadbalancer_FQDN:11000/oozie/versions?user.name=oozie, status: 403, message: Forbidden Any idea Thanks
... View more
05-22-2016
11:08 AM
15 Kudos
I have written ambari-admin utility to simplify efforts required to find and trigger API curl call. I'm planning to add many more features than ambari-shell. Currently below features are supported. Demo on multinode cluster: 1. Clone our github repo to your local machine or any of the node in your cluster.
[root@sme-ambari-server ~]# git clone https://github.com/crazyadmins/useful-scripts.git
Initialized empty Git repository in /root/useful-scripts/.git/
remote: Counting objects: 106, done.
remote: Total 106 (delta 0), reused 0 (delta 0), pack-reused 106
Receiving objects: 100% (106/106), 16.89 KiB, done.
Resolving deltas: 100% (37/37), done.
2. Goto useful-scripts/ambari/
[root@sme-ambari-server ~]# cd useful-scripts/ambari/
3. Edit ambari.props and modify value of below parameters as per your cluster environment
[root@sme-ambari-server ambari]# cat ambari.props
CLUSTER_NAME=sme
AMBARI_ADMIN_USER=admin
AMBARI_ADMIN_PASSWORD=admin
AMBARI_HOST=sme-ambari-server.hwxblr.com
KDC_HOST=sme-ambari-server.hwxblr.com
REALM=HWX.COM
KERBEROS_CLIENTS=sme-ambari-server.hwxblr.com,kknew1.hwxblr.com,kknew2.hwxblr.com,kknew3.hwxblr.com
##### Notes #####
#1. KERBEROS_CLIENTS - Comma separated list of Kerberos clients in case of multinode cluster
#2. Admin princial is admin/admin and password is hadoop
Note - You can ignore Kerberos related parameters for now, in future this script will have functionality to setup Kerberos. 4. To print usage you can run ambari-admin.sh without any argument
Please refer first screenshot provided at the beginning of this article. 5. Stop All services
[root@sme-ambari-server ambari]# ./ambari-admin.sh stopall
HTTP/1.1 202 Accepted
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=14oisfe8i5bclm8tdk3npm390;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 152
Server: Jetty(8.1.17.v20150415)
{
"href" : "http://sme-ambari-server.hwxblr.com:8080/api/v1/clusters/sme/requests/61",
"Requests" : {
"id" : 61,
"status" : "Accepted"
}
6. Start all the services
[root@sme-ambari-server ambari]# ./ambari-admin.sh startall
HTTP/1.1 202 Accepted
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=1lo2x6u1r5xq319suwh8xiiquw;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 152
Server: Jetty(8.1.17.v20150415)
{
"href" : "http://sme-ambari-server.hwxblr.com:8080/api/v1/clusters/sme/requests/62",
"Requests" : {
"id" : 62,
"status" : "Accepted"
}
7. Get list of all services installed in your cluster, script also shows list of installed components host_wise
[root@sme-ambari-server ambari]# ./ambari-admin.sh listall
Below is the list of installed services in your cluster:
HDFS
MAPREDUCE2
SMARTSENSE
TEZ
YARN
ZOOKEEPER
########################### List of Host-wise installed components ###########################
kknew1.hwxblr.com
"component_name" | "DATANODE"
"component_name" | "HDFS_CLIENT"
"component_name" | "HST_AGENT"
"component_name" | "HST_SERVER"
"component_name" | "MAPREDUCE2_CLIENT"
"component_name" | "NAMENODE"
"component_name" | "NODEMANAGER"
"component_name" | "YARN_CLIENT"
"component_name" | "ZOOKEEPER_CLIENT"
kknew2.hwxblr.com
"component_name" | "DATANODE"
"component_name" | "HDFS_CLIENT"
"component_name" | "HST_AGENT"
"component_name" | "MAPREDUCE2_CLIENT"
"component_name" | "NODEMANAGER"
"component_name" | "SECONDARY_NAMENODE"
"component_name" | "YARN_CLIENT"
"component_name" | "ZOOKEEPER_CLIENT"
"component_name" | "ZOOKEEPER_SERVER"
kknew3.hwxblr.com
"component_name" | "APP_TIMELINE_SERVER"
"component_name" | "DATANODE"
"component_name" | "HDFS_CLIENT"
"component_name" | "HISTORYSERVER"
"component_name" | "HST_AGENT"
"component_name" | "MAPREDUCE2_CLIENT"
"component_name" | "NODEMANAGER"
"component_name" | "RESOURCEMANAGER"
"component_name" | "TEZ_CLIENT"
"component_name" | "YARN_CLIENT"
"component_name" | "ZOOKEEPER_CLIENT"
"component_name" | "ZOOKEEPER_SERVER"
sme-ambari-server.hwxblr.com
"component_name" | "HDFS_CLIENT"
"component_name" | "HST_AGENT"
"component_name" | "MAPREDUCE2_CLIENT"
"component_name" | "TEZ_CLIENT"
"component_name" | "YARN_CLIENT"
"component_name" | "ZOOKEEPER_CLIENT"
8. Stop individual service
[root@sme-ambari-server ambari]# ./ambari-admin.sh stop yarn
HTTP/1.1 202 Accepted
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=10tyime2kd7pr1e0o4t8gwg2jv;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 152
Server: Jetty(8.1.17.v20150415)
{
"href" : "http://sme-ambari-server.hwxblr.com:8080/api/v1/clusters/sme/requests/63",
"Requests" : {
"id" : 63,
"status" : "Accepted"
}
}
9. Start Individual service
[root@sme-ambari-server ambari]# ./ambari-admin.sh start yarn
HTTP/1.1 202 Accepted
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=a7wdqn56clk8176d99rm20hz5;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 152
Server: Jetty(8.1.17.v20150415)
{
"href" : "http://sme-ambari-server.hwxblr.com:8080/api/v1/clusters/sme/requests/64",
"Requests" : {
"id" : 64,
"status" : "Accepted"
}
}
10. Stop Individual service component
[root@sme-ambari-server ambari]# ./ambari-admin.sh stop hst_agent kknew2.hwxblr.com
HTTP/1.1 202 Accepted
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=vrmnhicrgog42lmu7exfqdm4;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 152
Server: Jetty(8.1.17.v20150415)
{
"href" : "http://sme-ambari-server.hwxblr.com:8080/api/v1/clusters/sme/requests/65",
"Requests" : {
"id" : 65,
"status" : "Accepted"
}
}
11. Start Individual service component
[root@sme-ambari-server ambari]# ./ambari-admin.sh start hst_agent kknew2.hwxblr.com
HTTP/1.1 202 Accepted
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=1fqqp5vqpourgjll9ydnyev3e;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 152
Server: Jetty(8.1.17.v20150415)
{
"href" : "http://sme-ambari-server.hwxblr.com:8080/api/v1/clusters/sme/requests/66",
"Requests" : {
"id" : 66,
"status" : "Accepted"
}
}
12. Remove any Hadoop client from any of the host
[root@sme-ambari-server ambari]# ./ambari-admin.sh remove tez_client kknew3.hwxblr.com
13. Add any Hadoop client on any of the host
[root@sme-ambari-server ambari]# ./ambari-admin.sh add tez_client kknew3.hwxblr.com
HTTP/1.1 201 Created
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=pl31hyai9aqt1eyeaj5ehe9i;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 0
Server: Jetty(8.1.17.v20150415)
HTTP/1.1 202 Accepted
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=18tzs8uctj2061pmjegp129aqz;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 152
Server: Jetty(8.1.17.v20150415)
{
"href" : "http://sme-ambari-server.hwxblr.com:8080/api/v1/clusters/sme/requests/67",
"Requests" : {
"id" : 67,
"status" : "Accepted"
}
}
Sleeping for 5 seconds before starting TEZ_CLIENT
HTTP/1.1 200 OK
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
User: admin
Set-Cookie: AMBARISESSIONID=1bh3u5foki8vh1fgg240i49h6x;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 0
Server: Jetty(8.1.17.v20150415)
14. Backup database for Hive/Oozie/Ambari
Note - Please enter your database password, for e.g. in this demo I have entered default password('bigdata') for Ambari postgresql DB.
[root@sme-ambari-server ambari]# ./ambari-admin.sh backup ambari postgresql sme-ambari-server.hwxblr.com
Password:
[root@sme-ambari-server ambari]# ls -lrt ~/ambari_db_backup_2016_05_22_11_32.sql
-rw-r--r-- 1 root root 6812385 May 22 11:33 /root/ambari_db_backup_2016_05_22_11_32.sql
[root@sme-ambari-server ambari]# date
Sun May 22 11:33:20 UTC 2016
[root@sme-ambari-server ambari]#
Note - I'm planning to add below features to this script, please feel to provide your feedback on this 🙂 Please feel free to suggest if you would like to have any more features! Stay Tuned for Part-2 Happy Hadooping!! 🙂
... View more
Labels:
02-23-2018
11:50 AM
@Kuldeep Kulkarni Add "deploy JCE policies" steps as prerequisites. I tried without JCE and it fails for me. Let me know if i am missing anything.
... View more
01-24-2017
11:35 AM
@Kuldeep Kulkarni, Yes it was a problem with my rpm command, that's why yum hung. I rebuild rpmdb and everything working fine now. thanks a ton. 🙂
... View more
08-19-2016
08:46 AM
HI Kuldeep, Thanks for the post. Need one help. I am trying to run the above example and checking for previous day date in input-events using <instance>${coord:current(-1)}</instance> But it is failing. When I use <instance>${coord:current(0)}</instance> then it runs successfully . here is my dryrun oozie output. Please help with hints/suggestions ***coordJob after parsing: ***<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="my_Scheduler_5f" frequency="1" start="2016-08-17T23:40Z" end="2016-08-19T23:45Z" timezone="America/Los_Angeles" freq_timeunit="DAY" end_of_duration="NONE"> <controls> <timeout>30</timeout> </controls> <input-events> <data-in name="coordInput_1" dataset="input1"> <dataset name="input1" frequency="1" initial-instance="2016-08-17T00:00Z" timezone="America/Los_Angeles" freq_timeunit="DAY" end_of_duration="NONE"> <uri-template>${nameNode}/myHdfsPath/Finalpath1/${YEAR}${MONTH}${DAY}/00/</uri-template> <done-flag>_Complete</done-flag> </dataset> <instance>${coord:current(-1)}</instance> </data-in> <data-in name="coordInput_2" dataset="input2"> <dataset name="input2" frequency="1" initial-instance="2016-08-17T23:00Z" timezone="America/Los_Angeles" freq_timeunit="DAY" end_of_duration="NONE"> <uri-template>${nameNode}/myHdfsPath/Finalpath2/${YEAR}${MONTH}${DAY}/00/</uri-template> <done-flag>_Complete</done-flag> </dataset> <instance>${coord:current(-1)}</instance> </data-in> </input-events> <action> <workflow> <app-path>${nameNode}/myHdfsPath/My_POC/wf-app-dir</app-path> <configuration> <property> <name>date</name> <value>${coord:formatTime(coord:dateOffset(coord:actualTime(),-1,'DAY'), "yyyyMMdd")}</value> </property> </workflow> </action></coordinator-app>***actions for instance***
Question with full details. https://community.hortonworks.com/questions/52412/how-to-configure-oozie-coordinator-dataset-for-pre.html
... View more
07-04-2017
03:08 PM
hi, I need to write in a local file using a action. but similarly I get the permission error. do you have any idea?
... View more
05-21-2019
04:59 PM
@Kuldeep Kulkarni - how we can schedule a weekly cron job on edge node which will run balancer and send the results via email to Hadoop Admin. Kindly help.
... View more
04-09-2018
06:10 PM
1 Kudo
@Harendra Sinha - No Oozie WebUI is readonly. You can have a look at Workflow manager from Ambari which has great features to design/run/re-run oozie workflows.
... View more
- « Previous
- Next »