Member since
01-06-2016
36
Posts
104
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
468 | 03-16-2017 08:21 AM | |
7170 | 03-15-2017 10:05 AM | |
1568 | 03-15-2017 08:18 AM | |
947 | 09-26-2016 09:22 AM | |
880 | 08-18-2016 05:00 AM |
06-25-2018
09:48 PM
2 Kudos
OBJECTIVE:
Verify the Beacon SSO setup is configured correctly.
OVERVIEW:
After you enable Knox SSO for beacon service as per the documentation, it is good to verify the Beacon Knox SSO setup is correctly configured before adding the clusters to the Dataplane.
STEPS:
STEP1:
From the ambari configs, get the beacon.sso.knox.providerurl property and execute the following curl command:
Syntax:
curl -iku $knox-username:$knox-password "<beacon.sso.knox.providerurl>/originalUrl=http://<beacon_server>:25986/api/becaon/cluster/list"
Example:
Command:
curl -iku $username:$password "https://hostname.hwx.site:8443/gateway/knoxsso/api/v1/websso?originalUrl=http://hostname.hwx.site:25968/api/beacon/cluster/list"
Output:
HTTP/1.1 307 Temporary Redirect
Date: Thu, 14 Jun 2018 21:26:27 GMT
X-Frame-Options: DENY
Set-Cookie: JSESSIONID=1abzrcp2xl7sm1k4jgtevkwv6x;Path=/gateway/knoxsso;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/knoxsso; Max-Age=0; Expires=Wed, 13-Jun-2018 21:26:27 GMT
Set-Cookie: hadoop-jwt=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbjEiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI5MTExNTg3fQ.lB4ZPXjqAQZJSS8vWJ3exXD4HcOCjTS6L4b9uIf6ZWc80eBTVuEv-u4iiEr02V44hxuEwVAeDVcDW1w0DGauW5L9hHfTKf_y87kaPhPKk2yN20aFtbbrA0lzgawxWIFFaj4wMxwyzDyyKlF6NRijamFhH00TWAH1vRITagVQWEc;Path=/;Domain=.hwx.site;HttpOnly
Location: http://hostname.hwx.site:25968/api/beacon/cluster/list
Content-Length: 0
Server: Jetty(9.2.15.v20160210)
Note the hadoop-jwt from the output
Syntax:
curl -ivL -u : --cookie "hadoop-jwt=<hadoop-jwt> " http://<beacon_server>:25986/api/becaon/cluster/list
Command:
curl -ivL -u : --cookie "hadoop-jwt=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbjEiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI5MTExNTg3fQ.lB4ZPXjqAQZJSS8vWJ3exXD4HcOCjTS6L4b9uIf6ZWc80eBTVuEv-u4iiEr02V44hxuEwVAeDVcDW1w0DGauW5L9hHfTKf_y87kaPhPKk2yN20aFtbbrA0lzgawxWIFFaj4wMxwyzDyyKlF6NRijamFhH00TWAH1vRITagVQWEc" http://hostname.hwx.site:25968/api/beacon/cluster/list
Output:
* About to connect() to hostname.hwx.site port 25968 (#0)
* Trying 172.27.54.132...
* Connected to hostname.hwx.site (172.27.54.132) port 25968 (#0)
* Server auth using Basic with user ''
> GET /api/beacon/cluster/list HTTP/1.1
> Authorization: Basic Og==
> User-Agent: curl/7.29.0
> Host: hostname.hwx.site:25968
> Accept: */*
> Cookie: hadoop-jwt=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbjEiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI5MTExNTg3fQ.lB4ZPXjqAQZJSS8vWJ3exXD4HcOCjTS6L4b9uIf6ZWc80eBTVuEv-u4iiEr02V44hxuEwVAeDVcDW1w0DGauW5L9hHfTKf_y87kaPhPKk2yN20aFtbbrA0lzgawxWIFFaj4wMxwyzDyyKlF6NRijamFhH00TWAH1vRITagVQWEc
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Expires: Thu, 01-Jan-1970 00:00:00 GMT
Expires: Thu, 01-Jan-1970 00:00:00 GMT
< Set-Cookie: JSESSIONID=xfigpewrhjme16egdblo9iz6p;Path=/
Set-Cookie: JSESSIONID=xfigpewrhjme16egdblo9iz6p;Path=/
< Content-Type: application/json
Content-Type: application/json
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Server: Jetty(6.1.26.hwx)
Server: Jetty(6.1.26.hwx)
<
* Connection #0 to host hostname.hwx.site left intact
{"totalResults":0,"results":0,"cluster":[]}
From the response, you can see the list of clusters added and paired. This verifies that Knox SSO setup with Beacon is working !!
... View more
- Find more articles tagged with:
- dataplane
- dlm
- FAQ
- Governance & Lifecycle
- knoxsso
Labels:
06-25-2018
09:05 PM
OBJECTIVE: Delete the cluster added in the Dataplane using utility script provided in the product. OVERVIEW: DPS-1.1 doesn't support the deletion of added clusters from the UI. If the users want to delete the clusters added in the Dataplane , an utility script is provided which can be used for deleting the HDP Clusters added. PREREQUISITES: Install jq : jq is a lightweight and flexible command-line JSON processor. STEPS: By default, "rm_dp_cluster.sh" script will be located at /usr/dp/current/core/bin. Execute the rm_dp_cluster.sh script by providing the following parameters
DP_JWT : Value of dp_jwt cookie from a valid users browser sessionn HADOOP_JWT : Value of hadoop-jwt cookie from a valid users browser session DP_HOST_NAME : Hostname or IP address of the DataPlane server CLUSTER_NAME: Ambari Cluster Name of the cluster to delete DATA_CENTER_NAME : Name of the cluster datacenter to delete For e.g: Format: ./rm_dp_cluster.sh <DP_JWT> <HADOOP_JWT> <DP_HOST_NAME> <CLUSTER_NAME> <DATA_CENTER_NAME> Executing the script: [root@dphost-mramasami-dlm-test-1 bin]# ./rm_dp_cluster.sh "eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE1Mjk5NjAzMTksInVzZXIiOiJ7XCJpZFwiOjIsXCJ1c2VybmFtZVwiOlwiYWRtaW4xXCIsXCJhdmF0YXJcIjpudWxsLFwicm9sZXNcIjpbXCJTVVBFUkFETUlOXCIsXCJJTkZSQUFETUlOXCJdLFwic2VydmljZXNcIjpbXCJkbG1cIl0sXCJkaXNwbGF5XCI6XCJhZG1pbjFcIixcInBhc3N3b3JkXCI6XCJcIixcImFjdGl2ZVwiOnRydWUsXCJkYk1hbmFnZWRcIjpmYWxzZSxcImdyb3VwTWFuYWdlZFwiOmZhbHNlLFwidXBkYXRlZEF0XCI6MTUyOTYxNTA1MX0ifQ.rM_L4m2vTb6pN3Qlz-pmfWjC83kEc29-u6SDzBrzz_1vhNinUaYTYiyqw3ELKtsJ062BaUmIhAiiv9NDMsaHlDfCmu7QrhfG4ki6YK-idgmWUhcnJS0O0xkq4evS4oYXHlOYV9RAWAzNiD378h-9-1pk8cpqH9FFHdq3KXH9tUfXV0AWYHeDhMlvAl_948-8DfCGeVjg5aBAWXKYO8PseILXB7skF812uaf5SlqCeobHAgZ1lUT7f9ZhN_i4jUXPc-uvoQK5_NYNu3gY8H9W1ECX7BXTzSqiws2etQNYOFBgIUwtbGFcVQOrjPKJi95avEXQxi0sFko_m1sHYh7X0krf25yYcb4AU195U2TqSYJ5pfD7OjXz9XGpalVGARNDa5l5qs_La4odZ9wmqWAgi4jBm2O15a1Faz8qkUTc2IoQ-Sldcfa2POs-mK9a2Elj6fDFCQk250ysWuH-N7gr_JGFhPWwt_4Kq9fCkANTrbZC2cGMVDm6lUY0i1DSwMIu3ZbUOVpu5-5xwR267V5nDaB08jL1MRwpHLaItumbGhO5iJKIVVSfYtoiMUZEbI-LbFAKQJqBLBZdr3hwCKY9Lavd391XLIYawT_wxZgMQYHnjm1luW350jQ34nEVDa-2T28Lvt5sLiKrUjV33NgPVY-1mjtzrSdesng5ybLv7sY" "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbjEiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI5OTYwMzE5fQ.D-vHyFj4c59qbN2Eop8FLfkZ1wibL20T4vHZGFYRV8ZBwD5x3X4dk-iwvY9i88aMENpaXt9whtOUMusBLMlBUge5TLlx8jXUAZ24BuoF2D6bTWzH1CUrBa16clH2hwvXuYZnx26jfrlLCKm9qdaZF83cD8LU-GwnY3dfEWCLi-gb3JDfdWQHTEluxLF4J-E86fZ2hiKc7F2o5aaRnGJugc-uEPSuNocCYmCYCpoh55sHzjj2VVRHa4-t7-pPYQNQaCX_vjXnvjQW2UYvMZpoyMRsAcaNHTAUm0zyidJ3q7zOLzasjZx4iRYnp1ttYa2F9Cdb8FpEJ6Qh8xNoZCLM0HH9mJC8fCMrd87IOX0Gw6dP9rYe58IRIMy3pvPW3sMnIUV_mtxFLGrL7tV1i8ubpXA3kSPOpKtk-YqshWy48Q2IPabGkI5mdAHukyKpW8IZHtTcYdMKHN9p2W7nCFu03gDorxDF4MkvLVgv3LF5-RL0zaqzgjY0kQ-gwZ--8-qX8QdvdNhr2Vg88DG2GnVRZc1tXFuHZDfHfQ3nCSiyITRw8hGlPN6GidZwRyXgg8-Ku9rSqv9AbUbEUeF9_XHR4fc5G_qsBmdF5KurTYmifXF0PcVFejjl1_10kVpUeTK4J_Qg21FLHWDwd6rwCZWspaXLyUmhS0kzHymrsutSmN0" 172.22.118.100 onpremdlm SFO_DC Output: % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 29166 0 29166 0 0 93490 0 --:--:-- --:--:-- --:--:-- 93781
Found cluster with name onpremdlm in DataCenter SFO_DC having id 2
Do you want to delete this cluster? (yes / no)? yes
Deleting cluster...
true
Now the onpremdlm successfully deleted.
... View more
Labels:
03-20-2017
04:13 PM
3 Kudos
@Sunile Manjee Comments functionality is supported in Phoenix as well. Ex: "SELECT /* this is a comment */ CAST(USNIG_LONG_ID as DECIMAL) FROM <Table_Name> ORDER BY USNIG_LONG_ID DESC limit 3;" Reference: https://phoenix.apache.org/language/#comments
... View more
03-20-2017
01:11 PM
3 Kudos
For Mirroring we don't need to create the feed and process entity. We need to create the source and target cluster and submit the mirror job.
... View more
03-17-2017
12:32 PM
4 Kudos
@Guillaume Roger One of the solution is you can specify it in the config-default.xml. This file should be present in the folder where your workflow.xml are present, and it is automatically parsed for properties.This file has default values for variables that are not defined via the job.properties file or the -D option. You can use the same job.properties file for all invocations of the workflow or use a different properties file for different runs. But the config-default.the xml file is valid for all invocations of the job. Reference: https://issues.apache.org/jira/browse/OOZIE-1673
... View more
03-16-2017
08:21 AM
7 Kudos
@Santhosh B Gowda There is no direct rest api call where we find out "isHAEnabled" for HDFS. But when we enabled HA for HDFS, "dfs.nameservices" parameter will gets set. So we get the value of dfs.nameservices from the configs and if it is empty then HA is not enabled. If it contains value HA is enabled. you can get the configs using the following API. http://<AMBARI-SERVER>/api/v1/clusters/cl1/configurations?type=hdfs-site&tag=TOPOLOGY_RESOLVED
... View more
03-16-2017
06:05 AM
2 Kudos
CTAS has these restrictions:
The target table cannot be a partitioned table. The target table cannot be an external table. The target table cannot be a list bucketing table.
... View more
03-16-2017
05:44 AM
2 Kudos
Thanks @Gnanasekaran G please accept best answer. we can close the thread
... View more
03-16-2017
04:58 AM
1 Kudo
@Gnanasekaran G if this helped, please vote/accept best answer. we can close the thread
... View more
03-16-2017
04:58 AM
@Gnanasekaran G if this helped, please vote/accept best answer. we can close the thread
... View more
03-16-2017
02:33 AM
@zaenal rifai If this helped, please vote/accept best answer.
... View more
03-15-2017
10:51 AM
2 Kudos
@Gnanasekaran G Use the following ( <=>) operator instead of (=) operator SELECT CASE WHEN (NULL<=>NULL) THEN "equals" ELSE "not equals" end AS value; Reference: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-RelationalOperators
... View more
03-15-2017
10:05 AM
8 Kudos
@Gnanasekaran G Use the following ( <=>) operator instead of (=) operator SELECT CASE WHEN (NULL<=>NULL) THEN "equals" ELSE "not equals" end AS value; Reference: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-RelationalOperators
... View more
03-15-2017
09:11 AM
1 Kudo
@zaenal rifai No. It is the limitation from oozie UI.
... View more
03-15-2017
08:18 AM
7 Kudos
@zaenal rifai The maximum number of action oozie DAG will show is 25. Reference https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/GraphGenerator.java#L62 https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/GraphGenerator.java#L341
... View more
11-03-2016
03:52 AM
2 Kudos
@zhixun he Yes. Whenever there is a change, snapshot will get created in source and falcon process instance will trigger based on the frequency
... View more
10-25-2016
05:29 PM
10 Kudos
HDFS Snapshots are read-only point-in-time copies of the file system. Snapshots can be taken on a subtree of the file system or the entire file system.Snapshots are very efficient because they only copy data that are changed. We can restore the data to any previous snapshot. Some common use cases of snapshots are Data backup and Disaster recovery. HDFS Snapshot Extension: Falcon will support HDFS snapshot-based replication through HDFS Snapshot extension. Using this feature, create and manage snapshots on source/target directories. Mirror data from source to target for disaster recovery using these snapshots. Perform retention on the snapshots created on source and target. Snapshot replication will only work from single source directory to single target directory. For snapshot to work, we expect users to do the following Both source and target clusters must have a version of Hadoop 2.7.0 or higher. The user submitting and scheduling the falcon extension should have permissions on both source and target directories. Both directories should be snap shotable.
To perform the HDFS Snapshot replication in Falcon, We need to create the source, target cluster entities and also need to create/give permissions to the staging and working directories. Please use the following steps to accomplish it. Source Cluster: hdfs dfs -rm -r /tmp/fs /tmp/fw
hdfs dfs -mkdir -p /tmp/fs
hdfs dfs -chmod 777 /tmp/fs
hdfs dfs -mkdir -p /tmp/fw
hdfs dfs -chmod 755 /tmp/fw
hdfs dfs -chown falcon /tmp/fs
hdfs dfs -chown falcon /tmp/fw Target Cluster : hdfs dfs -rm -r /tmp/fs /tmp/fw
hdfs dfs -mkdir -p /tmp/fs
hdfs dfs -chmod 777 /tmp/fs
hdfs dfs -mkdir -p /tmp/fw
hdfs dfs -chmod 755 /tmp/fw
hdfs dfs -chown falcon /tmp/fs
hdfs dfs -chown falcon /tmp/fw Cluster Entities: primaryCluster.xml <?xml version="1.0" encoding="UTF-8"?>
<cluster xmlns="uri:falcon:cluster:0.1" colo="USWestOregon" description="oregonHadoopCluster" name="primaryCluster">
<interfaces>
<interface type="readonly" endpoint="webhdfs://mycluster1:20070" version="0.20.2" />
<interface type="write" endpoint="hdfs://mycluster1:8020" version="0.20.2" />
<interface type="execute" endpoint="primaryCluster-12.openstacklocal:8050" version="0.20.2" />
<interface type="workflow" endpoint="http://primaryCluster-14.openstacklocal:11000/oozie" version="3.1" />
<interface type="messaging" endpoint="tcp://primaryCluster-9.openstacklocal:61616?daemon=true" version="5.1.6" />
<interface type="registry" endpoint="thrift://primaryCluster-14.openstacklocal:9083" version="0.11.0" />
</interfaces>
<locations>
<location name="staging" path="/tmp/fs" />
<location name="temp" path="/tmp" />
<location name="working" path="/tmp/fw" />
</locations>
<ACL owner="ambari-qa" group="users" permission="0755" />
<properties>
<property name="dfs.namenode.kerberos.principal" value="nn/_HOST@EXAMPLE.COM" />
<property name="hive.metastore.kerberos.principal" value="hive/_HOST@EXAMPLE.COM" />
<property name="hive.metastore.sasl.enabled" value="true" />
<property name="hadoop.rpc.protection" value="authentication" />
<property name="hive.metastore.uris" value="thrift://primaryCluster-14.openstacklocal:9083" />
<property name="hive.server2.uri" value="hive2://primaryCluster-14.openstacklocal:10000" />
</properties>
</cluster> falcon entity -submit -type cluster -file primaryCluster.xml --> primaryCluster backupCluster : <?xml version="1.0" encoding="UTF-8"?>
<cluster xmlns="uri:falcon:cluster:0.1" colo="USWestOregon" description="oregonHadoopCluster" name="backupCluster">
<interfaces>
<interface type="readonly" endpoint="webhdfs://mycluster2:20070" version="0.20.2" />
<interface type="write" endpoint="hdfs://mycluster2:8020" version="0.20.2" />
<interface type="execute" endpoint="backupCluster-5.openstacklocal:8050" version="0.20.2" />
<interface type="workflow" endpoint="http://backupCluster-6.openstacklocal:11000/oozie" version="3.1" />
<interface type="messaging" endpoint="tcp://backupCluster-1.openstacklocal:61616" version="5.1.6" />
<interface type="registry" endpoint="thrift://backupCluster-6.openstacklocal:9083" version="0.11.0" />
</interfaces>
<locations>
<location name="staging" path="/tmp/fs" />
<location name="temp" path="/tmp" />
<location name="working" path="/tmp/fw" />
</locations>
<ACL owner="ambari-qa" group="users" permission="0755" />
<properties>
<property name="dfs.namenode.kerberos.principal" value="nn/_HOST@EXAMPLE.COM" />
<property name="hive.metastore.kerberos.principal" value="hive/_HOST@EXAMPLE.COM" />
<property name="hive.metastore.sasl.enabled" value="true" />
<property name="hadoop.rpc.protection" value="authentication" />
<property name="hive.metastore.uris" value="thrift://backupCluster-6.openstacklocal:9083" />
<property name="hive.server2.uri" value="hive2://backupCluster-6.openstacklocal:10000" />
</properties>
</cluster> falcon entity -submit -type cluster -file backupCluster.xml --> backupCluster HDFS Snapshot Replication: Source: [ Create directory and copy the data] hdfs dfs -mkdir -p /tmp/falcon/HDFSSnapshot/source
hdfs dfs -put NYSE-2000-2001.tsv /tmp/falcon/HDFSSnapshot/source Note: you can download the NYSE-2000-2001.tsv file from https://s3.amazonaws.com/hw-sandbox/tutorial1/NYSE-2000-2001.tsv.gz Allow Snapshot to the directory: ddfs dfsadmin -allowSnapshot /tmp/falcon/HDFSSnapshot/source [ hdfs]
hdfs lsSnapshottableDir [ ambari-qa] Target Cluster hdfs dfs -mkdir -p /tmp/falcon/HDFSSnapshot/target
hdfs dfsadmin -allowSnapshot /tmp/falcon/HDFSSnapshot/target hdfs-snapshot.properties jobName=HDFSSnapshot
jobClusterName=primaryCluster
jobValidityStart=2016-05-09T06:25Z
jobValidityEnd=2016-05-09T08:00Z
jobFrequency=days(1)
sourceCluster=primaryCluster
sourceSnapshotDir=/tmp/falcon/HDFSSnapshot/source
sourceSnapshotRetentionAgeLimit=days(1)
sourceSnapshotRetentionNumber=3
targetCluster=backupCluster
targetSnapshotDir=/tmp/falcon/HDFSSnapshot/target
targetSnapshotRetentionAgeLimit=days(1)
targetSnapshotRetentionNumber=3
jobAclOwner=ambari-qa
jobAclGroup=users
jobAclPermission="0x755" Submit And schedule the job using the property file: falcon extension -extensionName hdfs-snapshot-mirroring -submitAndSchedule -file hdfs-snapshot.properties By using the jobName we can find the oozie job it has launched falcon extension -instances -jobName HDFSSnapshot Once the job is completed, we can see in source the snapshot will be automatically created and snapshot along with source content are replicated in the target cluster : Source Cluster HDFS Content: hdfs dfs -ls -R hdfs://mycluster1:8020//tmp/falcon/HDFSSnapshot/source/ drwxr-xr-x - ambari-qa hdfs 0 2016-10-25 02:27 hdfs://mycluster1:8020/tmp/falcon/HDFSSnapshot/source/source -rw-r--r-- 3 ambari-qa hdfs 44005963 2016-10-25 02:27 hdfs://mycluster1:8020/tmp/falcon/HDFSSnapshot/source/source/NYSE-2000-2001.tsv Target Cluster HDFS Content: hdfs dfs -ls -R hdfs://mycluster2:8020//tmp/falcon-HDFSSnapshot/target/ drwxr-xr-x - ambari-qa hdfs 0 2016-10-25 02:28 hdfs://mycluster2:8020/tmp/falcon/HDFSSnapshot/target/source -rw-r--r-- 3 ambari-qa hdfs 44005963 2016-10-25 02:28 hdfs://mycluster2:8020/tmp/falcon/HDFSSnapshot/target/source/NYSE-2000-2001.tsv We can see the data has been replicated from source to target cluster. Source Snapshot Directory: hdfs dfs -ls hdfs://mycluster1:8020//tmp/falcon/HDFSSnapshot/source/.snapshot Found 1 items drwxr-xr-x - ambari-qa hdfs 0 2016-10-25 02:27 hdfs://mycluster1:8020/tmp/falcon/HDFSSnapshot/source/.snapshot/falcon-snapshot-HDFSSnapshot-2016-05-09-06-25-1477362461509 Target Snapshot Directory: hdfs dfs -ls hdfs://mycluster2:8020//tmp/falcon/HDFSSnapshot/target/.snapshot Found 1 itemsdrwxr-xr-x - ambari-qa hdfs 0 2016-10-25 02:28 hdfs://mycluster2:8020/tmp/falcon/HDFSSnapshot/target/.snapshot/falcon-snapshot-HDFSSnapshot-2016-05-09-06-25-1477362461509 We can see the snapshot directory has been automatically created in source and also replicated from source to target cluster.
... View more
- Find more articles tagged with:
- disaster-recovery
- Distcp
- Falcon
- Governance & Lifecycle
- HDFS
- How-ToTutorial
- replication
- snapshot
Labels:
09-27-2016
01:08 PM
3 Kudos
@Saurabh Service check fail If there is delay in completing the Oozie job [ Max duration : 300 secs] , Please check that ResourceManager must not be in overwhelming state and there must be sufficient capacity in cluster to execute job. Also can you please check the application log for the oozie service check whether it is completed or not. If the application is failed, can you please give the failed application log.
... View more
09-26-2016
01:32 PM
4 Kudos
@Saurabh Can you run the following command in your ambari server and let us know the output for further debugging. /usr/bin/yum install hdp-select
... View more
09-26-2016
10:30 AM
1 Kudo
Good descriptive article on how to install Atlas HA via Ambari
... View more
09-26-2016
09:58 AM
2 Kudos
@Anshul Sisodia In the UI, Cluster entities will not be displayed in the Search output. Alternatively, you can use the following CLI command to list the cluster entities submitted. falcon entity -list -type cluster
... View more
09-26-2016
09:22 AM
6 Kudos
@Santhosh B Gowda To Increase the size of the workflow job definition size. Please add the following property to oozie-site.xml: oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength=<The maximum length of the workflow definition in bytes> For Example: oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength=1000000
... View more
08-18-2016
05:00 AM
5 Kudos
@ Gaurab D can you share your shell script? Looks like the script has been executed but the desired output which we are comparing are not correct? Thanks
... View more
08-18-2016
04:46 AM
1 Kudo
@Rajib Mandal Looks like kerberos principal specified in the property file is incorrect. you can refer the following apache doc: https://oozie.apache.org/docs/3.1.3-incubating/DG_UsingHadoopKerberos.html Also you can find the dfs.namenode.kerberos.principal in hdfs-site.xml and mapreduce.jobtracker.kerberos.principal in mapred-site.xml
... View more
08-17-2016
09:49 AM
5 Kudos
@William Bolton You can use the below ambari api to get the Active NN: http://{HOST}:8080/api/v1/clusters/{CLUSTER_NAME}/host_components?HostRoles/component_name=NAMENODE&metrics/dfs/FSNamesystem/HAState=active
... View more
08-12-2016
12:21 PM
1 Kudo
@Prakash M Configurations looks good. Can you try setting the properties, create the table and try to insert/update?
... View more
08-12-2016
11:59 AM
1 Kudo
@Sanjib Behera can you list the directory structure [${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce] ? Can you tell whether the libraries are copied to the hdfs folder [ ${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/lib] folder?
... View more
08-11-2016
05:57 PM
1 Kudo
Thanks @Vaibhav Kumar . In that case you can use the "Data Viewer" on source and check it if is retrieving some rows. if it does, you can also use "Data Viewer" in your last transformation to see how many rows it isn getting.
... View more
08-11-2016
02:36 PM
2 Kudos
@Sanjib Behera Can you please share your job.properties file? Please check the ports in job.properties This is usually problem with namenode and jobtracker ports. Make sure your jobtracker port is pointing to the resource manager port in job.properties file.
... View more
08-11-2016
01:54 PM
1 Kudo
Thanks @Shary M for providing the workflow. Looks like the arguments which we have passed might not be passed to the java application . The way we need to specify the argument in the application as args[0] .. args[n], where arg[0] is the argument passed first in the oozie workflow. In the above one,
arg[0] --> -logtype and arg[1] --> adraw You can refer the following examples.
Sample workflow: https://github.com/apache/oozie/blob/master/examples/src/main/apps/spark/workflow.xml Sample java application : https://github.com/apache/oozie/blob/master/examples/src/main/java/org/apache/oozie/example/SparkFileCopy.java Please let us know if you need more information. If it failing again, please share the snippet of your application also.
... View more