Member since
11-07-2016
637
Posts
253
Kudos Received
144
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1319 | 12-06-2018 12:25 PM | |
1167 | 11-27-2018 06:00 PM | |
961 | 11-22-2018 03:42 PM | |
1746 | 11-20-2018 02:00 PM | |
3049 | 11-19-2018 03:24 PM |
12-06-2018
12:25 PM
2 Kudos
@nur majid, You can use the below API to validate the config curl -u username:password -X POST -d '{post-body-json-mentioned-below}' -H "Content-Type:application/json" http://{ranger-host}:{port}/service/plugins/services/validateConfig Sample post body looks like below {
"id": 1,
"guid": "fd82acf1-d3e9-4707-9a75-a839a9802cd9",
"isEnabled": true,
"createdBy": "hdfs",
"updatedBy": "hdfs",
"createTime": 1544045853000,
"updateTime": 1544045853000,
"version": 1,
"type": "hdfs",
"name": "cl1_hadoop",
"description": "hdfs repo",
"configs": {
"username": "hadoop",
"password": "*****",
"fs.default.name": "hdfs://mycluster",
"hadoop.security.authorization": true,
"hadoop.security.authentication": "kerberos",
"hadoop.security.auth_to_local": "RULE:[1:$1@$0](ambari-qa@EXAMPLE.COM)s/.*/ambari-qa/RULE:[1:$1@$0](hbase@EXAMPLE.COM)s/.*/hbase/RULE:[1:$1@$0](hdfs@EXAMPLE.COM)s/.*/hdfs/RULE:[1:$1@$0](yarn-ats@EXAMPLE.COM)s/.*/yarn-ats/RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//RULE:[2:$1@$0](amshbase@EXAMPLE.COM)s/.*/ams/RULE:[2:$1@$0](amsmon@EXAMPLE.COM)s/.*/ams/RULE:[2:$1@$0](amszk@EXAMPLE.COM)s/.*/ams/RULE:[2:$1@$0](atlas@EXAMPLE.COM)s/.*/atlas/RULE:[2:$1@$0](dn@EXAMPLE.COM)s/.*/hdfs/RULE:[2:$1@$0](hbase@EXAMPLE.COM)s/.*/hbase/RULE:[2:$1@$0](hive@EXAMPLE.COM)s/.*/hive/RULE:[2:$1@$0](jhs@EXAMPLE.COM)s/.*/mapred/RULE:[2:$1@$0](jn@EXAMPLE.COM)s/.*/hdfs/RULE:[2:$1@$0](knox@EXAMPLE.COM)s/.*/knox/RULE:[2:$1@$0](nfs@EXAMPLE.COM)s/.*/hdfs/RULE:[2:$1@$0](nm@EXAMPLE.COM)s/.*/yarn/RULE:[2:$1@$0](nn@EXAMPLE.COM)s/.*/hdfs/RULE:[2:$1@$0](rangeradmin@EXAMPLE.COM)s/.*/ranger/RULE:[2:$1@$0](rangerkms@EXAMPLE.COM)s/.*/keyadmin/RULE:[2:$1@$0](rangertagsync@EXAMPLE.COM)s/.*/rangertagsync/RULE:[2:$1@$0](rangerusersync@EXAMPLE.COM)s/.*/rangerusersync/RULE:[2:$1@$0](rm@EXAMPLE.COM)s/.*/yarn/RULE:[2:$1@$0](yarn@EXAMPLE.COM)s/.*/yarn/RULE:[2:$1@$0](yarn-ats-hbase@EXAMPLE.COM)s/.*/yarn-ats/DEFAULT",
"dfs.datanode.kerberos.principal": "dn/test-node-4.openstacklocal@EXAMPLE.COM",
"dfs.namenode.kerberos.principal": "nn/test-node-4.openstacklocal@EXAMPLE.COM",
"dfs.secondary.namenode.kerberos.principal": "nn/test-node-4.openstacklocal@EXAMPLE.COM",
"hadoop.rpc.protection": "privacy",
"commonNameForCertificate": "-",
"tag.download.auth.users": "hdfs",
"policy.download.auth.users": "hdfs"
},
"policyVersion": 3,
"policyUpdateTime": 1544045856000,
"tagVersion": 1,
"tagUpdateTime": 1544045853000,
"tagService": ""
} You can get the exact json for your cluster from the browser's developer tools Right Click -> Inspect -> Network -> Click on the request -> Request payload . If this helped you , please take a moment to login and "Accept" the answer 🙂
... View more
11-27-2018
06:09 PM
@joo Antoine, I guess the date format should be "dd/MM/yy HH:mm" instead of "d/MM/yy HH:mm". Looks like a 'd' is missing
... View more
11-27-2018
06:00 PM
1 Kudo
@Sami Ahmad For Knox , you have to make 2 curl calls. The second curl call should be made to the "Location" header obtained from the 1st curl call response. 1st curl call curl -i -k -u admin:admin-password -X GET 'https://localhost:8443/gateway/default/webhdfs/v1/tmp/uname.txt?op=OPEN'
HTTP/1.1 307 Temporary Redirect
Date: Tue, 27 Nov 2018 16:21:44 GMT
Set-Cookie: JSESSIONID=1219u2f8zreb11eu9fuxlggxhq;Path=/gateway/default;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Mon, 26-Nov-2018 16:21:44 GMT
Cache-Control: no-cache
Expires: Tue, 27 Nov 2018 16:21:44 GMT
Date: Tue, 27 Nov 2018 16:21:44 GMT
Pragma: no-cache
Expires: Tue, 27 Nov 2018 16:21:44 GMT
Date: Tue, 27 Nov 2018 16:21:44 GMT
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Location: https://hadoop1:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/tmp/uname.txt?_=AAAACAAAABAAAACgLvtILkFAljr5PIP7MVSOAump8j0kSwFCPdGCP2R_b1tCZ0V2KGOQuiRiI4_IU7GDG6NqRtK2Vu7DOZeOhbuQUaP1FYtD_-IV3P-VXMbOFbPfbwpNseAuN-RyQduRm5S1mrk0GVbYKQg4NscgsoF0GGsvqKDyPtECwhwkX96E37Jc5_yCnlkw3LVKUY41Hg6LOt96W8-3rTmnrbo7o26dOcpPv1_uv4Q1F18b4yk5N5BNf6HTZdVZ6Q
Content-Type: application/octet-stream
Server: Jetty(6.1.26.hwx)
Content-Length: 0
2nd curl call (url is taken from Location header obtained in 1st curl call) curl -i -k -u admin:admin-password -X GET https://hadoop1:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/tmp/uname.txt?_=AAAACAAAABAAAACgLvtILkFAljr5PIP7MVSOAump8j0kSwFCPdGCP2R_b1tCZ0V2KGOQuiRiI4_IU7GDG6NqRtK2Vu7DOZeOhbuQUaP1FYtD_-IV3P-VXMbOFbPfbwpNseAuN-RyQduRm5S1mrk0GVbYKQg4NscgsoF0GGsvqKDyPtECwhwkX96E37Jc5_yCnlkw3LVKUY41Hg6LOt96W8-3rTmnrbo7o26dOcpPv1_uv4Q1F18b4yk5N5BNf6HTZdVZ6Q
... View more
11-22-2018
03:42 PM
1 Kudo
@chris herssens, Looks like there is a resource crunch. Can you try adding additional NodeManagers if possible. Check if any applications are running in YARN. You can kill any app if it's not used and see if your spark job goes to running state from accepted.
... View more
11-22-2018
03:33 PM
@Guillaume Roger, I guess ATSv2 is running as a service and not embedded mode. Can you filter for "is_hbase_system_service" in YARN configs and check the value. If it is set to true, then ATS v2 will be running as a yarn application. Else, it will be running in embedded mode. If it is running as a yarn application, then it can be started on any of the node where NodeManagers are present with proper resources. Can you check in the Yarn application logs , if HBase master and region servers are able to come up properly.
... View more
11-21-2018
03:46 AM
@Andreas Kühnert, Glad that the issue is resolved. Since this is a different issue, I suggest to open a new thread for this issue so that the main thread doesn't get deviated. I'm not sure of that issue, may be other experts can help 🙂
... View more
11-20-2018
03:18 PM
@seongbeom shin, You need to give the complete path of the filesystem. You can just change it like below jsonData = spark.read.json("/user/maria_dev/example.json") Spark will figure it out from fs.defaultFS value. If you still want to pass the complete path, then you can try with the IP address of the namenode instead of localhost. .
... View more
11-20-2018
03:15 PM
@Denise O Regan, Can you try running these commands on the ambari server node and mysql node where hive metastore is pointing and see if it works # yum install mysql-connector-java
# ls /usr/share/java/mysql-connector-java.jar Run this command only on ambari server node # ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar . If this works, please take a moment to login and "Accept" the answer 🙂
... View more
11-20-2018
02:26 PM
/etc/hosts file just translates the hostname to the ip. Even without the mapping you can open the urls by passing the ip. If you want to access it using hostname, then you need the entries in /etc/hosts files. It doesn't do any access control.
... View more
11-20-2018
02:04 PM
@Prashant Ahire, If this helped , please take a moment to login and "Accept" the answer. This will be useful for other community users.
... View more
11-20-2018
02:00 PM
@kanna k, Did you make the /etc/hosts entries in your laptop/desktop from where you are accessing the ambari cluster. If not, please add entries in your desktop as well.
... View more
11-19-2018
03:24 PM
@Andreas Kühnert, I guess the directory got deleted somehow from HDFS. You can create it and try starting RM # su hdfs
# kinit -kt /etc/security/keytabs/hdfs.headless.keytab {principal} ---> Run this if your environment is kerberized
# hdfs dfs -mkdir -p /ats/done
# hdfs dfs -chown -R yarn:hadoop /ats/done After running the commands, try to restart RM. If it fails with /ats/active directory not found, repeat the same steps changing the directory name. . If this works , please take a moment to login and "Accept" the answer.
... View more
11-19-2018
03:10 PM
@Prashant Ahire, Yes. The incremental import is not supported for CHAR,VARCHAR data types. You can use integer or date datatypes for -check-column.
... View more
11-15-2018
11:58 AM
You can work with pyspark if you know python. All the features will be available in pyspark as well. You need not learn scala or java
... View more
11-15-2018
06:32 AM
You need not learn both Java and Scala to start with spark. If need to be familiar with either of Java,Scala, Python or R to work with spark. You can start with this tutorial to understand the basics https://hortonworks.com/tutorial/hands-on-tour-of-apache-spark-in-5-minutes/ https://www.tutorialspoint.com/pyspark/index.htm
... View more
11-14-2018
05:44 AM
@Madhura Mhatre, Yes, Those are the Thrift servers which will be running forever. You can stop the thrift server from ambari and see that jobs will be stopped. This is the expected behavior. . If this helps, please take a moment login and "Accept" the answer.
... View more
10-30-2018
12:14 PM
@Yuyan Huang, Looks like the mapred user got deleted somehow. Can you try running the below commands on the node where HistoryServer is running and also on other nodes to confirm if the user is deleted from all nodes. # id mapred
# id 1005 If the user is not found, you can try adding the user mapred. Kill the HistoryServer process if running from backend and try starting it again
... View more
10-30-2018
12:02 PM
@Lenu K, You can use this link to install ranger. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_security/content/ranger_install.html
... View more
10-24-2018
05:07 PM
Can you try solution mentioned in this question https://community.hortonworks.com/questions/61415/ranger-audit-to-solr-problem.html
... View more
10-24-2018
05:01 PM
@JAy PaTel, You can use Kafka,Scribe(from facebook), jms, tailfile data sources. https://github.com/apache/flume/tree/trunk/flume-ng-sources
... View more
10-24-2018
04:30 PM
@Lok! Reddy, Did you enable Audit to Solr under Ranger Audit sections. Also, I do not see any services in Ranger. Did you restart all the services after enabling ranger plugins.
... View more
10-23-2018
04:41 PM
@Anpan K, u is the unicode representation of the string in python. To convert it into a normal string, you can use str() method. You can test it in python console # a = u'teststring'
# a
>> u'teststring'
# b = str(a)
# b
>> 'teststring' For your use case, you can use datacontent = sc.textFile("file:///tmp/password.txt")
datacontent = datacontent.map(lambda x: str(x))
datacontent.collect()
... View more
10-22-2018
01:47 PM
@Michail Smyrlis, HBase is dependent on HDFS. Since NameNode was down it might have stopped but now since NameNode is up and running HBase master should start properly. Did you check all the memory settings and logs of hbase master
... View more
10-22-2018
12:09 PM
@Michail Smyrlis, Can you check the value of 'free -m' in the node where HBase master is running. Are both HBase master and region server are running on same node ? If "free -m" returns less than 1gb then there is not enough memory for hbase to start. Just for testing, you can stop some other services running on that node and try starting HBase master.
... View more
10-22-2018
12:06 PM
@Lucky_Luke, This is required only from one node. You should be able to do passwordless ssh from ambari server node to all the other nodes. # cat .ssh/id_rsa.pub | ssh root@{ambari-ip} 'cat >> .ssh/authorized_keys'
... View more
10-22-2018
11:54 AM
@Michail Smyrlis, Can you check your heap memory of HBase master and the available memory in your node. This might be mostly because of memory issues.
... View more
10-19-2018
01:12 AM
@Felix Albani, This worked like a charm. Thanks a lot for your help. Really appreciate 🙂 However in the latest version of Ambari, it should have been handled by Ambari itself. I do not see the manual step in this doc. Must be a doc bug or ambari issue in my cluster. https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/managing-high-availability/content/amb_enable_namenode_high_availability.html
... View more
10-18-2018
05:13 PM
@Christos Stefanopoulos, Do you mean the Ambari users? I'm not aware of any export/import of users. You can do below steps Get users from cluster1 curl -iv -u admin:admin http://{ambari-host}:{ambari-port}/api/v1/users Post them to cluster2 using some default password and ask them to change again curl -iv -u admin:admin -H "X-Requested-By: ambari"-X POST -d '{"Users/user_name":"username","Users/password":"defaultpwd","Users/active":"true","Users/admin":"true/false"}' http://{ambari-host}:{ambari-port}/api/v1/users You can write a script to read from the 1st API and pass it to 2nd API
... View more
10-18-2018
02:39 PM
@Felix Albani, Yes I tried copying both core-site.xml and also hdfs-site.xml but still facing the same issue. Attaching some logs, spark thrift server start logs in debug mode and corresponding yarn application logs. yarn-app-logs.txt spark-spark-orgapachesparksqlhivethriftserverhivet.zip Also made sure that "/hadoop/yarn/local/usercache/spark/filecache/10/__spark_conf__.zip/__spark_conf__/__hadoop_conf__/core-site.xml" has correct content.
... View more
10-18-2018
09:25 AM
1 Kudo
@Christos Stefanopoulos, That is the expected behaviour. If you want to achieve this, then you need to create separate configs for each host. For ex: hostA will have only /grid in datanode dirs and hostB will have /grid0 and /grid1 in their datanode dirs. You can do that using Ambari config groups https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/using_host_config_groups.html
... View more