Member since
12-20-2016
18
Posts
0
Kudos Received
0
Solutions
08-29-2017
11:58 AM
Hi, Is there any configuration changes that has to be made for Apache Eagle to sustain its data..like the sites, applications, policies, everything created is lost every time the service is restarted. Thanks in advance!
... View more
07-06-2017
01:02 PM
Hi I'm trying to add new nodes to an existing cluster through ambari api..I'm able to add datanodes, nodemanagers and clients referring this but to add services like spark, kafka, ranger, hive, etc for the first time to an existing cluster I'm not able to find an API that can add the service with "default configuration". I tried this but in ambari UI the configuration changes are not getting reflected. So instead of giving key values I would like to have the default configurations.. Thanks in advance!
... View more
Labels:
06-28-2017
07:27 AM
@Mugdha Test connection works..The database is on ranger host.. Is the default DB port 3306..?
... View more
06-27-2017
12:44 PM
Hi, Getting these two errors when trying to install ranger, Internal Exception: java.sql.SQLException: No database selected
Error Code: 1046 [E] applying java patch PatchPasswordEncryption_J10001 failed When I googled regarding this, ranger-admin-site.xml has bad/missing values for: - ranger.externalurl - ranger.jpa.audit.jdbc.url - ranger.jpa.jdbc.url Of which I have only one property configured in my blueprint(ranger.jpa.jdbc.url)
what should be the values of this..
It was jdbc:mysql://ip_of_ranger_host Then I changed it to jdbc:mysql://ip_of_ranger_host/ranger Thanks in Advance!
... View more
Labels:
04-10-2017
06:58 AM
This is the part of ansible playbook.. - name: Waiting for ambari-agents to register
uri: url="http://{{ ansible_default_ipv4.address }}:8080/api/v1/hosts/{{ hostvars[item]['ansible_fqdn'] | lower }}"
method=GET
force_basic_auth=yes
user=admin
password=admin
HEADER_X-Requested-By="ambari"
status_code=200,201,202,404
with_items: "{{ groups['all'] }}"
register: result
until: "result.status != 404"
retries: 200
delay: 5
... View more
04-10-2017
06:45 AM
@Jay SenSharma So ideally ambari-agent.ini file should contain the ambari-server's hostname and /etc/hosts file must contain dns of both server and agent machines which was provided.. FYR.. ambari-agent.ini [server] hostname=ambari_server_hostname url_port=8440 secured_url_port=8441 Is it required to provide the ambari-agent's hostname in the ambari-agent.ini file..?
... View more
04-10-2017
06:18 AM
Hi I've been working with ansible to create hadoop clusters, everything works fine except ambari agents fail to connect to ambari server. Even though ambari agents are in running state and that ambari-agent.ini file is updated with ambari-servers hostname, versions also match. I've written the ansible playbook in such a way that ambari-server tries to connect with ambari-agents for 200 times with 5 second delay, but at the end of the loop it fails. If I manually log in into each agent machine and restart the agents during the 200 tries, it gets connected with the server. Thanks in advance!!
... View more
Labels:
03-31-2017
04:00 PM
Is it possible to view hive tables and their contents from ranger UI? If ranger provides only creating policies for hive database
... View more
03-31-2017
12:43 PM
Hi, I would like to know if ranger provides details about hive, hbase and hdfs tables. I've come across something called "hive view" of ambari which provides such service. Is there something similar for hbase and hdfs. Thanks in advance!
... View more
- Tags:
- Hadoop Core
- HBase
- HDFS
- Ranger
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
Labels:
01-23-2017
07:26 AM
@Kuldeep Kulkarni I've attached the yarn log file..the cluster was non HA initially then from ambari UI added HA namenode..all the services were up n running when it was a non HA cluster. Thanks in advance! yarnlog.txt
... View more
01-20-2017
12:57 PM
Mapreduce- History server not getting started changed a non ha cluster to ha cluster yarn logs says 2017-01-20 01:09:48,109 ERROR hs.JobHistoryServer (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2017-01-20 01:09:48,112 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping JobHistoryServer metrics system...
2017-01-20 01:09:48,112 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - JobHistoryServer metrics system stopped.
2017-01-20 01:09:48,113 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - JobHistoryServer metrics system shutdown complete.
2017-01-20 01:09:48,113 INFO ipc.Server (Server.java:stop(2725)) - Stopping server on 10033
2017-01-20 01:09:48,113 INFO ipc.Server (Server.java:run(913)) - Stopping IPC Server listener on 10033
2017-01-20 01:09:48,115 INFO ipc.Server (Server.java:run(1050)) - Stopping IPC Server Responder
2017-01-20 01:09:48,128 INFO ipc.Server (Server.java:stop(2725)) - Stopping server on 10020
2017-01-20 01:09:48,130 INFO ipc.Server (Server.java:run(913)) - Stopping IPC Server listener on 10020
2017-01-20 01:09:48,130 INFO ipc.Server (Server.java:run(1050)) - Stopping IPC Server Responder
2017-01-20 01:09:48,156 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:19888
2017-01-20 01:09:48,157 WARN http.HttpServer2 (HttpServer2.java:isRunning(527)) - HttpServer Acceptor: isRunning is false. Rechecking.
2017-01-20 01:09:48,157 WARN http.HttpServer2 (HttpServer2.java:isRunning(536)) - HttpServer Acceptor: isRunning is false
2017-01-20 01:09:48,158 INFO hs.JobHistory (JobHistory.java:serviceStop(146)) - Stopping JobHistory
2017-01-20 01:09:48,158 INFO hs.JobHistory (JobHistory.java:serviceStop(148)) - Stopping History Cleaner/Move To Done
2017-01-20 01:09:48,168 ERROR delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:run(659)) - ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2017-01-20 01:09:48,172 INFO hs.JobHistoryServer (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
... View more
01-13-2017
04:30 AM
@Dezka Dex @mqureshi I've been working on creating asymmetric cluster(decoupling storage and compute) with node labeling. Take a look this for my query. It would be really helpful if you could share the progress of your work.
Thanks in advance.
... View more
01-10-2017
09:19 AM
Followed this link https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_yarn_resource_mgt/content/configuring_node_labels.html
... View more
01-10-2017
09:17 AM
Hi, I would like to setup an asymmetric cluster with 6 datanodes. I labelled them as datanode1, datanode2, datanode3-compute datanode4, datanode5, datanode6-storage using this command yarn rmadmin -replaceLabelsOnNode "data1.aws.asymmetric.com:45454=compute data2.aws.asymmetric.com:45454=compute data3.aws.asymmetric.com:45454=compute data4.aws.asymmetric.com:45454=storage data5.aws.asymmetric.com:45454=storage data6.aws.asymmetric.com:45454=storage" then changed the yarn configuration, so that the queue(default) can be accessed only by the compute nodes yarn.scheduler.capacity.root.queues=default yarn.scheduler.capacity.root.default.capacity=100 yarn.scheduler.capacity.root.default.accessible-node-labels=compute yarn.scheduler.capacity.root.default.accessible-node-labels.compute.capacity=100 How to check if the compute nodes only performs computation and that storage nodes stores only the data.
... View more
Labels:
01-02-2017
05:07 AM
Hi, i've upgraded ambari from 2.2 to 2.4 and hdp to 2.5 hiveserver2 and history server are not getting started if i try to start history server from ambari i am getting the following error Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 190, in <module>
HistoryServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 101, in start
host_sys_prepped=params.host_sys_prepped)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 257, in copy_to_hdfs
replace_existing_files=replace_existing_files,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in action_create_on_execute
self.action_delayed("create")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 255, in action_delayed
self._create_resource()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 269, in _create_resource
self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 322, in _create_file
self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 179, in run_command
_, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.3.0-37/hadoop/mapreduce.tar.gz 'http://nnode.aws.atlas.bdaas.com:50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmp0SGg9U 2>/tmp/tmpf9vt5e' returned 52. curl: (52) Empty reply from server
100 after starting this server, the namenode automatically gets stopped.
... View more
Labels:
01-02-2017
05:03 AM
servererror.pngHi, The problem was that i did not add Hbase service to the cluster, also now when i try to add i am facing the following issues, hiveserver2 and history server are not getting started if i try to start history server from ambari i am getting the following error Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 190, in <module>
HistoryServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 101, in start
host_sys_prepped=params.host_sys_prepped)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 257, in copy_to_hdfs
replace_existing_files=replace_existing_files,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in action_create_on_execute
self.action_delayed("create")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 255, in action_delayed
self._create_resource()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 269, in _create_resource
self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 322, in _create_file
self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 179, in run_command
_, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.3.0-37/hadoop/mapreduce.tar.gz 'http://nnode.aws.atlas.bdaas.com:50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmp0SGg9U 2>/tmp/tmpf9vt5e' returned 52. curl: (52) Empty reply from server
100 after starting this server, the namenode automatically gets stopped.
... View more
12-22-2016
09:07 AM
Hi, I've been trying to add atlas 0.7 to ambari(2.4) cluster, the following configuration settings were asked to be filled in the customize service tab, atlas.audit.hbase.zookeeper.quorum atlas.graph.index.search.solr.zookeeper-url atlas.graph.storage.hostname I'm not sure what needs to be filled here..the services in the cluster includes HDFS,YARN,MAPREDUCE,TEZ,HIVE,PIG,ZOOKEEPER,RANGER,SLIDER in addition to that kafka was also required by atlas.. I'm getting an error saying "server error" when trying to add atlas, kafka and deploy the cluster.
... View more
12-20-2016
06:06 AM
Hi, I've upgraded ambari from 2.2 to 2.4. when ambari was started using "ambari-server start" got the error as follows, DB configs consistency check failed. Run "ambari-server start --skip-database-check" to skip. If you use this "--skip-database-check" option, do not make any changes to your cluster topology or perform a cluster upgrade until you correct the database consistency issues. See "/var/log/ambari-server/ambari-server-check-database.log" for more details on the consistency issues. Ambari database consistency check finished In /var/log/ambari-server/ambari-server-check-database.log 2016-12-20 00:33:28,188 INFO - ******************************* Check database started ******************************* 2016-12-20 00:33:47,630 INFO - Checking for configs not mapped to any cluster 2016-12-20 00:33:47,703 INFO - Checking for configs selected more than once 2016-12-20 00:33:47,762 INFO - Checking for hosts without state 2016-12-20 00:33:47,792 INFO - Checking host component states count equals host component desired states count 2016-12-20 00:33:47,826 INFO - Checking services and their configs 2016-12-20 00:34:00,127 ERROR - Required config(s): usersync-properties,ranger-site is(are) not available for service RANGER with service config version 3 in cluster atlas So instead started ambari using "ambari-server start --skip-database-check" and the it worked fine. In /var/log/ambari-server/ambari-server-check-database.log 2016-12-20 00:34:00,131 INFO - ******************************* Check database completed ******************************* 2016-12-20 00:35:26,917 INFO - Checking DB store version 2016-12-20 00:35:30,076 INFO - DB store version is compatible But if logged into the ambari page, "start services" option is in disabled mode. How to start the services.. Thanks!!
... View more
Labels: