Member since
07-15-2014
57
Posts
9
Kudos Received
6
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 6730 | 06-05-2015 05:09 PM | |
| 1621 | 12-19-2014 01:03 PM | |
| 3158 | 12-17-2014 08:23 PM | |
| 8091 | 12-16-2014 03:07 PM | |
| 13597 | 08-30-2014 11:14 PM |
05-28-2018
09:00 PM
I installed my HDP Cluster without any errors. I got 100% success and I could see in Ambari that all services were started. But I notice that under user there is no directory for admin. (I am logged into amber as admin/admin). When I try to upload a file on HDFS I get error Permission denied: user=admin, access=WRITE, inode="/foo.py":hdfs:hdfs:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736) I googled and found this https://community.hortonworks.com/questions/97703/hive-write-permission-denied.html But for me the solution above does not work because there is no directory called `/user/admin` Edit: I solved the problem with these two commands sudo -u hdfs hadoop fs -mkdir /user/admin sudo -u hdfs hadoop fs -chown -R admin:admin /user/admin After this I can work in these directories. but I am curios to know why out o the box the admin user has no permissions in my cluster (In the Hortonworks sandbox. admin user can work on all directories).
... View more
Labels:
- Labels:
-
Apache Hadoop
05-28-2018
06:01 PM
when I try this command the setup doesn't complete sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/home/ubuntu/mysql-connector-java.jar Using python /usr/bin/python
Setup ambari-server
Copying /home/ubuntu/mysql-connector-java.jar to /var/lib/ambari-server/resources
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully It says setup is successful. but it doesn't do any setup. just comes out. when I do `ambari-server setup` then it runs the setup but after the configuration of the cluster many services don't start because they cannot find the mysql-connector-java.jar
... View more
05-26-2018
05:59 PM
I completed the installation of a 4 node HDP cluster. Almost everything is up and running except for Hive Server. when I try to start the hive server. I get an error Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 161, in <module>
HiveServer().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 77, in start
self.configure(env) # FOR SECURITY
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 120, in locking_configure
original_configure(obj, *args, **kw)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 51, in configure
hive(name='hiveserver2')
File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 310, in hive
jdbc_connector(params.hive_jdbc_target, params.hive_previous_jdbc_jar)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 527, in jdbc_connector
content = DownloadSource(params.driver_curl_source))
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 123, in action_create
content = self._get_content()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 160, in _get_content
return content()
File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 52, in __call__
return self.get_content()
File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 197, in get_content
raise Fail("Failed to download file from {0} due to HTTP error: {1}".format(self.url, str(ex)))
resource_management.core.exceptions.Fail: Failed to download file from http://ip-172-31-44-126.us-west-2.compute.internal:8080/resources/mysql-connector-java.jar due to HTTP error: HTTP Error 404: Not Found
... View more
Labels:
- Labels:
-
Apache Hive
12-09-2015
07:08 AM
OK. The cloudera manager API solution looks good I tried http://cloudera.mycompany.com/api/v9/clusters/cluster2/services/hdfs1/roles The output is comprehensive... but it doesn't have the machine name anywhere. {
"name" : "hdfs1-NAMENODE-f307e0a1ebd0702da50cbfb68356cadf",
"type" : "NAMENODE",
"serviceRef" : {
"clusterName" : "cluster2",
"serviceName" : "hdfs1"
},
"hostRef" : {
"hostId" : "01008a9e-8b44-4e73-bc29-510fbe00b632"
},
"roleUrl" : "http://clouderamgr01.mycompany.com:7180/cmf/roleRedirect/hdfs1-NAMENODE-f307e0a1ebd0702da50cbfb68356cadf",
"roleState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "NAME_NODE_DATA_DIRECTORIES_FREE_SPACE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_DIRECTORY_FAILURES",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_FILE_DESCRIPTOR",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_HA_CHECKPOINT_AGE",
"summary" : "DISABLED"
}, {
"name" : "NAME_NODE_HEAP_DUMP_DIRECTORY_FREE_SPACE",
"summary" : "DISABLED"
}, {
"name" : "NAME_NODE_HOST_HEALTH",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_JOURNAL_NODE_SYNC_STATUS",
"summary" : "DISABLED"
}, {
"name" : "NAME_NODE_LOG_DIRECTORY_FREE_SPACE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_PAUSE_DURATION",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_ROLLING_UPGRADE_STATUS",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_RPC_LATENCY",
"summary" : "DISABLED"
}, {
"name" : "NAME_NODE_SAFE_MODE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_SCM_HEALTH",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_SWAP_MEMORY_USAGE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_UNEXPECTED_EXITS",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_UPGRADE_STATUS",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_WEB_METRIC_COLLECTION",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"haStatus" : "STANDBY",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"commissionState" : "COMMISSIONED",
"roleConfigGroupRef" : {
"roleConfigGroupName" : "hdfs1-NAMENODE-BASE"
} and {
"name" : "hdfs1-NAMENODE-e8e705638eaaa8233bd2729af511f874",
"type" : "NAMENODE",
"serviceRef" : {
"clusterName" : "cluster2",
"serviceName" : "hdfs1"
},
"hostRef" : {
"hostId" : "4127ba89-71a8-4a27-af75-51900f9f0a2e"
},
"roleUrl" : "http://clouderamgr01.mycompany.com:7180/cmf/roleRedirect/hdfs1-NAMENODE-e8e705638eaaa8233bd2729af511f874",
"roleState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "NAME_NODE_DATA_DIRECTORIES_FREE_SPACE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_DIRECTORY_FAILURES",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_FILE_DESCRIPTOR",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_HA_CHECKPOINT_AGE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_HEAP_DUMP_DIRECTORY_FREE_SPACE",
"summary" : "DISABLED"
}, {
"name" : "NAME_NODE_HOST_HEALTH",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_JOURNAL_NODE_SYNC_STATUS",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_LOG_DIRECTORY_FREE_SPACE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_PAUSE_DURATION",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_ROLLING_UPGRADE_STATUS",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_RPC_LATENCY",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_SAFE_MODE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_SCM_HEALTH",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_SWAP_MEMORY_USAGE",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_UNEXPECTED_EXITS",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_UPGRADE_STATUS",
"summary" : "GOOD"
}, {
"name" : "NAME_NODE_WEB_METRIC_COLLECTION",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"haStatus" : "ACTIVE",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"commissionState" : "COMMISSIONED",
"roleConfigGroupRef" : {
"roleConfigGroupName" : "hdfs1-NAMENODE-BASE"
} The following is the screenshot from the cloudera manager UI. which clearly shows 02 as standby If I take the hostId provided by the Json above and then do a http://cloudera.mycompany.com/api/v9/clusters/cluster2/hosts/01008a9e-8b44-4e73-bc29-510fbe00b632 but this only gives me a blank page. So I cannot get machine name.
... View more
12-07-2015
10:31 PM
The reason why this solution will not work for me is because haadmin requires super user (HDFS Admin) permission to run. This is absolutely impossible for a remote machine to provide.
... View more
12-07-2015
08:52 PM
I am using Cloudera 5.4.1. I have authored an oozie workflow. in one of the tasks, I need to use Oozie EL to generate a string which contains the URL of the "active" name node. My cloudera cluster has active and secondary name nodes. At any point of time, I need to be able to dynamically query which is the active name node. I have been looking around the internet but have found no answer. This is what I have thought 1. Use the SSH Oozie task to execute `hdfs getconfig -namenode` but the problem is that this command returns everything and not just the name of the active name node. 2. Use Cloudera Manager REST API to query the active name node. I have not seen any documentation or example which would allow me to do this. If possible, please be kind enough to let me know how can I query and dynamically determine the active name node of a hadoop cluster
... View more
Labels:
- Labels:
-
Apache Oozie
06-05-2015
05:09 PM
OK. I solved it. it was my age old nemisis iptables. I did these 2 steps to solve the problem service iptables stop chkconfig iptables off now CM is appearning nice and easy in the browser.
... View more
06-05-2015
04:31 PM
I am installing cloudera manager on a vm on amazon aws cloud. I followed the instructions given Please see my security group and you can see that I have opened the required inbound traffic In my logs I can see that the cloudera-scm-server is starting correctly without any errors. the last line which I see in my log is 2015-06-05 23:26:45,222 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4 2015-06-05 23:26:45,231 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 2015-06-05 23:26:45,231 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 2015-06-05 23:26:45,241 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:126 2015-06-05 23:26:45,241 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2015-06-05T23:26:45.241Z 2015-06-05 23:26:46,129 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2015-06-05T23:26:46.129Z and yet when I put the public dns name of my VM http://<<public dns>>:7180/ it does not connect and just says "this web page is not available"
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Security
01-16-2015
06:23 PM
I found a very good tool called White Elephant http://data.linkedin.com/opensource/white-elephant It looks like the only tool to perform capacity planning on Hadoop clusters. I want to know if anyone can confirm if this tool would work on CDH 5.2.1
... View more
Labels:
01-06-2015
11:56 PM
I looked at the URL which you gave me and its still very thin for Service providers. it just says Service providers deliver Apache Hadoop-based solutions and services to end users interested in leveraging Cloudera’s training resources, support and brand. * How is the support offered to a partner any different from normal cludera support which I get if I purchase Data Hub Edition? * Are there regular training events? or just 1 training which you give at the time of signing up the partner agreement? If there are perioidic trainings then is there a schedule for these partner trainings? * Can you eloborate on what "brand" means above?
... View more