Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.
abhishes
Rising Star

Permission denied: user=admin

by Rising Star in Support Questions
‎05-28-2018 09:00 PM
‎05-28-2018 09:00 PM
I installed my HDP Cluster without any errors. I got 100% success and I could see in Ambari that all services were started. But I notice that under user there is no directory for admin. (I am logged into amber as admin/admin). When I try to upload a file on HDFS I get error Permission denied: user=admin, access=WRITE, inode="/foo.py":hdfs:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736) I googled and found this https://community.hortonworks.com/questions/97703/hive-write-permission-denied.html But for me the solution above does not work because there is no directory called `/user/admin` Edit: I solved the problem with these two commands sudo -u hdfs hadoop fs -mkdir /user/admin sudo -u hdfs hadoop fs -chown -R admin:admin /user/admin After this I can work in these directories. but I am curios to know why out o the box the admin user has no permissions in my cluster (In the Hortonworks sandbox. admin user can work on all directories). ... View more
Labels:

Re: Hive 2 Server Not Staring

by Rising Star in Support Questions
‎05-28-2018 06:01 PM
‎05-28-2018 06:01 PM
when I try this command the setup doesn't complete sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/home/ubuntu/mysql-connector-java.jar Using python /usr/bin/python Setup ambari-server Copying /home/ubuntu/mysql-connector-java.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully It says setup is successful. but it doesn't do any setup. just comes out. when I do `ambari-server setup` then it runs the setup but after the configuration of the cluster many services don't start because they cannot find the mysql-connector-java.jar ... View more

Hive 2 Server Not Staring

by Rising Star in Support Questions
‎05-26-2018 05:59 PM
‎05-26-2018 05:59 PM
I completed the installation of a 4 node HDP cluster. Almost everything is up and running except for Hive Server. when I try to start the hive server. I get an error Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 161, in <module> HiveServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 77, in start self.configure(env) # FOR SECURITY File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 51, in configure hive(name='hiveserver2') File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 310, in hive jdbc_connector(params.hive_jdbc_target, params.hive_previous_jdbc_jar) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 527, in jdbc_connector content = DownloadSource(params.driver_curl_source)) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 123, in action_create content = self._get_content() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 160, in _get_content return content() File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 52, in __call__ return self.get_content() File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 197, in get_content raise Fail("Failed to download file from {0} due to HTTP error: {1}".format(self.url, str(ex))) resource_management.core.exceptions.Fail: Failed to download file from http://ip-172-31-44-126.us-west-2.compute.internal:8080/resources/mysql-connector-java.jar due to HTTP error: HTTP Error 404: Not Found ... View more
Labels:

HWP Registration of Hosts Failing

by Rising Star in Support Questions
‎05-26-2018 03:24 PM
‎05-26-2018 03:24 PM
Hello, I am trying to setup a cluster of HDP using Ambari on AWS. I have tried using HDP 2.6.5.0 and 2.6.4.0 In both cases I get an error INFO 2018-05-26 15:14:03,271 Controller.py:170 - Registering with ip-172-31-45-101.us-west-2.compute.internal (172.31.45.101) (agent=\'{"hardwareProfile": {"kernel": "Linux", "domain": "us-west-2.compute.internal", "physicalprocessorcount": 2, "kernelrelease": "4.14.33-51.37.amzn1.x86_64", "uptime_days": "0", "memorytotal": 8170572, "swapfree": "0.00 GB", "memorysize": 8170572, "osfamily": "redhat", "swapsize": "0.00 GB", "processorcount": 2, "netmask": "255.255.240.0", "timezone": "UTC", "hardwareisa": "x86_64", "memoryfree": 4256940, "operatingsystem": "amazon", "kernelmajversion": "4.14", "kernelversion": "4.14.33", "macaddress": "06:C2:18:6B:CE:E6", "operatingsystemrelease": "2018.03", "ipaddress": "172.31.45.101", "hostname": "ip-172-31-45-101", "uptime_hours": "1", "fqdn": "ip-172-31-45-101.us-west-2.compute.internal", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "4074696", "used": "56", "percent": "1%", "device": "devtmpfs", "mountpoint": "/dev", "type": "devtmpfs", "size": "4074752"}, {"available": "37499988", "used": "3551552", "percent": "9%", "device": "/dev/xvda1", "mountpoint": "/", "type": "ext4", "size": "41151788"}], "hardwaremodel": "x86_64", "uptime_seconds": "5676", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.6.2.0", "agentEnv": {"transparentHugePage": "", "hostHealth": {"agentTimeStampAtReporting": 1527347643270, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntpd or chronyd", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "iptables", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1527347643228, "hostname": "ip-172-31-45-101.us-west-2.compute.internal", "responseId": -1, "publicHostname": "ec2-35-160-48-195.us-west-2.compute.amazonaws.com"}\') INFO 2018-05-26 15:14:03,272 NetUtil.py:70 - Connecting to https://ip-172-31-45-101.us-west-2.compute.internal:8440/connection_info INFO 2018-05-26 15:14:03,327 security.py:93 - SSL Connect being called.. connecting to the server INFO 2018-05-26 15:14:03,380 security.py:60 - SSL connection established. Two-way SSL authentication is turned off on the server. ERROR 2018-05-26 15:14:03,383 Controller.py:190 - Cannot register host with not supported os type, hostname=ip-172-31-45-101.us-west-2.compute.internal, serverOsType=amazon2018, agentOsType=amazon2018 INFO 2018-05-26 15:14:03,383 Controller.py:516 - Registration response from ip-172-31-45-101.us-west-2.compute.internal was FAILED INFO 2018-05-26 15:14:03,383 Controller.py:531 - Registration response from %s didn\'t contain \'response\' as a key I googled and found this issue. https://issues.apache.org/jira/browse/AMBARI-21409?attachmentOrder=asc Is there no way to do the installation on the Amazon Linux AMI? ... View more
Labels:

Re: Query the Active Name Node from an Oozie Job

by Rising Star in Support Questions
‎12-09-2015 07:08 AM
‎12-09-2015 07:08 AM
OK. The cloudera manager API solution looks good   I tried   http://cloudera.mycompany.com/api/v9/clusters/cluster2/services/hdfs1/roles   The output is comprehensive... but it doesn't have the machine name anywhere.   { "name" : "hdfs1-NAMENODE-f307e0a1ebd0702da50cbfb68356cadf", "type" : "NAMENODE", "serviceRef" : { "clusterName" : "cluster2", "serviceName" : "hdfs1" }, "hostRef" : { "hostId" : "01008a9e-8b44-4e73-bc29-510fbe00b632" }, "roleUrl" : "http://clouderamgr01.mycompany.com:7180/cmf/roleRedirect/hdfs1-NAMENODE-f307e0a1ebd0702da50cbfb68356cadf", "roleState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "NAME_NODE_DATA_DIRECTORIES_FREE_SPACE", "summary" : "GOOD" }, { "name" : "NAME_NODE_DIRECTORY_FAILURES", "summary" : "GOOD" }, { "name" : "NAME_NODE_FILE_DESCRIPTOR", "summary" : "GOOD" }, { "name" : "NAME_NODE_HA_CHECKPOINT_AGE", "summary" : "DISABLED" }, { "name" : "NAME_NODE_HEAP_DUMP_DIRECTORY_FREE_SPACE", "summary" : "DISABLED" }, { "name" : "NAME_NODE_HOST_HEALTH", "summary" : "GOOD" }, { "name" : "NAME_NODE_JOURNAL_NODE_SYNC_STATUS", "summary" : "DISABLED" }, { "name" : "NAME_NODE_LOG_DIRECTORY_FREE_SPACE", "summary" : "GOOD" }, { "name" : "NAME_NODE_PAUSE_DURATION", "summary" : "GOOD" }, { "name" : "NAME_NODE_ROLLING_UPGRADE_STATUS", "summary" : "GOOD" }, { "name" : "NAME_NODE_RPC_LATENCY", "summary" : "DISABLED" }, { "name" : "NAME_NODE_SAFE_MODE", "summary" : "GOOD" }, { "name" : "NAME_NODE_SCM_HEALTH", "summary" : "GOOD" }, { "name" : "NAME_NODE_SWAP_MEMORY_USAGE", "summary" : "GOOD" }, { "name" : "NAME_NODE_UNEXPECTED_EXITS", "summary" : "GOOD" }, { "name" : "NAME_NODE_UPGRADE_STATUS", "summary" : "GOOD" }, { "name" : "NAME_NODE_WEB_METRIC_COLLECTION", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "haStatus" : "STANDBY", "maintenanceMode" : false, "maintenanceOwners" : [ ], "commissionState" : "COMMISSIONED", "roleConfigGroupRef" : { "roleConfigGroupName" : "hdfs1-NAMENODE-BASE" }   and    { "name" : "hdfs1-NAMENODE-e8e705638eaaa8233bd2729af511f874", "type" : "NAMENODE", "serviceRef" : { "clusterName" : "cluster2", "serviceName" : "hdfs1" }, "hostRef" : { "hostId" : "4127ba89-71a8-4a27-af75-51900f9f0a2e" }, "roleUrl" : "http://clouderamgr01.mycompany.com:7180/cmf/roleRedirect/hdfs1-NAMENODE-e8e705638eaaa8233bd2729af511f874", "roleState" : "STARTED", "healthSummary" : "GOOD", "healthChecks" : [ { "name" : "NAME_NODE_DATA_DIRECTORIES_FREE_SPACE", "summary" : "GOOD" }, { "name" : "NAME_NODE_DIRECTORY_FAILURES", "summary" : "GOOD" }, { "name" : "NAME_NODE_FILE_DESCRIPTOR", "summary" : "GOOD" }, { "name" : "NAME_NODE_HA_CHECKPOINT_AGE", "summary" : "GOOD" }, { "name" : "NAME_NODE_HEAP_DUMP_DIRECTORY_FREE_SPACE", "summary" : "DISABLED" }, { "name" : "NAME_NODE_HOST_HEALTH", "summary" : "GOOD" }, { "name" : "NAME_NODE_JOURNAL_NODE_SYNC_STATUS", "summary" : "GOOD" }, { "name" : "NAME_NODE_LOG_DIRECTORY_FREE_SPACE", "summary" : "GOOD" }, { "name" : "NAME_NODE_PAUSE_DURATION", "summary" : "GOOD" }, { "name" : "NAME_NODE_ROLLING_UPGRADE_STATUS", "summary" : "GOOD" }, { "name" : "NAME_NODE_RPC_LATENCY", "summary" : "GOOD" }, { "name" : "NAME_NODE_SAFE_MODE", "summary" : "GOOD" }, { "name" : "NAME_NODE_SCM_HEALTH", "summary" : "GOOD" }, { "name" : "NAME_NODE_SWAP_MEMORY_USAGE", "summary" : "GOOD" }, { "name" : "NAME_NODE_UNEXPECTED_EXITS", "summary" : "GOOD" }, { "name" : "NAME_NODE_UPGRADE_STATUS", "summary" : "GOOD" }, { "name" : "NAME_NODE_WEB_METRIC_COLLECTION", "summary" : "GOOD" } ], "configStalenessStatus" : "FRESH", "haStatus" : "ACTIVE", "maintenanceMode" : false, "maintenanceOwners" : [ ], "commissionState" : "COMMISSIONED", "roleConfigGroupRef" : { "roleConfigGroupName" : "hdfs1-NAMENODE-BASE" } The following is the screenshot from the cloudera manager UI. which clearly shows 02 as standby     If I take the hostId provided by the Json above and then do a    http://cloudera.mycompany.com/api/v9/clusters/cluster2/hosts/01008a9e-8b44-4e73-bc29-510fbe00b632   but this only gives me a blank page. So I cannot get machine name. ... View more

Re: Query the Active Name Node from an Oozie Job

by Rising Star in Support Questions
‎12-07-2015 10:31 PM
‎12-07-2015 10:31 PM
The reason why this solution will not work for me is because haadmin requires super user (HDFS Admin) permission to run. This is absolutely impossible for a remote machine to provide. ... View more

Query the Active Name Node from an Oozie Job

by Rising Star in Support Questions
‎12-07-2015 08:52 PM
‎12-07-2015 08:52 PM
I am using Cloudera 5.4.1. I have authored an oozie workflow. in one of the tasks, I need to use Oozie EL to generate a string which contains the URL of the "active" name node.   My cloudera cluster has active and secondary name nodes. At any point of time, I need to be able to dynamically query which is the active name node.   I have been looking around the internet but have found no answer. This is what I have thought   1. Use the SSH Oozie task to execute `hdfs getconfig -namenode` but the problem is that this command returns everything and not just the name of the active name node.   2. Use Cloudera Manager REST API to query the active name node. I have not seen any documentation or example which would allow me to do this.   If possible, please be kind enough to let me know how can I query and dynamically determine the active name node of a hadoop cluster ... View more
Labels:

Re: Not able to connect to cloudera manager admin ...

by Rising Star in Support Questions
‎06-05-2015 05:09 PM
‎06-05-2015 05:09 PM
OK. I solved it.   it was my age old nemisis iptables.   I did these 2 steps to solve the problem   service iptables stop chkconfig iptables off   now CM is appearning nice and easy in the browser. ... View more

Not able to connect to cloudera manager admin cons...

by Rising Star in Support Questions
‎06-05-2015 04:31 PM
‎06-05-2015 04:31 PM
I am installing cloudera manager on a vm on amazon aws cloud.   I followed the instructions given   Please see my security group and you can see that I have opened the required inbound traffic         In my logs I can see that the cloudera-scm-server is starting correctly without any errors. the last line which I see in my log is   2015-06-05 23:26:45,222 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4 2015-06-05 23:26:45,231 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 2015-06-05 23:26:45,231 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 2015-06-05 23:26:45,241 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:126 2015-06-05 23:26:45,241 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2015-06-05T23:26:45.241Z 2015-06-05 23:26:46,129 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2015-06-05T23:26:46.129Z   and yet when I put the public dns name of my VM http://<<public dns>>:7180/     it does not connect and just says "this web page is not available"   ... View more
Labels:

White Elephant with Clouera?

by Rising Star in Support Questions
‎01-16-2015 06:23 PM
‎01-16-2015 06:23 PM
I found a very good tool called White Elephant   http://data.linkedin.com/opensource/white-elephant   It looks like the only tool to perform capacity planning on Hadoop clusters.   I want to know if anyone can confirm if this tool would work on CDH 5.2.1 ... View more
Contact Me
Online
Offline
Last Visited
‎08-16-2017 08:06 PM
Public Statistics
Date Registered ‎07-15-2014 11:42 AM
Last Visited ‎08-16-2017 08:06 PM
Total Messages Posted 57
Total Kudos Received 10