Bug : https://issues.apache.org/jira/browse/AMBARI-14466 [root@phdns01 ~]# ambari-server start Using python /usr/bin/python2 Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... WARNING: setpgid(31734, 0) failed - [Errno 13] Permission denied Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... Ambari Server 'start' completed successfully. [root@phdns01 ~]# wget https://issues.apache.org/jira/secure/attachment/12779059/AMBARI-14466.patch patch -p1 < AMBARI-14466.patch File to patch: /usr/sbin/ambari_server_main.py Issue resolved. ... View more

Use case: There is lot of data in the locally managed table and we want to convert those table into external table because we are working on a use case where our spark and home grown application has trouble reading locally managed tables. Solution: alter table table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); Test: Locally managed table dropped hive> drop table sample_07; OK Time taken: 0.602 seconds Data deleted as it's locally managed table. hive> dfs -ls /apps/hive/warehouse/sample_07; ls: `/apps/hive/warehouse/sample_07': No such file or directory Command failed with exit code = 1 Query returned non-zero code: 1, cause: null Converting sample_07_test from locally managed to external to keep the data there after drop. hive> alter table sample_07_test SET TBLPROPERTIES('EXTERNAL'='TRUE'); OK Time taken: 0.463 seconds Table dropped after converting into external table. hive> drop table sample_07_test; OK Time taken: 0.552 seconds Data is still on HDFS hive> dfs -ls /apps/hive/warehouse/sample_07_test; Found 1 items -rwxrwxrwx 3 hive hdfs 46059 2016-02-10 03:24 /apps/hive/warehouse/sample_07_test/000000_0 hive> Table removed from HCatalog hive> select * from sample_07_test; FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'sample_07_test' hive> ... View more

Customer: Is Hadoop Enterprise Ready? Me: Standing next to the whiteboard, Yes and that's why we use the term "Enterprise Ready Data Lake" Imagine that there are 3 points Point 1 -> You need to prove your identity to get access to Lake and then need permissions or authority to access data. Point 2 -> Once you proved your authenticity then demands comes to manage the lifecycle of data from it's requirement to retirement "Automated process" Point 3 -> Life Cycle Management process needs to be integrated with a Governance solution to manage data of data "metadata" , data lineage, auditing and more to fullfil security and compliance requirement. Point 1 --> Entry Point: You must have strong Authentication in place to get into the system and more users will be coming in to access data as we move away from silos of data to a centralized repository. The access management must be easier to manage i,e Security solution should have a centralized place toAdmin (create, define and manage) security policies. Once users gets in and has access then we need to track their actions and that's Auditing. At last, Data Encryption in motion & at rest Point 2 --> Security is place and now we know that Data ingestion is occurring with full security. Now, business wants to manage the lifecycle of data in one common place "Data replication, retention, handling late data arrival rules, data mirroring and visualize the complete data pipeline" Point 3 --> Once data lifecycle management in place then we will be generating more data of data "metadata" and there is existing legacy metadata that need to be exchange with Hadoop system. This generates the requirement of Data Governance solution. This solution should provide complete data lineage, exchange, search functionality Customer: Yes, this is exactly what we are looking for. All this must be well integrated and please provide this as 100% open source but enterprise ready solution. Solution: Security Data Lifecycle Management Data Governance Happy Hadooping!!! Kerberos is must in production ... View more

Problem: File"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140,in _call_wrapper result = _call(command,**kwargs_copy) File"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291,in _call raiseFail(err_msg) resource_management.core.exceptions.Fail:Execution of 'yarn resourcemanager -format-state-store' returned 255.15/10/2616:11:16 INFO resourcemanager.ResourceManager: STARTUP_MSG: 15/10/2616:11:17 INFO recovery.ZKRMStateStore: org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread thread interrupted!Exiting! 15/10/2616:11:17 INFO zookeeper.ZooKeeper:Session:0x150a4b3429b0002 closed 15/10/2616:11:17 FATAL resourcemanager.ResourceManager:Error starting ResourceManager org.apache.zookeeper.KeeperException$NotEmptyException:KeeperErrorCode=Directorynot empty for/rmstore/ZKRMStateRoot/RMAppRoot at org.apache.zookeeper.KeeperException.create(KeeperException.java:125) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.recursiveDeleteWithRetriesHelper(ZKRMStateStore.java:1049) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.recursiveDeleteWithRetriesHelper(ZKRMStateStore.java:1045) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.access$500(ZKRMStateStore.java:89) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$10.run(ZKRMStateStore.java:1032) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$10.run(ZKRMStateStore.java:1029) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1104) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1125) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.deleteWithRetries(ZKRMStateStore.java:1029) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.deleteStore(ZKRMStateStore.java:825) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.deleteRMStateStore(ResourceManager.java:1267) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1190) 15/10/2616:11:17 INFO zookeeper.ClientCnxn:EventThread shut down 15/10/2616:11:17 INFO resourcemanager.ResourceManager: SHUTDOWN_MSG: Solution: Error details: FATAL resourcemanager.ResourceManager:Error starting ResourceManager org.apache.zookeeper.KeeperException$NotEmptyException:KeeperErrorCode=Directory not empty for /rmstore/ZKRMStateRoot/RMAppRoot Please see this. In my case, I have all the application data sitting under that particular location [zk: localhost:2181(CONNECTED) 2] ls /rmstore/ZKRMStateRoot/RMAppRoot [application_1445593412630_0002, application_1445593412630_0001, application_1445366030467_0002, application_1445366030467_0001, application_1445366030467_0004, application_1445366030467_0003, application_1445593412630_0006, application_1445366030467_0005, application_1445593412630_0005, application_1445593412630_0004, application_1445593412630_0003, application_1445173693339_0006, application_1445173693339_0005, application_1445173693339_0004, application_1445173693339_0003, application_1445173693339_0002, application_1445173693339_0001, application_1445394313024_0004, application_1445394313024_0003, application_1445394313024_0002, application_1445394313024_0001, application_1445394313024_0008, application_1445394313024_0007, application_1445394313024_0006, application_1445394313024_0005] [zk: localhost:2181(CONNECTED) 3] quit Quitting... [zk: localhost:2181(CONNECTED) 3] rmr /rmstore/ZKRMStateRoot/RMAppRoot [zk: localhost:2181(CONNECTED) 4] ls /rmstore/ZKRMStateRoot/RMAppRoot Node does not exist: /rmstore/ZKRMStateRoot/RMAppRoot Restart Yarn and I got the location back [zk: localhost:2181(CONNECTED) 6] ls /rmstore/ZKRMStateRoot/RMAppRoot [] [zk: localhost:2181(CONNECTED) 7] [zk: localhost:2181(CONNECTED) 7] ls /rmstore/ZKRMStateRoot [AMRMTokenSecretManagerRoot, RMAppRoot, EpochNode, RMDTSecretManagerRoot, RMVersionNode] [zk: localhost:2181(CONNECTED) 8] You can try this but if you are not sure or its prod then open support ticket. "Consult support before doing this in production" ... View more