About java_etl_bigdat

java_etl_bigdat · ‎02-28-2017

java_etl_bigdat · ‎02-17-2017

Hi, I am using HDP 2.5 sandbox, installed Ranger KMS by using following link http://coheigea.blogspot.in/2016/08/installing-apache-ranger-key-management.html https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+Installation While i am trying to create a service in Ranger KMS getting "Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin, connection refused" in cat /var/log/ranger/admin/xa_portal.log file 2017-02-17 09:46:20,870 [http-bio-6080-exec-6] INFO org.apache.ranger.security.listener.SpringEventListener (SpringEventListener.java:71) - Login Successful:keyadmin | Ip Address:10.77.217.76 | sessionId=AEF3F945241899271625CBFE1FCDDCC3 | Epoch=1487324780870 2017-02-17 09:46:20,871 [http-bio-6080-exec-6] INFO org.apache.ranger.biz.SessionMgr (SessionMgr.java:429) - keyadmin is a valid user 2017-02-17 09:46:21,190 [http-bio-6080-exec-6] INFO org.apache.ranger.biz.SessionMgr (SessionMgr.java:232) - UserSession Updated to set new Permissions to User: keyadmin 2017-02-17 09:46:21,191 [http-bio-6080-exec-6] INFO org.apache.ranger.biz.SessionMgr (SessionMgr.java:184) - Login Success: loginId=keyadmin, sessionId=371, sessionId=AEF3F945241899271625CBFE1FCDDCC3, requestId=10.77.217.76, epoch=1487324781191 2017-02-17 09:46:35,464 [timed-executor-pool-0] ERROR apache.ranger.services.kms.client.KMSClient (KMSClient.java:285) - Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused (Connection refused) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:131) at com.sun.jersey.api.client.Client.handle(Client.java:616) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:559) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:72) at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:454) at org.apache.ranger.services.kms.client.KMSClient.getKeyList(KMSClient.java:177) at org.apache.ranger.services.kms.client.KMSClient.getKmsKey(KMSClient.java:382) at org.apache.ranger.services.kms.client.KMSClient.testConnection(KMSClient.java:317) at org.apache.ranger.services.kms.client.KMSResourceMgr.validateConfig(KMSResourceMgr.java:41) at org.apache.ranger.services.kms.RangerServiceKMS.validateConfig(RangerServiceKMS.java:55) at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560) at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547) at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:218) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:129) ... 16 more 2017-02-17 09:46:35,465 [timed-executor-pool-0] ERROR apache.ranger.services.kms.client.KMSResourceMgr (KMSResourceMgr.java:43) - <== KMSResourceMgr.validateConfig Error: org.apache.ranger.plugin.client.HadoopException: Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin 2017-02-17 09:46:35,465 [timed-executor-pool-0] ERROR org.apache.ranger.services.kms.RangerServiceKMS (RangerServiceKMS.java:57) - <== RangerServiceKMS.validateConfig Error:org.apache.ranger.plugin.client.HadoopException: Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin 2017-02-17 09:46:35,465 [timed-executor-pool-0] ERROR org.apache.ranger.biz.ServiceMgr$TimedCallable (ServiceMgr.java:510) - TimedCallable.call: Error:org.apache.ranger.plugin.client.HadoopException: Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin 2017-02-17 09:46:35,466 [http-bio-6080-exec-5] ERROR org.apache.ranger.biz.ServiceMgr (ServiceMgr.java:188) - ==> ServiceMgr.validateConfig Error:org.apache.ranger.plugin.client.HadoopException: org.apache.ranger.plugin.client.HadoopException: Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin 2017-02-17 09:46:54,424 [http-bio-6080-exec-1] INFO org.apache.ranger.biz.ServiceDBStore (ServiceDBStore.java:2503) - Creating default KMS policy item for hdfs 2017-02-17 09:46:54,430 [http-bio-6080-exec-1] INFO org.apache.ranger.biz.ServiceDBStore (ServiceDBStore.java:2518) - Creating default KMS policy item for hive [root@sandbox ambari]# Connection Failed. Unable to retrieve any files using given parameters, You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check ranger_admin.log for more info. org.apache.ranger.plugin.client.HadoopException: Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin. Exception while getting Kms Key List. URL : http://localhost:9292/kms/v1/keys/names?user.name=keyadmin. java.net.ConnectException: Connection refused (Connection refused). Connection refused (Connection refused). Can anyone please help me to fix the issue. install-properties.txtservice.png

java_etl_bigdat · ‎02-13-2017

Hi, I am trying to install Ranger KMS in HDP 2.5 sandbox using following link. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Ranger_KMS_Admin_Guide/content/ch02s01.html Getting following error message in ambari-agent.log file. AgentException: 'Script /var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py does not exist' WARNING 2017-02-13 11:48:03,902 FileCache.py:168 - Error occurred during cache update. Error tolerate setting is set to true, so ignoring this error and continuing with current cache. Error details: ('Can not invalidate cache directory {0}: {1}', '/var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package', "[Errno 22] Invalid argument: '/var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package/scripts'") ERROR 2017-02-13 11:48:03,903 CustomServiceOrchestrator.py:232 - Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py does not exist' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/CustomServiceOrchestrator.py", line 157, in runCommand script_path = self.resolve_script_path(base_dir, script) File "/usr/lib/python2.6/site-packages/ambari_agent/CustomServiceOrchestrator.py", line 305, in resolve_script_path raise AgentException(message) AgentException: 'Script /var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py does not exist' WARNING 2017-02-13 11:48:03,903 ActionQueue.py:554 - 'str' object has no attribute 'has_key' Can anyone please help me.

java_etl_bigdat · ‎02-13-2017

Hi @khadeer mhmd, Have you installed Ranger KMS in HDP2.5 sandbox? I am getting below error while adding Ranger KMS service in Ambari (HDP 2.5 sandbox). Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py does not exist' Could anyone please help me on this issue.

java_etl_bigdat · ‎01-29-2017

Getting below error. [root@sandbox /]# sudo rm -rf /var/lib/pgsql/* rm: cannot remove `/var/lib/pgsql/data/pg_stat_tmp': Invalid argument rm: cannot remove `/var/lib/pgsql/data/global': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_xlog': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_clog': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/1': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/11564': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/16384': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/11563': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_log': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_multixact/members': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_multixact/offsets': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_subtrans': Invalid argument

java_etl_bigdat · ‎01-29-2017

Hi, I am getting below log message while trying to start ambari server in HDP 2.5. [root@sandbox ~]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... No errors were found. ERROR: Exiting with exit code 1. REASON: Database check failed to complete. Please check /var/log/ambari-server/ambari-server.log and /var/log/ambari-server/ambari-server-check-database.log for more information. ambari-server-check-database.txt ambari-server.txt Then I search the root cause of issue in Google and found following solution. Try deleting /var/lib/pgsql and restart ambari server. It worked for me However i unable to delete the /var/lib/pgsql directory; [root@sandbox ~]# sudo rmdir /var/lib/pgsql/* rmdir: failed to remove `/var/lib/pgsql/data': Directory not empty Can anyone please help me to solve the above ambari-server issue and to delete pgsql directory. Thanks,

java_etl_bigdat · ‎07-11-2016

Hi, I am trying to fetch data from hive table (ExecuteSQL) and load into file system(PutFile). I have kept all required hive jar in lib folder of nifi. However getting Error message in ExecuteSQL processor. Please let me know if i have done any mistake anywhere and help me to complete the job.

java_etl_bigdat · ‎07-11-2016

As suggested by @Jobin George , I have added PutSQL processor and it inserted only one record. However i have 100 records are coming from ExecuteSql processor( contains select query: select * from table_name) and want to insert 100 records. Please help me if any configuration required for this requirement.

java_etl_bigdat · ‎07-08-2016

Hi, I want to load data from hive table to MySql table using HDF (nifi). I have created the following dataflow. After execution of dataflow, it is showing 151 bytes data has written the destination table. However there is no data in the destination table. Please help me if there is any configuration issue.

Online	Offline
Last Visited	‎03-01-2017 01:42 PM

Date Registered	‎07-08-2016 12:21 PM
Last Visited	‎03-01-2017 01:42 PM
Total Messages Posted	12
Total Kudos Received	1

How atlas export metadata into downstream systems?...

Exception while getting Kms Key List in Ranger KMS

Fail to install Ranger KMS in HDP2.5 sandbox

Re: Ranger KMS Tutorial

Re: Unable to start Ambari server in HDP 2.5

Unable to start Ambari server in HDP 2.5

Unable to fetch data from hive table using Apache ...

Re: Unable to insert data in MySQL using ConvertJS...

Unable to insert data in MySQL using ConvertJSONTo...