About dvt_isoft

dvt_isoft · ‎09-29-2017

Ok I have the solution. We were using ODBC via OTL and so we needed to set a particular flag "ImplicitSelect" to "true" directly on the connection level. When this flag is not set, "SHOW" queries return empty results. Thanks, Sylvain.

dvt_isoft · ‎09-29-2017

No, I'm using my own software based on Hortonworks ODBC driver (SHOW queries used to work without kerberos).

dvt_isoft · ‎09-28-2017

Hi, I'm facing the following problem : When I run SHOW queries through HortonWorks ODBC Driver with my kerberos-enable cluster, it returns an empty result (there is no error, just an empty result). However, I can perform SELECT queries, I can also run CREATE DATABASE or CREATE TABLE queries. Note : I can perform SHOW queries using beeline client. Note 2 : I tried to activate TRACE logging for the driver, and I get one error : "Sep 28 18:48:03.508 ERROR 2741458688 Connection::SQLSetConnectAttr: [Hortonworks][ODBC] (11470) Transactions are not supported." Any idea ? Thanks in advance, Sylvain.

dvt_isoft · ‎09-12-2017

I'm using a tomcat service with the Authentication filter defined as I found in this documentation. https://hadoop.apache.org/docs/stable/hadoop-auth/Configuration.html I have defined a service.xml and rewrite.xml : <service role="MyService" name="HTTPListener" version="0.0.1"> <routes> <route path="/HTTPListener/?**"/> <route path="/HTTPListener/**?**"/> </routes> </service> <rules> <rule dir="IN" name="MyService/HTTPListener/path/inbound" pattern="*://*:*/**/HTTPListener/{path=**}?{**}"> <rewrite template="{$serviceUrl[MyService]}/{path=**}?{**}"/> </rule> <rule dir="IN" name="MyService/HTTPListener/path2/inbound" pattern="*://*:*/**/HTTPListener/?{**}"> <rewrite template="{$serviceUrl[MyService]}/?{**}"/> </rule> </rules> It works, but I'm trying to re-use the HTTP connection for performance reasons. Without knox, I can see the connection is properly reuse. With Knox, the client use the same http connection, but I can see in gateway.log a lot of checks : 2017-09-12 15:34:09,939 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:10,955 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:11,997 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:12,258 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:12,276 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:13,288 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:14,301 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:15,320 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:16,334 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:17,289 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:17,323 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:17,446 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:17,459 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:18,472 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:19,490 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:19,531 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:19,539 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2017-09-12 15:34:20,560 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true And I can see on my servlet and with tcpdump that there is renegotiations on each call. I don't understand why the token is not kept by knox to prevent renegociation. Any idea ? Thanks in advance, Alex

dvt_isoft · ‎07-12-2017

Thanks. I was just misunderstanding the meaning of the property "Reserved space for HDFS". I actually thought it was the disk space we set for file storage ...

dvt_isoft · ‎07-03-2017

Hi, I am facing an alert while starting my services on ambari : HDFS tells me that it has no more disk space available, even though I haven't right anything on HDFS yet. The alert is the following : Capacity Used:[100%, 36864], Capacity Remaining:[0] The thing is that I have set 300000000000 bytes of capacity for HDFS (property "Reserved space for HDFS"), so I can't see where my problem is ... Thanks in advance for your answer ! Sylvain.

dvt_isoft · ‎06-30-2017

It looks like it was indeed. I killed the process and I did not get the error anymore. Thanks.

dvt_isoft · ‎06-26-2017

Hi, I juste installed a 4-nodes Hadoop cluster, and I can't start the services because the start-up fails at starting NFSGateway on my 4-th node, where it throws : resource_management.core.exceptions.ExecutionFailed The whole error is : Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/nfsgateway.py", line 89, in <module> NFSGateway().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/nfsgateway.py", line 58, in start nfsgateway(action="start") File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_nfsgateway.py", line 74, in nfsgateway create_log_dir=True File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 274, in service Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh -H -E /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start nfs3' returned 1. nfs3 running as process 3138. Stop it first. Thanks in advance for your answer ! Sylvain.

dvt_isoft · ‎03-30-2017

Hi @Predrag Minovic and @ssathish, I tried with WebHCat, it works. I already tried with the Distributed Shell application, but I didn't success. Maybe I did wrong. But I think I will try with Oozie soon. Thanks for you help

dvt_isoft · ‎03-30-2017

I just have another question : do Temporary table have better performances than normal tables ?

dvt_isoft · ‎03-29-2017

Yes I can connect to Hive with ODBC driver. The problem was that a new session was generated at each call from my client. Thanks, Sylvain.

dvt_isoft · ‎03-29-2017

Thank you. As you said, the client I am using is generating a new session at each call, that was the problem. Sylvain.

dvt_isoft · ‎03-28-2017

My cluster is not kerberized and I do not have ranger policies for Hive.

dvt_isoft · ‎03-28-2017

Yes, the tables created via ODBC are showing up in beeline. Creating a database via ODBC works too. The problem seems to be only appearing with temporary tables.

dvt_isoft · ‎03-28-2017

Thank you mliem, I am able to access hive through beeline. I can create my temporary table, and see it appearing with a "show tables" command using beeline. I tried creating my temporary table in the default database, and in a specified one, but I get the same result. Sylvain.

dvt_isoft · ‎03-28-2017

Hi, I am facing the following problem : Using the Hortonworks ODBC driver for Hive, I can create a temporary table (hiveserver2.log tell me that it is done without error), but I can't retrieve this table with a SELECT query thereafter. When I try to do so, I get a "table not found" error. Does it mean the ODBC driver does not keep the session between 2 hive calls ? Is there something I am doing wrong or some configuration I need to set ? I believe since hive 0.14, the "temporary table" part in the ODBC driver configuration is not necessary, or is it ? Thanks in advance, Sylvain.

dvt_isoft · ‎03-27-2017

Thanks @ssathish, here are my log files.

dvt_isoft · ‎03-23-2017

I forgot this : Ambari version : Version 2.4.2.0 Hadoop version 2.5.0.0-1245 installed from Ambari (not a sandbox)

dvt_isoft · ‎03-22-2017

Hello, I'm trying to execute some existing examples using the Rest API (with or without using the Knox gateway) It seems to work, but the task is always marked as failed in the Yarn Web UI. I Use the hadoop-mapreduce-examples.jar to launch a wordcount example. It creates a sub task which is properly executed, it creates all the expected files in /user/yarn/output10. But the parent task always failed with an exit code = 0 ( looks like a success 😞 , but it's not) I send a POST request on this URL (I did the new-application command before to get my application-id) https://guest:guest-password@serverip:8443/gateway/default/resourcemanager/v1/cluster/apps (I tried also without knox : http://serverip:8088/ws/v1/cluster/apps, with the same result) And I transmit this JSON command : { "application-id":"application_1490174038938_0018", "application-name":"WordCount", "am-container-spec": { "commands": { "command":"yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount /user/yarn/input /user/yarn/output10" } }, "max-app-attempts":1, "application-type":"YARN" } I tried a lot of other JSON config with the same result (this one is the simpliest I tried) Do you have any idea what I did wrong ? Thanks in advance

dvt_isoft · ‎03-20-2017

It should be alright now.

dvt_isoft · ‎03-20-2017

Hi, I am running Hadoop on a 3 nodes cluster (3 virtual machines) with respectively 20Gb, 10Gb and 10Gb of disk space available. When I run this command on the namenode : hadoop fs -df -h / I get the following result : When I run this command : hadoop fs -du -s -h / I get the following result : Knowing that the replication number is set to 3, shouldn't I get 3*2,7 = 8,1G in the first screenshot ? I tried to execute expunge command and it did not change the result. Thanks in advance ! Sylvain.

dvt_isoft · ‎03-16-2017

Hi, I don't understand what is happenning. I am desperately trying to make work a simple "SHOW VIEWS" command and I always get a ParseException error (cannot recognize input near 'show' 'views'). I believe the command must not return an error even if no view were created ? Anyway, I created a view juste in case. Here are the command I tried and that returned me ParseException errors: SHOW VIEWS; SHOW VIEWS IN mydatabase; SHOW VIEWS FROM mydatabase; Very simple ones. I am running a 2.4 version of Hive so I believe it is not a version issue. Thanks in advance ! Sylvain.

dvt_isoft · ‎03-16-2017

Thank you, it seems like TBLPROPERTIES don't like french characters ...

dvt_isoft · ‎03-15-2017

Hi, I am trying to set custom properties (using "ALTER TABLE ... SET TBLPROPERTIES ..." command) that contains special characters, like 'ç' or 'é'. The problem is I am getting '\u00e7' and '\u00e9' as a result when I execute a "DESCRIBE TABLE ... FORMATTED" command. Is there a way to get the proper encoding in return ? Here is my command : ALTER TABLE mydatabase.mytable SET TBLPROPERTIES ('Test'='François'); Here is what I am getting as a result with a DESCRIBE TABLE ... FORMATTED command : Test Fran\\u00e7ois Thanks in advance for your answer ! Sylvain.

Online	Offline
Last Visited	‎06-28-2021 09:09 AM

Date Registered	‎03-15-2017 01:13 PM
Last Visited	‎06-28-2021 09:09 AM
Total Messages Posted	24
Total Kudos Received	3

Re: Cannot perform SHOW queries through Hortonwork...

Re: Cannot perform SHOW queries through Hortonwork...

Re: Cannot perform SHOW queries through Hortonwork...

Cannot perform SHOW queries through Hortonworks OD...

It is possible to prevent spnego on each knox acce...

Re: Remaining capacity of HDFS : 0

Remaining capacity of HDFS : 0

Re: Unable to start NFSGateway : resource_manageme...

Unable to start NFSGateway : resource_management.c...

Re: Execution of hadoop-mapreduce-examples.jar usi...

Re: Temporary table not found (ODBC driver issue ?...

Re: Temporary table not found (ODBC driver issue ?...

Re: Temporary table not found (ODBC driver issue ?...

Re: Temporary table not found (ODBC driver issue ?...

Re: Temporary table not found (ODBC driver issue ?...

Re: Temporary table not found (ODBC driver issue ?...

Temporary table not found (ODBC driver issue ?)

Re: Execution of hadoop-mapreduce-examples.jar usi...

Re: Execution of hadoop-mapreduce-examples.jar usi...

Execution of hadoop-mapreduce-examples.jar using t...

Re: Disk size used is bigger than replication numb...

Disk size used is bigger than replication number m...

ParseException while trying to execute a "SHOW VIE...

Re: Is that possible to display special characters...

Is that possible to display special characters in ...