About kkhan

kkhan · ‎10-04-2019

In case, you get below error, make sure you use Nifi host FQDN in API call and NOT IP address. Also, make sure DNS is configured correctly. <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /nifi-api/access/kerberos. Reason: <pre> Unauthorized</pre>

kkhan · ‎02-17-2017

try increasing heap size for metastore. Make sure DB connection is working fine.

kkhan · ‎02-16-2017

@rahul gulati Which version of Ambari? Is your cluster kerberized? Is Ambari SSL enabled? Is this on local Ambari Cluster? Can you share all setting from your FileView

kkhan · ‎01-26-2017

I guess the easiest would be to have a lookup table with all processor name records, then you can use something like below. SELECT b.processor_name, count(*) Cnt from process_types_with_count_table A, processor_lookup_table BWHERE instr(b.processor_name, a.processor_name) > 0 Group by b.processor_name; This is not very efficient as you are joining other table here, but will do the work. Other option is if your processor name has any pattern then you can use regex in hive query to get only processor_name that you want.

kkhan · ‎01-26-2017

is processor name always 4 letters word? If yes, you can use substr to get only first letters from processor name and group by that. If processor name length varies then look for set of character that comes after processor name and eliminate anything after those characters. Here you can use mix of instr and substr hive function.

kkhan · ‎12-02-2016

@Manish Gupta try adding hive-metastore.jar as well in squirrel jar list.

kkhan · ‎11-23-2016

Depends on what arguments you are providing to the hash function. If your argument values are unique, you would most likely get unique value from hash. Keep in mind hive hash function return int (which is 32bit) so you may see -ve numbers as well. You can use something like reflect('java.util.UUID','randomUUID') to generate uniqueID or comeup with some unique code. I would not suggest using hash function, if you want to generate unique ids.

kkhan · ‎08-03-2016

Try running set; command. It should display all the values for all the variables in current session.

kkhan · ‎07-29-2016

Hi Upendra, The recommendation is to use VARCHAR and Integer Types (TINYINT, SMALLINT, INT, BIGINT) where ever possible instead of using String. In hive String is treated as VARCHAR(32762). So, if you have data that is not more than 50 characters in length, using string here will have some overhead. Same for integer types. Hope this helps.

kkhan · ‎07-28-2016

Just starting to understand Spark memory management on yarn and got few questions that I thought would be better to ask experts here. 1. Is there a way to restrict max size that users can use for Spark executor and Driver when submitting jobs on Yarn cluster? 2. What the best practice around determining number of executor required for a job? Is there a max limit that users can be restricted to? 3. How RM handles resource allocation if most of the resources are consumed by Spark jobs in a queue? How preemption is handled?

kkhan · ‎07-26-2016

You can deploy Master Slave KDC. That will provide HA. I have done this before. You can setup replication between master and slave. http://www.tldp.org/HOWTO/Kerberos-Infrastructure-HOWTO/server-replication.html HTH

kkhan · ‎07-14-2016

If this fixed your issue, can you accept this as Answer. It would help others in community.

kkhan · ‎07-13-2016

Here is the link that has syntax to compute statisitcs on hive partitioned table. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_dataintegration/content/cost-based-opt.html ANALYZE TABLE employees PARTITION (dt) COMPUTE STATISTICS

kkhan · ‎07-13-2016

Run analyze table <table name> compute statistics;

kkhan · ‎07-11-2016

Are you able to ping each host from Ambari server nodes and vice-versa? The error says "org.apache.ambari.server.HostNotFoundException: Host not found, hostname=ctsc00675971901.cts.com"

kkhan · ‎06-07-2016

yes, you will have to choose some other column.

kkhan · ‎06-07-2016

yes, you will have to choose some other column.

kkhan · ‎06-07-2016

Is id_person part of primary key? If it is part of primary_key, it won't work.

kkhan · ‎06-07-2016

yes you can. Also, to investigate why it is using only one mapper, can you check your mapper size and size of that you are trying to import? How did you calculate your split size?

kkhan · ‎06-01-2016

Are you able to ping sandbox.hortonworks.com from your windows machine? If not, try using IP address instead of hostname.

kkhan · ‎06-01-2016

You can use Ambari views(HDFS, Hive, Pig) within Ambari web interface.

kkhan · ‎05-17-2016

Take a look at below properties. You can reduce the retention time of local logs. yarn.nodemanager.log.retain-seconds: Time in seconds to retain user logs on the individual nodes if log aggregation is disabled. Default is 10800 yarn.nodemanager.log.deletion-threads-count: Determines the number of threads used by the NodeManagers to clean-up logs once the log-retention time is hit for local log files when aggregation is disabled.

kkhan · ‎05-04-2016

Thanks Maxwell. I was using 2 days date range whereas data was only available for few hours. I was thinking it will still respond with whatever data it has. Now I ran with 2 mins range and it came back with metrics. However, this data looks very granular. Couple of questions for you: Is there a way to get 15 mins aggregate kind of metrics? Right now, it looks like every 30 second data is available. What is the use of last parameter in query (i.e. current timestamp) How does the step param(last parameter in date range query) work in time range query.

kkhan · ‎05-04-2016

I am trying to run some test API calls on HDP Sandbox 2.4 with Ambari 2.2.1.0 and facing some issues. Here is the call API call I am making. http://localhost:8080/api/v1/clusters/Sandbox?fields=metrics/load[1462233600,1462406399]&_=1462328977 and all I am getting in return is { "href" : "http://localhost:8080/api/v1/clusters/Sandbox?fields=metrics/load[1462233600,1462406399]&_=1462328977", "Clusters" : { "cluster_name" : "Sandbox", "version" : "HDP-2.4" } Would appreciate any help here.

Online	Offline
Last Visited	‎01-19-2021 12:46 PM

Date Registered	‎04-06-2016 08:12 PM
Last Visited	‎01-19-2021 12:46 PM
Total Messages Posted	47
Total Kudos Received	7

Re: Connecting to Hive from SquirreL SQL client

Re: Hive: hash function unique values

Re: Is there a recommended architecture/methods to...

Re: Use a Kerberos token for accessing NiFi REST A...

Re: Hive metastore goes down without any error log...

Re: Unable to open File View from Ambari UI

Re: Need output as summarized form of total.

Re: Need output as summarized form of total.

Re: Connecting to Hive from SquirreL SQL client

Re: Hive: hash function unique values

Re: Hive ORC defaults read out

Re: Hive STRING vs VARCHAR Performance

Is it possible to define max memory size for Spark...

Re: Is there a recommended architecture/methods to...

Re: Select * from and & Select count (*) different...

Re: Select * from and & Select count (*) different...

Re: Select * from and & Select count (*) different...

Re: Not able to register hosts-Ambari host registr...

Re: sqoop import

Re: sqoop import

Re: sqoop import

Re: sqoop import

Re: Spark Master UI

Re: Sandbox hue not working

Re: Can we change yarn.nodemanager.log-dirs value ...

Re: Not able to retrieve average load Ambari metri...

Not able to retrieve average load Ambari metric us...