About arunpoy

elserj · ‎09-02-2016

The Phoenix Query Server (PQS) is an optional service. You only need to add it if you intend to use it. PQS adds additional levels of connectivity into Apache Phoenix/HBase, but is not required for Phoenix access. Like Sunile points out, you can easily add this service to your installation later as the need/desire arises.

Enis · ‎08-24-2016

You do not have to worry about TTL by default unless you are explicitly setting the value. By default TTL will be 'FOREVER' which will never delete data as expected.

bwalter1 · ‎08-05-2016

There is an issue with the space in front of "EF": Let's use (you don't need the "escape" option, it can be used to e.g. get quotes into the dataframe if needed) val df = sqlContext.read.format("com.databricks.spark.csv") .option("header", "true") .option("delimiter", "|") .load("/tmp/test.csv") df.show() With space in front of "EF" +----+----+----+-----+ |Col1|Col2|Col3| Col4| +----+----+----+-----+ | AB| CD| DE| "EF"| +----+----+----+-----+ Without space in front of "EF": +----+----+----+----+ |Col1|Col2|Col3|Col4| +----+----+----+----+ | AB| CD| DE| EF| +----+----+----+----+ Can you remove the space before loading the csv into Spark?

leonardo_costa · ‎09-27-2016

Hi all, Could you please share more information about the following scenario... trying to run exactly the statement you mentioned... running from the active master_node : 16/09/26 17:42:53 INFO balancer.Balancer: namenodes = [hdfs://hadoop2, hdfs://linux.lab.domain.com:8020] 16/09/26 17:42:53 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, run during upgrade = false] 16/09/26 17:42:53 INFO balancer.Balancer: included nodes = [] 16/09/26 17:42:53 INFO balancer.Balancer: excluded nodes = [] 16/09/26 17:42:53 INFO balancer.Balancer: source nodes = [] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 16/09/26 17:42:53 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 16/09/26 17:42:53 INFO block.BlockTokenSecretManager: Setting block keys 16/09/26 17:42:53 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 16/09/26 17:42:54 INFO block.BlockTokenSecretManager: Setting block keys org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1872) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1306) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getServerDefaults(FSNamesystem.java:1618) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getServerDefaults(NameNodeRpcServer.java:595) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getServerDefaults(ClientNamenodeProtocolServerSideTranslatorPB.java:383) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2133) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2131) . Exiting ... . Exiting ... Sep 26, 2016 5:42:54 PM Balancing took 1.314 seconds it runs in less than 2 seconds....what seems not running 100% , right ? How many seconds it could take in average if it really runs fine ? or is there any log to check further information if some error is happenning ? I also tried to run the same from the Ambari console (balance all nodes)... the same result is reached. thanks and br

arunpoy · ‎08-04-2016

Have reached out to the pentaho PDI community

thanhtu3009 · ‎03-20-2019

Hi all, Do you have any suggestion for backing up and restore only 1 Phoenix table ? I believe if we restore the system table, it will have impact to all others, won't it ? Many thanks, Tu

ewalk · ‎07-26-2016

@ARUN Cloudbreak is the tool that will allow you to automate this using Ambari Blueprint.

arunpoy · ‎07-22-2016

Thanks Ankit, it works. but is it the effective way if we have large number of rows. AS we are using an IN condition in the query.

divakarreddy_a · ‎07-21-2016

@ARUN 1) Identifying directories usage in HDFS Try to run below for each directory to know the usage [root@abc01 ~]# hadoop fs -du -s -h <hdfs location> 50.9 M /abc [root@adc01 ~]# 2) namenode heap size is increasing and is around 80 - 90%. NN heap size increasing because of your cluster is occupying maximum space in HDFS for files & directories. Here is the link for increaseing values based on file size https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html I hope this will help you.

rlevas · ‎06-29-2016

@ARUNKUMAR RAMASAMY Assume by "secondary KDC", you either mean in a master/slave relationship or a trust relationship (where each KDC hosts its own realm) If this is a master/slave relationship, you need to edit that krb5.conf template (under the Advanced krb5-conf tab on the Kerberos service config page) to specify the additional KDC host and (optionally) the master KDC: [libdefaults] renew_lifetime = 7d forwardable = true default_realm = {{realm}} ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false #default_tgs_enctypes = {{encryption_types}} #default_tkt_enctypes = {{encryption_types}} {% if domains %} [domain_realm] {% for domain in domains.split(',') %} {{domain}} = {{realm}} {% endfor %} {% endif %} [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] {{realm}} = { admin_server = {{admin_server_host|default(kdc_host, True)}} kdc = fqdn.slave.kdc kdc = {{kdc_host}} master_kdc = {{kdc_host}} } {# Append additional realm declarations below #} Note the addition of the kdc and master_kdc entries in the realm definition. For the additional kdc entry, "fqdn.slave.kdc" should be change to the FQDN (and optionally port) of the slave KDC. If this is a trust relationship, then you need to add the additional realm to the krb5.conf template (under the Advanced krb5-conf tab on the Kerberos service config page). [libdefaults] renew_lifetime = 7d forwardable = true default_realm = {{realm}} ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false #default_tgs_enctypes = {{encryption_types}} #default_tkt_enctypes = {{encryption_types}} {% if domains %} [domain_realm] {% for domain in domains.split(',') %} {{domain}} = {{realm}} {% endfor %} {% endif %} [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] {{realm}} = { admin_server = {{admin_server_host|default(kdc_host, True)}} kdc = {{kdc_host}} } {# Append additional realm declarations below #} ADDITONAL.REALM = { admin_server = FQDN.admin.server kdc = fqdn.kdc } Note the additional realm, named "ADDITIONAL.REALM", which should be changed to the actual realm name. Also the admin_server and kdc values need to set appropriately. You will also want to add the additional realm to the "Additional Realms" value in the Kerberos admin page so that an entry will be created in the auto-generated auth-to-local rule sets. By editing the data on this page and saving it, I believe the configurations will be updated and you may have to restart some services. If not, you should click on the "Regenerate Keytabs" button and the configurations will be updated along with the new keytab files.

Online	Offline
Last Visited	‎03-14-2019 05:48 AM

Member Since	‎01-21-2016 11:26 AM
Last Visited	‎03-14-2019 05:48 AM
Posts	290
Kudos received	76

Cloudera Community

Re: Dead region server showing on ambari even afte...

Re: Zeppelin phoenix error

Re: job history not available

Re: Hbase Master

Re: Phoenix table TTL

Re: Escaping double quotes in spark dataframe

Re: Load not distributed in the cluster

Re: pentaho PDI (kettle) to phoenix

Re: Phoenix backup

Re: HDP on AWS

Re: phoenix delete based on joins

Re: files/directories occupying maximum space in h...

Re: configuring secondary kdc in ambari