About BharathN

BharathN · ‎01-17-2019

Can someone please assist here

BharathN · ‎11-29-2018

Hello Everyone - we have couple of queries which are simple create table or select queries taking long time. we set the below parameters: set hive.compute.query.using.stats=true; set hive.stats.fetch.column.stats=true; set hive.stats.fetch.partition.stats=true; set hive.vectorized.execution.enabled =true; set hive.vectorized.execution.reduce.enabled =true; Still there is no siginificant change. First of all, what should i look oe where should i look on Tez UI to figure out what i taking time. When a user comes with questions saying - query taking loner time, what things i should check into. Also - I dont understand Explain plan. Please help me understand, what parameters of fields to look for.

BharathN · ‎05-22-2018

I have a 4 node cluster (2 master & 2 Data nodes) - fresh Installation. One of the Datanode is not coming up - 2018-05-22 14:37:56,024 ERROR datanode.DataNode (BPServiceActor.java:run(780)) - Initialization failed for Block pool <registering> (DatanodeUuid unassigned) service to Host1.infosolco.net/10.215.78.41:8020. Exiting. java.io.IOException: All specified directories are failed to load. When I see the VERSION file, the : root@Datanode02:/spark/hdfs/data/current # cat VERSION #Tue May 22 14:00:02 PDT 2018 storageID=DS-0009b75a-e67a-4623-b7a2-12bf395c1d61 clusterID=CID-eb6df30f-7f16-4f94-826c-c7640e1e45a2 cTime=0 datanodeUuid=f005656a-673e-4c97-b25a-e19f04e1ec94 storageType=DATA_NODE layoutVersion=-56 __________________ root@Datanode01:/spark/hdfs/data/current # cat VERSION #Tue May 22 14:00:02 PDT 2018 storageID=DS-0009b75a-e67a-4623-b7a2-12bf395c1d61 clusterID=CID-eb6df30f-7f16-4f94-826c-c7640e1e45a2 cTime=0 datanodeUuid=f005656a-673e-4c97-b25a-e19f04e1ec94 storageType=DATA_NODE layoutVersion=-56 I see both datanodes have same Uuid, and 2nd data node is not coming up. Please suggest!

BharathN · ‎11-20-2017

I want to load data(~300 GB) from local filesystem to HDFS. And i will be doing similar activity once every month. What would be the feasible way to get this done. I am looking at Flume & HDFS Put options. These are some files (XML) and not log data. I dont need any conversion, its a straight copy to HDFS.

BharathN · ‎10-11-2017

I see edit logs created for every 2 minute interval @"/data/hadoop/hdfs/namenode/current" directory. Is there a best practice to maintain the edit logs in terms of purging. Also, we have a problem where Namenode fails to start and we use the following post "ttp://blog.cloudera.com/blog/2012/05/namenode-recovery-tools-for-the-hadoop-distributed-file-system/" to recover it. Is there something to do with the edits here.

BharathN · ‎10-04-2017

@Slim I started the Druid installation process, but ran into some errors ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install superset_2_6_0_3_8' returned 1. Error: Package: superset_2_6_0_3_8-0.15.0.2.6.0.3-8.x86_64 (HDP-2.6) Requires: libffi-devel I tried uninstalling "libffi-devel" but it has dependencies, and not able to uninstall root@Host:~ # rpm -e libffi-3.0.5-3.2.el6.x86_64 error: Failed dependencies: libffi.so.5()(64bit) is needed by (installed) python-libs-2.6.6-66.el6_8.x86_64 libffi.so.5()(64bit) is needed by (installed) python34-libs-3.4.5-1.el6.x86_64 root@host:~ # /usr/bin/yum -d 0 -e 0 -y install superset_2_6_0_3_8 --skip-broken Packages skipped because of dependency problems: openblas-0.2.19-2.el6.x86_64 from CDP openblas-devel-0.2.19-2.el6.x86_64 from CDP openblas-openmp-0.2.19-2.el6.x86_64 from CDP openblas-threads-0.2.19-2.el6.x86_64 from CDP superset_2_6_0_3_8-0.15.0.2.6.0.3-8.x86_64 from HDP-2.6

BharathN · ‎10-04-2017

@Slim For historicals and middle manager, do they need to be installed on Data nodes or Edge nodes from where users generally access the cluster?

BharathN · ‎10-04-2017

Hello Everyone - We have a 16 node cluster [12 data nodes, 2 Master nodes, 2 edge nodes].All the servers have 32 cores & 252GB RAM. For Installing druid, how to choose the following : Cordinator,Superset,Broker,Overlord,Router.Can I have all these sitting on 1 instance? And for the Slaves & client: Druid Historical & Druid MiddleManager, how to pick on what servers these to be installed. Once the installation is done, are there other steps involved to integrate this with hive or is it done as part of installation? As part of Druid Installation, are there any other services that gets impacted or needs to be restarted? I have gone through this article: https://community.hortonworks.com/questions/108316/how-to-choose-servers-for-druid.html Please let me know if I miss anything or any other steps involved prior to the installation. Thanks in advance.

BharathN · ‎10-02-2017

@Jay SenSharma # ps -ef | grep hiveserver2 I see the process running, Also Ambari UI has the hiveserver2 running screen-shot-2017-10-02-at-21921-pm.png 2. # netstat -tnlpa | grep `cat /var/run/hive/hive-server.pid` I see a list, but I don't see any process with 10000.Instead I see couple of them with 10001 3. less /var/log/hive/hiveserver2.log > 2017-10-02 00:00:36,108 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(145)) - Could not validate cookie sent, will try to generate a new cookie 2017-10-02 00:00:36,108 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(204)) - Cookie added for clientUserName anonymous 2017-10-02 00:00:36,108 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(316)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 2017-10-02 00:00:36,109 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.ObjectStore (ObjectStore.java:initializeHelper(377)) - ObjectStore, initialize called 2017-10-02 00:00:36,111 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is OTHER 2017-10-02 00:00:36,111 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.ObjectStore (ObjectStore.java:setConf(291)) - Initialized ObjectStore 2017-10-02 00:00:36,111 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.HiveMetaStore (HiveMetaStore.java:init(533)) - Begin calculating metadata count metrics. 2017-10-02 00:00:36,113 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: metastore.HiveMetaStore (HiveMetaStore.java:init(535)) - Finished metadata count metrics: 18 databases, 1009 tables, 33 partitions. 2017-10-02 00:00:36,113 WARN [HiveServer2-HttpHandler-Pool: Thread-82257]: metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge with name [init_total_count_dbs] already exists. The old gauge will be overwritten, but this is not recommended 2017-10-02 00:00:36,113 WARN [HiveServer2-HttpHandler-Pool: Thread-82257]: metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge with name [init_total_count_tables] already exists. The old gauge will be overwritten, but this is not recommended 2017-10-02 00:00:36,113 WARN [HiveServer2-HttpHandler-Pool: Thread-82257]: metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge with name [init_total_count_partitions] already exists. The old gauge will be overwritten, but this is not recommended 2017-10-02 00:00:36,125 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created local directory: /tmp/de11c83f-e087-406e-b715-dd6ba7148cfe_resources 2017-10-02 00:00:36,128 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created HDFS directory: /tmp/hive/anonymous/de11c83f-e087-406e-b715-dd6ba7148cfe 2017-10-02 00:00:36,129 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created local directory: /tmp/hive/de11c83f-e087-406e-b715-dd6ba7148cfe 2017-10-02 00:00:36,130 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.SessionState (SessionState.java:createPath(677)) - Created HDFS directory: /tmp/hive/anonymous/de11c83f-e087-406e-b715-dd6ba7148cfe/_tmp_space.db 2017-10-02 00:00:36,130 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(265)) - Operation log session directory is created: /tmp/hive/operation_logs/de11c83f-e087-406e-b715-dd6ba7148cfe 2017-10-02 00:00:36,189 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:00:36,190 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:00:36,196 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:00:36,196 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:00:36,210 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:00:36,211 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:00:36,211 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:acquireAfterOpLock(333)) - We are setting the hadoop caller context to de11c83f-e087-406e-b715-dd6ba7148cfe for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:00:36,212 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: session.HiveSessionImpl (HiveSessionImpl.java:releaseBeforeOpLock(357)) - We are resetting the hadoop caller context for thread HiveServer2-HttpHandler-Pool: Thread-82257 2017-10-02 00:03:36,081 INFO [HiveServer2-HttpHandler-Pool: Thread-82257]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(145)) - Could not validate cookie sent, wi/var/log/hadoop/hive/hiveserver2.log What I might be missing here?

BharathN · ‎10-02-2017

@Jay SenSharma I don't see anything under /var/log/hive -its empty I ran the jps command on node where hive server is running root@Hive-server:~ # jps 55267 QuorumPeerMain 186337 RunJar 38279 JournalNode 43241 Jps 183016 RunJar 184015 NameNode 9038 HistoryServer 36721 ActivityAnalyzerFacade 192339 Bootstrap 6547 HistoryServer 189617 RunJar 9460 SparkSubmit 39412 DFSZKFailoverController 8089 LivyServer 13246 HMaster 7006 SparkSubmit Also, I checked the "etc/hosts" file on both the hive-server node and node from where beeline command is executing./...its exactly the same. From Ambari Ui, I see the port# is 10000. screen-shot-2017-10-02-at-115713-am.png I am not sure what am I missing here. Please let me know how to open a port, if that is what causing the issue here?

Online	Offline
Last Visited	‎07-18-2017 02:05 PM

Member Since	‎07-13-2017 11:26 AM
Last Visited	‎07-18-2017 02:05 PM
Posts	19

Cloudera Community

Cloudera Community

Re: Hive Tez query is taking long time to run than...

Hive Tez query is taking long time to run than exe...

Databode uuid unassigned

Loading data into HDFS from local filesystem, Flum...

NameNode edit logs - purging/Best practises

Re: Druid Installation

Re: Druid Installation

Druid Installation

Re: Hive beeline connectivity issue

Re: Hive beeline connectivity issue