Member since
04-30-2015
61
Posts
4
Kudos Received
0
Solutions
12-18-2017
05:17 AM
Hiveserver2 interactive UI not showing up query summaries. active sessions, open queries, last max 25 queries are not showing up in HS2I UI. is there any properties need to be set for this ? thanks, sathish
... View more
Labels:
09-22-2017
05:17 AM
using postgres database no logs has been captured in /var/log/superset. see the below warning and error message from amabri. any help much appreciated!! resource_management.core.exceptions.ExecutionFailed: Execution of 'source /etc/superset/conf/superset-env.sh ; /usr/hdp/current/druid-superset/bin/superset init' returned 1. /usr/hdp/2.6.2.0-205/superset/lib/python3.4/importlib/_bootstrap.py:1161: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead.
Trying to perform kerberos login via command: /usr/bin/kinit -r 3600s -kt /etc/security/keytabs/druid.service.keytab druid@HADOOP_xyz.COM
Exception in thread Kerberos-Login-Thread:
Traceback (most recent call last): 2017-09-22 04:54:28,483:WARNING:flask_appbuilder.models.filters:Filter type not supported for column: password 2017-09-22 04:54:28,577:WARNING:flask_appbuilder.models.filters:Filter type not supported for column: password 2017-09-22 04:54:28,606:WARNING:flask_appbuilder.models.filters:Filter type not supported for column: password File "/usr/hdp/2.6.2.0-205/superset/lib/python3.4/base64.py", line 90, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding
... View more
Labels:
07-17-2017
09:29 AM
this is the output and it's hung. its not moving further to the cli mode 17/07/17 09:28:29 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist 17/07/17 09:28:29 WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties this is the log from /tmp/userid/logs 2017-07-17 09:38:39,482 INFO[main]: hive.metastore (HiveMetaStoreClient.java:open(402)) - Trying to connect to metastore with URI thrift://hostname :9084 2017-07-17 09:38:39,657 INFO[main]: hive.metastore (HiveMetaStoreClient.java:open(498)) - Connected to metastore. its getting connected to the metastore but taking too much of time to launch the CLI thanks, sathish
... View more
07-17-2017
07:28 AM
Hive command line is hung and gives me below warnings WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist checked hive-site.xml and above parms are exist with "true". in logs not finding any details. please help me fixing the issue thanks, sathish
... View more
Labels:
07-13-2017
05:59 AM
is there any docs/links which talks more about Hive statistics ? please let me know
... View more
07-04-2017
09:08 AM
thanks for reply.. running auto stats on hive tables.. calculating stats on table which is default while create or insert.(hive.stats.autogather=true) compute stats for table , calculates number of rows on that table by scanning the table and there wont be significant impact on the cluster or analyze job wont run for longer time. compute stats for columns, it has to calculate num of distinct, nulls,avg min/max lenght of column etc., so, analyze jobs are running for longer time with more num.of mappers and reducers (this depends on the size of the table and num of columns). In such situations the impact of the cluster or resource utilisation is high. Is there any best practices before running stats for table columns ? Even though the stats task is run as a batch job , we want it to be executed as efficiently as possible. Basically, we expect to compute statistics on terabytes of data or more num of columns at a given time also, as part of stats calculation what are the important metastore tables involved or referred or updated? thanks, Sathish
... View more
07-04-2017
08:17 AM
what will be the impact on the cluster if we turn on auto stats ? or how can we calculate the impact ?
... View more
- Tags:
- Data Processing
- Hive
Labels:
07-03-2017
06:51 AM
This may be a direct question it would be great if you can give me some insights on "Running analyze".
updating stats always helps. but what'll be the burden cost of running analyze at the cluster level. how can we measure cost for running analyze on bigger tables(around TBs). please share me some details or docs if available about running stats on the hive tables. thanks, sathish
... View more
Labels:
02-28-2017
09:40 AM
i've configured the datasource connection but still i'm not able to connect to the hive databases. IODBC version -- IODBC-SDK-3.52.12-MacOSX Hive ODBC version -- 2.1.5.1006 Followed below link for setting up https://hortonworks.com/wp-content/uploads/2016/08/Hortonworks-Hive-ODBC-Driver-User-Guide.pdf --> page 32 and for kerberos enabled system below are my configs. Description = Hive Teast HOST = test1234.abc-xyz.com PORT = 10010 -> its correct port HiveServerType = 2 AuthMech = 1 Schema = test KrbRealm = hive/test1234.abc-xyz.com@HADOOP_TECH.ABC-XYZ.COM KrbHostFQDN = test1234.abc-xyz.com KrbServiceName = hive ThriftTransport = SASL DelegateKrbCreds = 1 Driver = /opt/hortonworks/hiveodbc/lib/universal/libhortonworkshiveodbc.dylib
... View more
01-19-2017
05:51 AM
please someone help me with below error [flume_tstr400367-1484119363347-a597ba80-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1484119363347-a597ba80-leader-finder-thread], Failed to find leader for Set([connect-test,0]) java.lang.NullPointerException thanks, sathish
... View more
01-11-2017
07:39 AM
i just wanna doubel check the compatibility between all these three things.. flume version : Flume 1.5.2.2.4.2.0-258 Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: e2cd47aa3cc959a659a97dfe800138d11744fa62 Compiled by jenkins on Mon Apr 25 08:49:00 UTC 2016 From source with checksum 2cd1bf99fd2acccc72128b0254e445ea kafka version : 0.9.0.2.4.2.0-258 security protocol : plaintextsasl thanks, sathish
... View more
01-11-2017
07:06 AM
i brought up the agent with kafka as source.. however hdfs sink is not working ... and am seeing below error in agent log. WARN [flume_tstr400367-1484117904365-8f70ac48-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1484117904365-8f70ac48-leader-finder-thread], Failed to find leader for Set([connect-test,0]) java.lang.NullPointerException at org.apache.kafka.common.utils.Utils.formatAddress(Utils.java:312) at kafka.cluster.Broker.connectionString(Broker.scala:62) at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89) at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:89) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
... View more
01-10-2017
11:13 AM
i was able to successfully create flume agent with normal spooldir source... its failing or not even starting the agent for kafka soruce
... View more
01-10-2017
10:58 AM
i'm not able to start the agent from ambari. i've started ambari service from amabri but not ambari agent. its neither logging anything nor starting... all i can see is "no flume to display". below are my config details.... kafka source and hdfs sink #flume agent config agent.sources = kafka-source-1 agent.channels = hdfs-channel-1 agent.sinks = hdfs-sink-1 #source properties agent.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource agent.sources.kafka-source-1.zookeeperConnect = tstr400360.xyz:2181 agent.sources.kafka-source-1.topic = connect-test agent.sources.kafka-source-1.batchSize = 100 agent.sources.kafka-source-1.channels = hdfs-channel-1 #sink properties agent.sinks.hdfs-sink-1.channel = hdfs-channel-1 agent.sinks.hdfs-sink-1.type = hdfs agent.sinks.hdfs-sink-1.hdfs.writeFormat = Text agent.sinks.hdfs-sink-1.hdfs.fileType = DataStream agent.sinks.hdfs-sink-1.hdfs.filePrefix = test-events agent.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true agent.sinks.hdfs-sink-1.hdfs.path = /user/flume/stage agent.sinks.hdfs-sink-1.hdfs.rollCount=100 agent.sinks.hdfs-sink-1.hdfs.rollSize=0 # channel properties agent.channels.hdfs-channel-1.type = memory agent.channels.hdfs-channel-1.capacity = 10000 agent.channels.hdfs-channel-1.transactionCapacity = 1000 thanks, sathish
... View more
01-06-2017
10:48 AM
i just copied all the jars from /kafka-server/libs to /flume-server/lib... but still am seeing the same error.. kafka version - 0.9 flume version - Flume 1.5.2.2.4.2.0-258 thanks, sathish
... View more
01-05-2017
11:00 AM
Hi - i'm trying to do setup "kafka(as source) to flume to hdfs" but getting below error message in flume logs. [flume_tstr400367-1483611425172-236ed292-leader-finder-thread] (kafka.utils.Logging$class.warn:89) - [flume_tstr400367-1483611425172-236ed292-leader-finder-thread], Failed to find leader for Set([connect-test,0]) java.lang.NullPointerException at org.apache.kafka.common.utils.Utils.formatAddress(Utils.java:312) at kafka.cluster.Broker.connectionString(Broker.scala:62) at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89) at kafka.client.ClientUtils$$anonfun$fetchTopicMetadata$5.apply(ClientUtils.scala:89) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:89) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) can you please help in fixing the issue
... View more
Labels:
01-03-2017
10:41 AM
i'm not using confluent platform .. please let me know the details for HDP .. thanks, sathish
... View more
01-03-2017
10:38 AM
Note : i need details for HDP platform... thanks, sathish
... View more
01-03-2017
10:11 AM
i'm trying to set up RDBMS (mysql) is source for kafka connect but its failing for the connector class..i tried with "io.confluent.connect.jdbc.JdbcSourceConnector" but kafka connect failed with below error ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:91) org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.connect.jdbc.JdbcSourceConnector for configuration connector.class: Class org.apache.kafka.connect.jdbc.JdbcSourceConnector could not be found. at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:255) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:145) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:49) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56) at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:71) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.startConnector(StandaloneHerder.java:194) at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:145) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:85) can you please help me ? thanks, sathish
... View more
Labels:
01-03-2017
08:56 AM
i should have been more specific.. i'm looking for kafka connector class for rdbms as source. can you please let me know the connector class or any doc for reference... thanks, sathish
... View more
01-03-2017
08:48 AM
its not specific to mysql... connector class for "rdbms" source. thanks, sathish
... View more
01-03-2017
06:57 AM
@Sandeep Nemuri right now I'm testing with RDMS source(mysql) and kafka connect is failing for "connector.class". how can i find the correct connector class for rdms(mysql database). i've tried with org.apache.kafka.connect.jdbc.JdbcSourceConnector,io.confulent.connect.jdbc.JdbcSourceConnector and both are not exist. thanks, sathish
... View more
01-03-2017
04:35 AM
yes.. its was problem with the security protocol.. i've changed them now and it started working now. is there any link or doc for parameter reference ? thanks, sathish
... View more
01-02-2017
10:49 AM
@Sandeep Nemuri i've setup port 6667 with sec.protocol to plaintextsasl.. but kafka connect by default running with producer properties (security.protocol = PLAINTEXT) . how can i override the parms for kafka connect. i've updated the parms in standalone.properties but kafka connect is not taking the parm while starting it. how should change the producer properties for kafka connect ? thanks, sathish
... View more
01-02-2017
10:11 AM
see the below error msg's ..it says connection refused for the broker host. DEBUG Connection with tstr400367.abc-test.com/10.246.131.35 disconnected (org.apache.kafka.common.network.Selector:307) java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:54) at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:72) at org.apache.kafka.common.network.Selector.poll(Selector.java:274) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) at java.lang.Thread.run(Thread.java:745) i've mentioned proper node name with port id, now am not sure what to check thanks, sathish
... View more
01-02-2017
10:02 AM
yes.. i've started in debug mode.. please gimme some time .. am going through the logs now
... View more
01-02-2017
09:58 AM
right now i see all below parms in connect-log4j properties log4j.rootLogger=INFO, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n log4j.logger.org.apache.zookeeper=ERROR log4j.logger.org.I0Itec.zkclient=ERROR thanks, sathish
... View more
01-02-2017
09:57 AM
there are no such parms for kafak connect-log4j.properties.. can you please let me know the parameter thanks, sathish
... View more
01-02-2017
09:13 AM
@Sandeep Nemuri yup., i've tried that too.. but still same error thanks, sathish
... View more
01-02-2017
09:06 AM
yes.. its localhost:6667. initially it was localhost:9092 but i changed to 6667 thanks, sathish
... View more