The java heap size of the catalog server in my cluster is 16G. Mem_rss metric of impala catalogd is 15G. What is the mem_rss metric of impala catalogd? If the mem_rss metric is high, should I increase the Java heap size? Version is CDH 5.14 ... View more

When I click impala query profile, Hue raise error. Exception cause Self-signed certificate. I am finding the solution.       bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)"       Current configuration in my cluster : Impala :     Enable TLS/SSL for Impala : disabled SSL/TLS Certificate for Impala component Webserver : Daemon, Catalog, Statestore enabled       I want to set PEM file path. Where are Hue - [Impala] - [[ssl]] properties used?     ... View more

Questions about how to configurate Hue Configuring TLS/SSL for Impala     [impala] # Port of the Impala Server ## server_port=21050 # URL of the Impala Coordinator Server. ## coordinator_url=localhost:25000 [[ssl]] # SSL communication enabled for this server. ## enabled=false # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Choose whether Hue should validate certificates received from the server. ## validate=true     Do Imapala TLS/SSL settings in Hue apply to both 21050 and 25000 communications? Impala in my cluster, Only 'SSL/TLS Certificate for Impala component Webserver' is configured. https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/impala_ssl.html#concept_gnk_2tt_qp   Port 25000 uses SSL. Port 21050 does not use SSL.   Would the connection to port 21050 fail If I set ssl for Imapala in hue.     When I click impala query profile, Hue raise error. I am finding the solution.     bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)"     ... View more

https://phoenix.apache.org/tracing.html   CDH 5.14.2   Hi. I try to setting Apache Phoenix tracing server. But SYSTEM.TRACING_STATS table is empty.   I think Phoenix enabled tracing server.  This is Hbase log.       2020-06-23 19:32:37,363 INFO org.apache.phoenix.trace.PhoenixMetricsSink: Writing tracing metrics to phoenix table 2020-06-23 19:32:37,365 INFO org.apache.phoenix.trace.PhoenixMetricsSink: Phoenix tracing writer started 2020-06-23 19:32:37,377 INFO org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink tracing started         I am set property in client code.        properties.setProperty("phoenix.trace.frequency", "always");         Can you give me a solution?  or Can you give me Guide how to set phoenix tracing server? ... View more

I am using Cloudera CDH 6.1. There are services Flume, HDFS, Hive, Hue, Impala, Kafka, Oozie, Spark, Yarn and Zookeepr. And I am using MySQL. There are databases amon, hue, metastore, oozie, rman, scm. I think some databases like rman and amon are not being used.   So, my question. I have to move the database another rack. But I want to continue saving data. ( kafka - flume - hdfs pipeline ) Can you give me a manual?   I think only some sesrvices are using the database. Maybe I only stop them, I don't have to stop the other services in cluster.   After only stop the services as Hive, Hue, Impala, Oozie and Spark, can I save datas into the cluster? Is Cluster good during the database be moved?       ... View more

cdh version is 6.1.0     1. Try merge small files.   insert overwrite table tdb.tb_activity partition(ymd) select * from tdb.tb_activity where ymd = '2019-07-22'; 2. But, Exception raised.    "UnknownReason Blacklisting behavior can be configured..." in HUE. And, Below is a spark container error log.   Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Number of input columns was different than output columns (in = 9 vs out = 8 at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:805) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:146) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:484) ... 19 more Caused by: org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Number of input columns was different than output columns (in = 9 vs out = 8 at org.apache.hadoop.hive.serde2.avro.AvroSerializer.serialize(AvroSerializer.java:75) at org.apache.hadoop.hive.serde2.avro.AvroSerDe.serialize(AvroSerDe.java:212) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:725) ... 25 more 19/07/23 17:25:32 ERROR executor.Executor: Exception in task 4.0 in stage 0.0 (TID 2)   3. I found a strange in "explain query". line24 : expressions: ..... extras (type: string), ....   line41: avro.schema.literal {"type":"record", ....... ,{"name":"extras","type":["null","string"],"default":null},  .....   line44: columns actiontype,contentid,contenttype,device,serviceid,timestamp,userip,userid No have "extras" field on line 44.   10 11 STAGE PLANS: 12 Stage: Stage-1 13 Spark 14 DagName: hive_20190723175708_8b93a3ff-d533-48cc-865e-6af87f576858:29 15 Vertices: 16 Map 1 17 Map Operator Tree: 18 TableScan 19 alias: tb_activity 20 filterExpr: (ymd = '2019-07-22') (type: boolean) 21 Statistics: Num rows: 16429 Data size: 13275304 Basic stats: COMPLETE Column stats: NONE 22 GatherStats: false 23 Select Operator 24 expressions: actiontype (type: string), contentid (type: string), contenttype (type: string), device (type: string), extras (type: string), serviceid (type: string), timestamp (type: bigint), userip (type: string), userid (type: string), '2019-07-22' (type: string) 25 outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9 26 Statistics: Num rows: 16429 Data size: 13275304 Basic stats: COMPLETE Column stats: NONE 27 File Output Operator 28 compressed: true 29 GlobalTableId: 1 30 directory: hdfs://nameservice1/etl/flume/tb_activity/.hive-staging_hive_2019-07-23_17-57-08_154_1733664706886256905-5/-ext-10002 31 NumFilesPerFileSink: 1 32 Statistics: Num rows: 16429 Data size: 13275304 Basic stats: COMPLETE Column stats: NONE 33 Stats Publishing Key Prefix: hdfs://nameservice1/etl/flume/tb_activity/.hive-staging_hive_2019-07-23_17-57-08_154_1733664706886256905-5/-ext-10000/ 34 table: 35 input format: org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat 36 output format: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat 37 properties: 38 DO_NOT_UPDATE_STATS true 39 EXTERNAL TRUE 40 STATS_GENERATED TASK 41 avro.schema.literal {"type":"record","name":"Activity","namespace":"com.bigdata.avro","doc":"Schema for com.bigdata.avro.Activity","fields":[{"name":"actionType","type":["null","string"]},{"name":"contentId","type":["null","string"]},{"name":"contentType","type":["null","string"]},{"name":"device","type":["null","string"]},{"name":"extras","type":["null","string"],"default":null},{"name":"serviceId","type":["null","string"]},{"name":"timestamp","type":["null","long"]},{"name":"userIp","type":["null","string"]},{"name":"userid","type":["null","string"]}]} 42 avro.schema.url hdfs:///metadata/avro/tb_activity.avsc 43 bucket_count -1 44 columns actiontype,contentid,contenttype,device,serviceid,timestamp,userip,userid 45 columns.comments 46 columns.types string:string:string:string:string:bigint:string:string 47 file.inputformat org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat 48 file.outputformat org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat 49 impala.lastComputeStatsTime 1559636056 50 location hdfs://nameservice1/etl/flume/tb_activity 51 name tdb.tb_activity 52 numRows 83020631 53 partition_columns ymd 54 partition_columns.types string 55 serialization.ddl struct tb_activity { string actiontype, string contentid, string contenttype, string device, string serviceid, i64 timestamp, string userip, string userid} 56 serialization.format 1 57 serialization.lib org.apache.hadoop.hive.serde2.avro.AvroSerDe 58 totalSize 6334562388 59 transient_lastDdlTime 1556875047 60 serde: org.apache.hadoop.hive.serde2.avro.AvroSerDe 61 name: tdb.tb_activity 62 TotalFiles: 1 63 GatherStats: true 64 MultiFileSpray: false   "extras" field on avro schema have "default" property. and other fields has no "default" property. I have been doing avro schema changes in the past. The "extras" field was then added.   What is wrong? ... View more

http://gethue.com/drag-drop-saved-hive-queries-into-your-workflows/#comment-78368 Did it not working at currrent version? ex, Hue4.3 etc.. Parameter Name is 'zip' when submit workflow. It's not 'zip_code' Input From. this like. I expected to look like this image.     But, It didn't working.           Please. Could you check my demo workflow. http://demo.gethue.com/hue/oozie/editor/workflow/edit/?workflow=292135 I 'am use cloudera cdh 6.1.0.     ... View more

If the client application exits before the impala connection is terminated Impala hosts have zombie tcp connections.   step 1 : connection to impalad using jdbc step 2 : query step 3 : application shutdown without jdbc connection closed   I closed it on the web ui(<impalad host>:25000/sessions ) of the impalad daemon, but the tcp connection does not disappear. I set 'idle_session_timeout' but tcp connection does not disappear.   ... View more

The data that I collect contains complex types and should guarantee a response time of less than 5 seconds. I might use Hbase, but I want to use Impala. I know that Impala does not support complex types. What I want is for Impala to skipping complex types. As a result of my checking, Impala skipping a complex type in a parqut format file.   How do I write a parquert format file to hdfs, hive, impala etc.? Can I write a parquert file using Flume, Morphline, etc.?   My system's data collection flow is as follows. Kafka -> Flume -> hdfs(avro file) -> hive ... View more

CDH 6.0.1   I want to build 10 Worker Hosts with High Availability. I referenced the following guide documents.   https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_host_allocations.html https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_hardware_requirements.html http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/     My cluster services Kafka, HDFS, YARN, Hive on Spark, Hue, Oozie, Spark Streamnig, Sqoop and Impala. I will deploy the host as shown below.     Gateway. Can I place the Gateway on the Worker host?   Utility. I understand about HA in Namenode, but I have not yet learned HA for other services. When I configure HA for all other services in the cluster, how many additional Hosts do I need? For example, services such as cloudera manager, Yarn, Hive, Hue, Spark, etc.   Hard Requirement. I collect 90 GB of text log per day. My cluster collects, ETLs, or aggregates (counts) text logs of 90 GB per day. The cluster handles about 10 workflows. If 32G of memory is allocated to the MasterNode and 64G of memory is used for the WorkNode, is the memory adequate?     I wait for your advice.   ... View more

My hbase version is 0.92.1-cdh4.0.1.   I do not know the history of the past, but the hbase version of one server is 0.94.15-cdh4.7.0.   So I tried to decommission the 0.94.15-cdh4.7.0 version of the server.   However, decommission failed.   Cloudera Manager can see the failure, but can not determine the cause. master server, region server The log does not contain a log of failure.   --------------------- Cloudera Manager > hbase > Commands Command Details: Decommission "Failed to decommission Region Servers"   Toggle Balancer     - Successfully toggled balancer Decommission     - Command completed with 0/1 successful subcommands. Toggle Balancer     - Successfully toggled balancer   ------------------------------------   s there a way I can solve this problem? ... View more