Member since
08-15-2016
33
Posts
6
Kudos Received
4
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 6373 | 02-21-2017 10:56 AM | |
| 831 | 01-18-2017 07:06 AM | |
| 339 | 04-07-2016 04:15 PM | |
| 659 | 04-04-2016 05:03 PM |
05-25-2017
12:34 PM
Tim - See the next post for the TEXT PLAN. Please let me know, if you figure whats causing the error message.
... View more
05-25-2017
12:29 PM
Estimated Per-Host Requirements: Memory=628.99MB VCores=3 PLAN-ROOT SINK | 66:EXCHANGE [UNPARTITIONED] | hosts=10 per-host-mem=unavailable | tuple-ids=20,42N row-size=210B cardinality=26977 | 32:HASH JOIN [LEFT OUTER JOIN, PARTITIONED] | hash predicates: campaign = campaign, carrier = carrier, market = market, sessiontype = sessiontype | hosts=10 per-host-mem=292.69KB | tuple-ids=20,42N row-size=210B cardinality=26977 | |--65:EXCHANGE [HASH(campaign,market,carrier,sessiontype)] | | hosts=10 per-host-mem=0B | | tuple-ids=42 row-size=101B cardinality=26977 | | | 64:AGGREGATE [FINALIZE] | | output: sum:merge(CASE WHEN carrier_count = 2 THEN samples ELSE 0 END), sum:merge(CAST(samples AS FLOAT)), sum:merge(CASE WHEN carrier_count = 3 THEN samples ELSE 0 END), sum:merge(CASE WHEN carrier_count > 1 THEN samples ELSE 0 END), sum:merge(CASE WHEN carrier_count > 1 THEN sum_total_bandwidth ELSE 0 END) | | group by: campaign, market, carrier, sessiontype | | hosts=10 per-host-mem=10.00MB | | tuple-ids=42 row-size=101B cardinality=26977 | | | 63:EXCHANGE [HASH(campaign,market,carrier,sessiontype)] | | hosts=10 per-host-mem=0B | | tuple-ids=42 row-size=101B cardinality=26977 | | | 31:AGGREGATE [STREAMING] | | output: sum(CASE WHEN carrier_count = 2 THEN samples ELSE 0 END), sum(CAST(samples AS FLOAT)), sum(CASE WHEN carrier_count = 3 THEN samples ELSE 0 END), sum(CASE WHEN carrier_count > 1 THEN samples ELSE 0 END), sum(CASE WHEN carrier_count > 1 THEN sum_total_bandwidth ELSE 0 END) | | group by: campaign, market, carrier, sessiontype | | hosts=10 per-host-mem=10.00MB | | tuple-ids=42 row-size=101B cardinality=26977 | | | 16:UNION | | hosts=10 per-host-mem=0B | | tuple-ids=40 row-size=78B cardinality=26977 | | | |--62:AGGREGATE [FINALIZE] | | | output: sum:merge(ltebwcum), count:merge(*) | | | group by: campaign, market, carrier, sessiontype, carrier_count, tech_mode | | | having: tech_mode = 'LTECA' | | | hosts=10 per-host-mem=10.00MB | | | tuple-ids=38 row-size=94B cardinality=13400 | | | | | 61:EXCHANGE [HASH(campaign,market,carrier,sessiontype,carrier_count,tech_mode)] | | | hosts=10 per-host-mem=0B | | | tuple-ids=38 row-size=94B cardinality=13400 | | | | | 30:AGGREGATE [STREAMING] | | | output: sum(ltebwcum), count(*) | | | group by: a.campaign, a.market, a.carrier, CASE WHEN SESSIONTYPE = 'HTTPDL_CAPACITY_L' THEN 'HTTPDL_CAPACITY' ELSE SESSIONTYPE END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 4 WHEN l_pdschbytes_scc2 > 0 THEN 3 WHEN l_pdschbytes_scc1 > 0 THEN 2 WHEN L_pdschbytes > 0 THEN 1 ELSE 0 END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END | | | hosts=10 per-host-mem=10.00MB | | | tuple-ids=38 row-size=94B cardinality=13400 | | | | | 29:HASH JOIN [INNER JOIN, PARTITIONED] | | | hash predicates: a.campaign = campaign, a.carrier = carrier, a.market = a.market, a.filename = filename | | | other predicates: unix_timestamp(udfs.totimestamp(a.time_stamp)) <= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeend))), unix_timestamp(udfs.totimestamp(a.time_stamp)) >= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeinit))) | | | runtime filters: RF015 <- campaign, RF017 <- a.market, RF016 <- carrier, RF018 <- filename | | | hosts=10 per-host-mem=1018.71KB | | | tuple-ids=31,32,34 row-size=468B cardinality=13400 | | | | | |--60:EXCHANGE [HASH(campaign,carrier,a.market,filename)] | | | | hosts=10 per-host-mem=0B | | | | tuple-ids=32,34 row-size=281B cardinality=33804 | | | | | | | 28:HASH JOIN [INNER JOIN, BROADCAST] | | | | hash predicates: a.market = market | | | | hosts=10 per-host-mem=17.62KB | | | | tuple-ids=32,34 row-size=281B cardinality=33804 | | | | | | | |--58:EXCHANGE [BROADCAST] | | | | | hosts=5 per-host-mem=0B | | | | | tuple-ids=34 row-size=29B cardinality=569 | | | | | | | | | 57:AGGREGATE [FINALIZE] | | | | | group by: market | | | | | hosts=5 per-host-mem=10.00MB | | | | | tuple-ids=34 row-size=29B cardinality=569 | | | | | | | | | 56:EXCHANGE [HASH(market)] | | | | | hosts=5 per-host-mem=0B | | | | | tuple-ids=34 row-size=29B cardinality=569 | | | | | | | | | 27:AGGREGATE [STREAMING] | | | | | group by: market | | | | | hosts=5 per-host-mem=10.00MB | | | | | tuple-ids=34 row-size=29B cardinality=569 | | | | | | | | | 26:SCAN HDFS [mobistat.allstats_packet, RANDOM] | | | | partitions=1/1 files=6 size=9.32MB | | | | predicates: bbdo_approved = 1, campaign = '17D1' | | | | table stats: 51137 rows total | | | | column stats: all | | | | hosts=5 per-host-mem=32.00MB | | | | tuple-ids=33 row-size=53B cardinality=1065 | | | | | | | 25:SCAN HDFS [mobistat.cdr_packet a, RANDOM] | | | partitions=4680/10328 files=4681 size=4.70GB | | | predicates: regexp_like(calldirection, 'HTTPDL_CAPACITY') = TRUE, regexp_like(endresult, 'HTTP SUCCESS') = TRUE, (modpctlte + isnull(modpctlteca, 0)) > 0.999 | | | table stats: 58699689 rows total | | | column stats: all | | | hosts=10 per-host-mem=304.00MB | | | tuple-ids=32 row-size=252B cardinality=2888511 | | | | | 59:EXCHANGE [HASH(a.campaign,a.carrier,a.market,a.filename)] | | | hosts=10 per-host-mem=0B | | | tuple-ids=31 row-size=187B cardinality=23267425 | | | | | 24:SCAN HDFS [mobistat.psr_packet_cdma a, RANDOM] | | partitions=2332/3707 files=2332 size=45.71GB | | predicates: regexp_like(SESSIONTYPE, 'HTTPDL_CAPACITY') = TRUE, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END IS NOT NULL | | runtime filters: RF015 -> a.campaign, RF017 -> a.market, RF016 -> a.carrier, RF018 -> a.filename | | table stats: 358488531 rows total | | column stats: all | | hosts=10 per-host-mem=608.00MB | | tuple-ids=31 row-size=187B cardinality=23267425 | | | 55:AGGREGATE [FINALIZE] | | output: sum:merge(ltebwcum), count:merge(*) | | group by: campaign, market, carrier, sessiontype, carrier_count, tech_mode | | having: tech_mode = 'LTECA' | | hosts=10 per-host-mem=10.00MB | | tuple-ids=29 row-size=94B cardinality=13577 | | | 54:EXCHANGE [HASH(campaign,market,carrier,sessiontype,carrier_count,tech_mode)] | | hosts=10 per-host-mem=0B | | tuple-ids=29 row-size=94B cardinality=13577 | | | 23:AGGREGATE [STREAMING] | | output: sum(ltebwcum), count(*) | | group by: a.campaign, a.market, a.carrier, CASE WHEN SESSIONTYPE = 'HTTPDL_CAPACITY_L' THEN 'HTTPDL_CAPACITY' ELSE SESSIONTYPE END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 4 WHEN l_pdschbytes_scc2 > 0 THEN 3 WHEN l_pdschbytes_scc1 > 0 THEN 2 WHEN L_pdschbytes > 0 THEN 1 ELSE 0 END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END | | hosts=10 per-host-mem=10.00MB | | tuple-ids=29 row-size=94B cardinality=13577 | | | 22:HASH JOIN [INNER JOIN, PARTITIONED] | | hash predicates: a.campaign = campaign, a.carrier = carrier, a.market = a.market, a.filename = filename | | other predicates: unix_timestamp(udfs.totimestamp(a.time_stamp)) <= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeend))), unix_timestamp(udfs.totimestamp(a.time_stamp)) >= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeinit))) | | hosts=10 per-host-mem=1018.71KB | | tuple-ids=22,23,25 row-size=468B cardinality=13577 | | | |--53:EXCHANGE [HASH(campaign,carrier,a.market,filename)] | | | hosts=10 per-host-mem=0B | | | tuple-ids=23,25 row-size=281B cardinality=33804 | | | | | 21:HASH JOIN [INNER JOIN, BROADCAST] | | | hash predicates: a.market = market | | | hosts=10 per-host-mem=17.62KB | | | tuple-ids=23,25 row-size=281B cardinality=33804 | | | | | |--51:EXCHANGE [BROADCAST] | | | | hosts=5 per-host-mem=0B | | | | tuple-ids=25 row-size=29B cardinality=569 | | | | | | | 50:AGGREGATE [FINALIZE] | | | | group by: market | | | | hosts=5 per-host-mem=10.00MB | | | | tuple-ids=25 row-size=29B cardinality=569 | | | | | | | 49:EXCHANGE [HASH(market)] | | | | hosts=5 per-host-mem=0B | | | | tuple-ids=25 row-size=29B cardinality=569 | | | | | | | 20:AGGREGATE [STREAMING] | | | | group by: market | | | | hosts=5 per-host-mem=10.00MB | | | | tuple-ids=25 row-size=29B cardinality=569 | | | | | | | 19:SCAN HDFS [mobistat.allstats_packet, RANDOM] | | | partitions=1/1 files=6 size=9.32MB | | | predicates: bbdo_approved = 1, campaign = '17D1' | | | table stats: 51137 rows total | | | column stats: all | | | hosts=5 per-host-mem=32.00MB | | | tuple-ids=24 row-size=53B cardinality=1065 | | | | | 18:SCAN HDFS [mobistat.cdr_packet a, RANDOM] | | partitions=4680/10328 files=4681 size=4.70GB | | predicates: regexp_like(calldirection, 'HTTPDL_CAPACITY') = TRUE, regexp_like(endresult, 'HTTP SUCCESS') = TRUE, (modpctlte + isnull(modpctlteca, 0)) > 0.999 | | table stats: 58699689 rows total | | column stats: all | | hosts=10 per-host-mem=304.00MB | | tuple-ids=23 row-size=252B cardinality=2888511 | | | 52:EXCHANGE [HASH(a.campaign,a.carrier,a.market,a.filename)] | | hosts=10 per-host-mem=0B | | tuple-ids=22 row-size=187B cardinality=23574499 | | | 17:SCAN HDFS [mobistat.psr_packet_gsm a, RANDOM] | partitions=2336/6581 files=2336 size=48.59GB | predicates: regexp_like(SESSIONTYPE, 'HTTPDL_CAPACITY') = TRUE, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END IS NOT NULL | table stats: 636668107 rows total | column stats: all | hosts=10 per-host-mem=608.00MB | tuple-ids=22 row-size=187B cardinality=23574499 | 48:AGGREGATE [FINALIZE] | output: sum:merge(carrier_count * samples), sum:merge(samples), sum:merge(sum_256qam), sum:merge(sum_total_frames), sum:merge(sum_total_bandwidth), sum:merge(sum_4tx_samples) | group by: campaign, market, carrier, sessiontype | hosts=10 per-host-mem=10.00MB | tuple-ids=20 row-size=109B cardinality=26977 | 47:EXCHANGE [HASH(campaign,market,carrier,sessiontype)] | hosts=10 per-host-mem=0B | tuple-ids=20 row-size=109B cardinality=26977 | 15:AGGREGATE [STREAMING] | output: sum(carrier_count * samples), sum(samples), sum(sum_256qam), sum(sum_total_frames), sum(sum_total_bandwidth), sum(sum_4tx_samples) | group by: campaign, market, carrier, sessiontype | hosts=10 per-host-mem=10.00MB | tuple-ids=20 row-size=109B cardinality=26977 | 00:UNION | hosts=10 per-host-mem=0B | tuple-ids=18 row-size=102B cardinality=26977 | |--46:AGGREGATE [FINALIZE] | | output: sum:merge(l_dlnum256qam), sum:merge(CAST(total_frames AS FLOAT)), sum:merge(ltebwcum), sum:merge(CASE WHEN l_dlmaxnumlayer = 4 THEN 1 ELSE 0 END), count:merge(*) | | group by: campaign, market, carrier, sessiontype, carrier_count, tech_mode | | hosts=10 per-host-mem=10.00MB | | tuple-ids=16 row-size=118B cardinality=13400 | | | 45:EXCHANGE [HASH(campaign,market,carrier,sessiontype,carrier_count,tech_mode)] | | hosts=10 per-host-mem=0B | | tuple-ids=16 row-size=118B cardinality=13400 | | | 14:AGGREGATE [STREAMING] | | output: sum(l_dlnum256qam), sum(CAST((l_dlnum256qam + l_dlnum64qam + l_dlnum16qam + l_dlnumqpsk) AS FLOAT)), sum(ltebwcum), sum(CASE WHEN a.l_dlmaxnumlayer = 4 THEN 1 ELSE 0 END), count(*) | | group by: a.campaign, a.market, a.carrier, CASE WHEN SESSIONTYPE = 'HTTPDL_CAPACITY_L' THEN 'HTTPDL_CAPACITY' ELSE SESSIONTYPE END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 4 WHEN l_pdschbytes_scc2 > 0 THEN 3 WHEN l_pdschbytes_scc1 > 0 THEN 2 WHEN L_pdschbytes > 0 THEN 1 ELSE 0 END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END | | hosts=10 per-host-mem=10.00MB | | tuple-ids=16 row-size=118B cardinality=13400 | | | 13:HASH JOIN [INNER JOIN, PARTITIONED] | | hash predicates: a.campaign = campaign, a.carrier = carrier, a.market = a.market, a.filename = filename | | other predicates: unix_timestamp(udfs.totimestamp(a.time_stamp)) <= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeend))), unix_timestamp(udfs.totimestamp(a.time_stamp)) >= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeinit))) | | runtime filters: RF005 <- campaign, RF006 <- carrier, RF007 <- a.market, RF008 <- filename | | hosts=10 per-host-mem=1018.71KB | | tuple-ids=9,10,12 row-size=488B cardinality=13400 | | | |--44:EXCHANGE [HASH(campaign,carrier,a.market,filename)] | | | hosts=10 per-host-mem=0B | | | tuple-ids=10,12 row-size=281B cardinality=33804 | | | | | 12:HASH JOIN [INNER JOIN, BROADCAST] | | | hash predicates: a.market = market | | | hosts=10 per-host-mem=17.62KB | | | tuple-ids=10,12 row-size=281B cardinality=33804 | | | | | |--42:EXCHANGE [BROADCAST] | | | | hosts=5 per-host-mem=0B | | | | tuple-ids=12 row-size=29B cardinality=569 | | | | | | | 41:AGGREGATE [FINALIZE] | | | | group by: market | | | | hosts=5 per-host-mem=10.00MB | | | | tuple-ids=12 row-size=29B cardinality=569 | | | | | | | 40:EXCHANGE [HASH(market)] | | | | hosts=5 per-host-mem=0B | | | | tuple-ids=12 row-size=29B cardinality=569 | | | | | | | 11:AGGREGATE [STREAMING] | | | | group by: market | | | | hosts=5 per-host-mem=10.00MB | | | | tuple-ids=12 row-size=29B cardinality=569 | | | | | | | 10:SCAN HDFS [mobistat.allstats_packet, RANDOM] | | | partitions=1/1 files=6 size=9.32MB | | | predicates: bbdo_approved = 1, campaign = '17D1' | | | table stats: 51137 rows total | | | column stats: all | | | hosts=5 per-host-mem=32.00MB | | | tuple-ids=11 row-size=53B cardinality=1065 | | | | | 09:SCAN HDFS [mobistat.cdr_packet a, RANDOM] | | partitions=4680/10328 files=4681 size=4.70GB | | predicates: regexp_like(calldirection, 'HTTPDL_CAPACITY') = TRUE, regexp_like(endresult, 'HTTP SUCCESS') = TRUE, (modpctlte + isnull(modpctlteca, 0)) > 0.999 | | table stats: 58699689 rows total | | column stats: all | | hosts=10 per-host-mem=304.00MB | | tuple-ids=10 row-size=252B cardinality=2888511 | | | 43:EXCHANGE [HASH(a.campaign,a.carrier,a.market,a.filename)] | | hosts=10 per-host-mem=0B | | tuple-ids=9 row-size=207B cardinality=23267425 | | | 08:SCAN HDFS [mobistat.psr_packet_cdma a, RANDOM] | partitions=2332/3707 files=2332 size=45.71GB | predicates: regexp_like(SESSIONTYPE, 'HTTPDL_CAPACITY') = TRUE, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END IS NOT NULL | runtime filters: RF005 -> a.campaign, RF006 -> a.carrier, RF007 -> a.market, RF008 -> a.filename | table stats: 358488531 rows total | column stats: all | hosts=10 per-host-mem=608.00MB | tuple-ids=9 row-size=207B cardinality=23267425 | 39:AGGREGATE [FINALIZE] | output: sum:merge(l_dlnum256qam), sum:merge(CAST(total_frames AS FLOAT)), sum:merge(ltebwcum), sum:merge(CASE WHEN l_dlmaxnumlayer = 4 THEN 1 ELSE 0 END), count:merge(*) | group by: campaign, market, carrier, sessiontype, carrier_count, tech_mode | hosts=10 per-host-mem=10.00MB | tuple-ids=7 row-size=118B cardinality=13577 | 38:EXCHANGE [HASH(campaign,market,carrier,sessiontype,carrier_count,tech_mode)] | hosts=10 per-host-mem=0B | tuple-ids=7 row-size=118B cardinality=13577 | 07:AGGREGATE [STREAMING] | output: sum(l_dlnum256qam), sum(CAST((l_dlnum256qam + l_dlnum64qam + l_dlnum16qam + l_dlnumqpsk) AS FLOAT)), sum(ltebwcum), sum(CASE WHEN a.l_dlmaxnumlayer = 4 THEN 1 ELSE 0 END), count(*) | group by: a.campaign, a.market, a.carrier, CASE WHEN SESSIONTYPE = 'HTTPDL_CAPACITY_L' THEN 'HTTPDL_CAPACITY' ELSE SESSIONTYPE END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 4 WHEN l_pdschbytes_scc2 > 0 THEN 3 WHEN l_pdschbytes_scc1 > 0 THEN 2 WHEN L_pdschbytes > 0 THEN 1 ELSE 0 END, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END | hosts=10 per-host-mem=10.00MB | tuple-ids=7 row-size=118B cardinality=13577 | 06:HASH JOIN [INNER JOIN, PARTITIONED] | hash predicates: a.campaign = campaign, a.carrier = carrier, a.market = a.market, a.filename = filename | other predicates: unix_timestamp(udfs.totimestamp(a.time_stamp)) <= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeend))), unix_timestamp(udfs.totimestamp(a.time_stamp)) >= unix_timestamp(udfs.totimestamp(concat(task_date, ' ', timeinit))) | runtime filters: RF000 <- campaign, RF001 <- carrier | hosts=10 per-host-mem=1018.71KB | tuple-ids=0,1,3 row-size=488B cardinality=13577 | |--37:EXCHANGE [HASH(campaign,carrier,a.market,filename)] | | hosts=10 per-host-mem=0B | | tuple-ids=1,3 row-size=281B cardinality=33804 | | | 05:HASH JOIN [INNER JOIN, BROADCAST] | | hash predicates: a.market = market | | hosts=10 per-host-mem=17.62KB | | tuple-ids=1,3 row-size=281B cardinality=33804 | | | |--35:EXCHANGE [BROADCAST] | | | hosts=5 per-host-mem=0B | | | tuple-ids=3 row-size=29B cardinality=569 | | | | | 34:AGGREGATE [FINALIZE] | | | group by: market | | | hosts=5 per-host-mem=10.00MB | | | tuple-ids=3 row-size=29B cardinality=569 | | | | | 33:EXCHANGE [HASH(market)] | | | hosts=5 per-host-mem=0B | | | tuple-ids=3 row-size=29B cardinality=569 | | | | | 04:AGGREGATE [STREAMING] | | | group by: market | | | hosts=5 per-host-mem=10.00MB | | | tuple-ids=3 row-size=29B cardinality=569 | | | | | 03:SCAN HDFS [mobistat.allstats_packet, RANDOM] | | partitions=1/1 files=6 size=9.32MB | | predicates: bbdo_approved = 1, campaign = '17D1' | | table stats: 51137 rows total | | column stats: all | | hosts=5 per-host-mem=32.00MB | | tuple-ids=2 row-size=53B cardinality=1065 | | | 02:SCAN HDFS [mobistat.cdr_packet a, RANDOM] | partitions=4680/10328 files=4681 size=4.70GB | predicates: regexp_like(calldirection, 'HTTPDL_CAPACITY') = TRUE, regexp_like(endresult, 'HTTP SUCCESS') = TRUE, (modpctlte + isnull(modpctlteca, 0)) > 0.999 | table stats: 58699689 rows total | column stats: all | hosts=10 per-host-mem=304.00MB | tuple-ids=1 row-size=252B cardinality=2888511 | 36:EXCHANGE [HASH(a.campaign,a.carrier,a.market,a.filename)] | hosts=10 per-host-mem=0B | tuple-ids=0 row-size=207B cardinality=23574499 | 01:SCAN HDFS [mobistat.psr_packet_gsm a, RANDOM] partitions=2336/6581 files=2336 size=48.59GB predicates: regexp_like(SESSIONTYPE, 'HTTPDL_CAPACITY') = TRUE, CASE WHEN l_pdschbytes_scc3 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc2 > 0 THEN 'LTECA' WHEN l_pdschbytes_scc1 > 0 THEN 'LTECA' WHEN L_pdschbytes > 0 THEN 'LTE' ELSE NULL END IS NOT NULL runtime filters: RF000 -> a.campaign, RF001 -> a.carrier table stats: 636668107 rows total column stats: all hosts=10 per-host-mem=608.00MB tuple-ids=0 row-size=207B cardinality=23574499
... View more
05-25-2017
09:38 AM
Hey Guys - I am using CDH5.10.1 and noticed the exact same error. In our case, Required mem_limit was 686MB and we gave it 3gb. At the time, when this query was running, there was no other query on the coordinator. So its quite confusing that it gives this error. Please let me know, if anyone of you had figured out a solution to this problem.
... View more
05-23-2017
01:54 PM
1 Kudo
We are using CDH 5.10.1, and notice frequent exceptions in HUE, when running Impala Queries in Notbook. Results Expired. Rerun quries to get results. Any idea, why this is happening, and what can be done to resolve it? Our Hive Metastore is PostgreSQL
... View more
04-18-2017
02:40 PM
We are running Impala v2.7.0-cdh5.10.1 (876895d), and are facing similar issue. While running alter table <table-name> recover partitions; - we get "Error communicating with impalad: TSocket read 0 bytes" error message. after multiple attempts to recover this table, we were finally able to recover partitions on the table. At which point, we ran "compute stats <table-name>;" and received this error message: CatalogException: Table was modified during stats computation. Although both these errors may be mutually exclusive, somehow they seems to only happen on the two tables with over 10,000 partition files.
... View more
Labels:
02-21-2017
12:34 PM
That was my first instint that too, that We are not passing hive-site.xml properly. We had the same issue back when we were using CDH 5.7.2 - and passing hive-site.xml from HDFS as --file in SPARK OPTIONS fixed it. Here in CDH 5.10, we are passing hive-site.xml as --file in SPARK OPTION, also we are attaching it under the new FILE attribute on Oozie Spark-Action. If its still the hive-site.xml, I am not sure why it would think its still missing when its in the SPARK OPTION. Any suggestions on how to troubleshoot this?
... View more
02-21-2017
10:56 AM
1 Kudo
On our cluster, Hive Metastore is on a External MySQL Database. So Oozie Spark-Submit should be looking for a mysql driver to connect to the metastore via thrift store connection. However, for some reason its looking for derby jar. Upon investigating, I noticed that derby jar is not begin passed with the spark-submit. I resolved the issue, by attaching derby jar with the SPARK ACTION. bizarde but this seems to resolve it. I am going to dig into the source code, to figure out the underlying problem. Will update this post, if and when, I find something.
... View more
02-21-2017
08:28 AM
1 Kudo
We are running CDH 5.10 with Impala and Spark. Our ETL uses HiveSQLContext, and has no issues executing as Spark-submit. However, when we try to use the same ETL Jar from Oozie Spark Action - we get the following error message. We had the same issue with CDH 5.7.2, at which point - we passed the hive-site.xml as --file argument in the Spark Action on Oozie and it worked file. However, the same does not seem to work with CDH 5.10. I did notice that oozie Spark Action was updated to have FILE field. So I removed the "--file" and attached hive-site.xml in the FILE field. This did not help out. Any suggestion on how to resolve this in CDH 5.10? 7/02/21 16:16:05 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
17/02/21 16:16:05 WARN Hive: Failed to register all functions.
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1530)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3230)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3249)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3474)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:209)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:332)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:293)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:268)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:529)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:220)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:210)
at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:442)
at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
at MobistatPSRETL$.main(MobistatPSRETL.scala:90)
at MobistatPSRETL.main(MobistatPSRETL.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1528)
... 32 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:781)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:411)
at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:440)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:335)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:291)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:57)
at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:648)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:626)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:679)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5995)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:203)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
... 37 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:281)
at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:239)
at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:292)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1069)
at org.datanucleus.NucleusContext.initialise(NucleusContext.java:359)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:768)
... 66 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("org.apache.derby.jdbc.EmbeddedDriver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:237)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:110)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
... 84 more
Caused by: org.datanucleus.store.rdbms.datasource.DatastoreDriverNotFoundException: The specified datastore driver ("org.apache.derby.jdbc.EmbeddedDriver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.datasource.AbstractDataSourceFactory.loadDriver(AbstractDataSourceFactory.java:58)
at org.datanucleus.store.rdbms.datasource.BoneCPDataSourceFactory.makePooledDataSource(BoneCPDataSourceFactory.java:61)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:217)
... 86 more
17/02/21 16:16:05 ERROR ApplicationMaster: User class threw exception: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:556)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:220)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:210)
at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:442)
at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
at MobistatPSRETL$.main(MobistatPSRETL.scala:90)
at MobistatPSRETL.main(MobistatPSRETL.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:214)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:332)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:293)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:268)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:529)
... 21 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1530)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3230)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3249)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3474)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:209)
... 25 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1528)
... 32 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:781)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:411)
at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:440)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:335)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:291)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:57)
at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:648)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:626)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:679)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5995)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:203)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
... 37 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:281)
at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:239)
at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:292)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1069)
at org.datanucleus.NucleusContext.initialise(NucleusContext.java:359)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:768)
... 66 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("org.apache.derby.jdbc.EmbeddedDriver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:237)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:110)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
... 84 more
Caused by: org.datanucleus.store.rdbms.datasource.DatastoreDriverNotFoundException: The specified datastore driver ("org.apache.derby.jdbc.EmbeddedDriver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.datasource.AbstractDataSourceFactory.loadDriver(AbstractDataSourceFactory.java:58)
at org.datanucleus.store.rdbms.datasource.BoneCPDataSourceFactory.makePooledDataSource(BoneCPDataSourceFactory.java:61)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:217)
... 86 more
... View more
02-08-2017
07:31 AM
We got ODBC Connection working with Kerberos. However, JDBC has issues identifying the Kerberos Principal. We thought about investigating the JDBC Connector source code, but other issues took priority. I have heard from other Big Data Engineerins thru meetups, that JDBC only works with username-password. For which, we need to setup LDAP Authentication for Hive and Impala. Before which, we need to setup cross-realm trust between LDAP/Active Directory and Kerberos - which prooves to be a pain by itself.
... View more
01-18-2017
07:06 AM
Thanks! We wrote a UDF, to handle this date convertion. It worked out well. Thanks, Krishna
... View more
10-25-2016
02:25 PM
I am not getting a whole different error. We are using a MySQL DB thats hosted seperatly, there is no need for it to try and connect to the host server. I checked the hue.ini file in the HUE_HOME directory under /var/run/cloudera-scm-agent/process/1166-hue-HUE_SERVER/, it does have the correct DATABASE settings. [root@ip-10-0-0-188 hue]# build/env/bin/hue useradmin_sync_with_unix Traceback (most recent call last): File "build/env/bin/hue", line 12, in <module> load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry execute_from_command_line(sys.argv) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line utility.execute() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/base.py", line 242, in run_from_argv self.execute(*args, **options.__dict__) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/base.py", line 285, in execute output = self.handle(*args, **options) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/useradmin/src/useradmin/management/commands/useradmin_sync_with_unix.py", line 47, in handle sync_unix_users_and_groups(min_uid, max_uid, min_gid, max_gid, check_shell) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/useradmin/src/useradmin/views.py", line 673, in sync_unix_users_and_groups hue_group = Group.objects.get(name=name) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/manager.py", line 151, in get return self.get_queryset().get(*args, **kwargs) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 304, in get num = len(clone) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 77, in __len__ self._fetch_all() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 857, in _fetch_all self._result_cache = list(self.iterator()) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 220, in iterator for row in compiler.results_iter(): File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/sql/compiler.py", line 713, in results_iter for rows in self.execute_sql(MULTI): File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/sql/compiler.py", line 785, in execute_sql cursor = self.connection.cursor() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 162, in cursor cursor = util.CursorWrapper(self._cursor(), self) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 132, in _cursor self.ensure_connection() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 127, in ensure_connection self.connect() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/utils.py", line 99, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 127, in ensure_connection self.connect() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 115, in connect self.connection = self.get_new_connection(conn_params) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/mysql/base.py", line 435, in get_new_connection conn = Database.connect(**conn_params) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg/MySQLdb/__init__.py", line 81, in Connect return Connection(*args, **kwargs) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg/MySQLdb/connections.py", line 193, in __init__ super(Connection, self).__init__(*args, **kwargs2) django.db.utils.OperationalError: (1045, "Access denied for user 'hue'@'10.0.0.188' (using password: YES)")
... View more
10-24-2016
05:49 AM
JDBC Kerberos Ticket. Let me know, if you have any insight http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/impala-kerberosed-jdbc-connection-from-SQL-Workbench-on-Windows/m-p/46415#M2142
... View more
10-21-2016
09:00 AM
Hello - I get the following error, when i try to sync unix accounts with HUE [root@ip-10-0-0-188 hue]# build/env/bin/hue useradmin_sync_with_unix Traceback (most recent call last): File "build/env/bin/hue", line 12, in <module> load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry execute_from_command_line(sys.argv) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line utility.execute() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command commands = get_commands() File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands apps = settings.INSTALLED_APPS File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__ self._setup(name) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup self._wrapped = Settings(settings_module) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__ mod = importlib.import_module(self.SETTINGS_MODULE) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module __import__(name) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/settings.py", line 242, in <module> conf.initialize(_desktop_conf_modules, _config_dir) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/lib/conf.py", line 591, in initialize conf_data = load_confs(_configs_from_dir(config_dir)) File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/lib/conf.py", line 523, in load_confs for in_conf in conf_source: File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/lib/conf.py", line 499, in _configs_from_dir for filename in sorted(os.listdir(conf_dir)): OSError: [Errno 36] File name too long: '/var/run/cloudera-scm-agent/process/total 0\ndrwxr-xr-x 4 root root 100 Oct 21 11:55 ccdeploy_etc_-7499926885451336301\ndrwxr-x--x 3 zookeeper zookeeper 280 Oct 21 12:09 932-zookeeper-server\ndrwxr-x--x 3 hdfs hdfs 500 Oct 21 12:09 942-hdfs-NAMENODE\ndrwxr-x--x 3 sentry sentry 280 Oct 21 12:10 944-sentry-SENTRY_SERVER\ndrwxr-x--x 3 mapred hadoop 380 Oct 21 12:10 952-yarn-JOBHISTORY\ndrwxr-x--x 3 yarn hadoop 520 Oct 21 12:10 955-yarn-RESOURCEMANAGER\ndrwxr-x--x 5 sqoop2 sqoop 280 Oct 21 12:11 957-sqoop-SQOOP_SERVER\ndrwxr-x--x 11 root root 220 Oct 21 12:12 .\ndrwxr-x--x 6 hue hue 320 Oct 21 12:12 973-hue-HUE_SERVER\ndrwxr-x--x 4 hue hue 260 Oct 21 12:12 974-hue-KT_RENEWER\ndrwxr-xr-x 6 cloudera-scm cloudera-scm 180 Oct 21 15:51 ..' [root@ip-10-0-0-188 hue]#
... View more
10-21-2016
07:33 AM
It would be great if JDBC connection to a Kerberosed Cluster worked. I opened a seperate ticket on that, lets see where that goes.
... View more
10-18-2016
07:35 AM
Yes, I am KINITed. From the MIT Kerberos Desktop client, I can see both the Kerberos Tocket and the Windows Domain Account ticket. I also went to command prompt and verified that, C:\Program Files\MIT\Kerberos\bin\klist shows the ticket.
... View more
10-14-2016
03:21 PM
Hey Guys - we have data, where the timestamp field is of datatype string, the values are non-zero padded timestamps. For Example: 1/1/2015 1:34:45 PM 1/10/2014 1:02:45 AM 11/1/2014 11:04:45 AM When we do a cast as timestamp, we get NULL result - reason begin, impala expects the day and month values to be 2 digit, that is zero-padded. Is there a Regex function that will allow me to parse these non-zero padded timestamp to zero-padded and then convert to timestamp datatype?
... View more
10-13-2016
08:21 AM
I strongly recommend, to tune YARN memory configurations first, before changing anything else. This helped solve alot of problems at our end. https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html Second, if you are using YARN Admision Control - then ideal Default Query Memory Limit and set it. The smaller the number, more concurrent queries.
... View more
10-13-2016
08:16 AM
Check the Impala Daemon Mem Limit from Cloudera Manager. What is it set at? I have noticed that, if the MEM_LIMIT for Daemon is 8GB. And if the per-host mem cost is larger than the set 8GB, I get a similar memory error message. The ideal solution is to bump up the Mem Limit per Daemon, and if you have Yarn Admission Control enabled, you will need to adjust those values too.
... View more
10-13-2016
08:12 AM
We are running a big query on Impala Tables. Once the results appear, when we try to export the results to CSV, we get the following error page. Any ideas, on what could be wrong here? We are using CDH 5.7.2 - Impala, Spark on YARN, HUE along with HDFS. Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1215, in communicate
req.respond()
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 576, in respond
self._respond()
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 590, in _respond
for chunk in response:
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/six.py", line 535, in next
return type(self).__next__(self)
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/http/response.py", line 292, in __next__
return self.make_bytes(next(self._iterator))
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/desktop/core/src/desktop/lib/export_csvxls.py", line 86, in create_generator
for headers, data in content_generator:
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/beeswax/src/beeswax/data_export.py", line 71, in HS2DataAdapter
results = db.fetch(handle, start_over=start_over, rows=FETCH_SIZE)
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 282, in fetch
return self.client.fetch(query_handle, start_over, rows)
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 1035, in fetch
data_table = self._client.fetch_data(operationHandle, orientation=orientation, max_rows=max_rows)
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 786, in fetch_data
results, schema = self.fetch_result(operation_handle, orientation, max_rows)
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 819, in fetch_result
schema = self.call(self._client.GetResultSetMetadata, meta_req)
File "/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 642, in call
raise QueryServerException(Exception('Bad status for request %s:\n%s' % (req, res)), message=message)
QueryServerException: Bad status for request TGetResultSetMetadataReq(operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\n%?\xfe\x0f\x0eFR\x88T\x16iM\xe2(\x03', guid='\n%?\xfe\x0f\x0eFR\x88T\x16iM\xe2(\x03'))):
TGetResultSetMetadataResp(status=TStatus(errorCode=None, errorMessage='Invalid query handle', sqlState='HY000', infoMessages=None, statusCode=3), schema=None)
... View more
10-12-2016
02:55 PM
This is what I see in the JDBC Trace Log Files. Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.DSIConnection(com.cloudera.impala.hivecommon.core.HiveJDBCEnvironment@1bbd56aa): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(101, Variant[type: TYPE_WSTRING, value: ImpalaJDBC]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(139, Variant[type: TYPE_WSTRING, value: User]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(22, Variant[type: TYPE_WSTRING, value: Impala]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(58, Variant[type: TYPE_WSTRING, value: `]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(66, Variant[type: TYPE_UINT16, value: -1]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(68, Variant[type: TYPE_UINT16, value: -1]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(76, Variant[type: TYPE_UINT16, value: -1]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(81, Variant[type: TYPE_UINT16, value: -1]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(83, Variant[type: TYPE_UINT16, value: -1]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.setProperty(80, Variant[type: TYPE_WSTRING, value: N]): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.dsi.core.impl.DSIConnection.registerWarningListener(com.cloudera.impala.jdbc.common.SWarningListener@157abbad): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.hivecommon.core.HiveJDBCConnection.updateConnectionSettings(): +++++ enter +++++ Oct 12 17:50:52.989 TRACE 27 com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect({AuthMech=Variant[type: TYPE_WSTRING, value: 1], ConnSchema=Variant[type: TYPE_WSTRING, value: NULL], DatabaseType=Variant[type: TYPE_WSTRING, value: Impala], HiveServerType=Variant[type: TYPE_WSTRING, value: 2], Host=Variant[type: TYPE_WSTRING, value: gateway.mygws.com], KrbHostFQDN=Variant[type: TYPE_WSTRING, value: ip-10-0-0-186.ec2.internal], KrbRealm=Variant[type: TYPE_WSTRING, value: MOBISTAT], KrbServiceName=Variant[type: TYPE_WSTRING, value: impala], LogLevel=Variant[type: TYPE_WSTRING, value: 6], LogPath=Variant[type: TYPE_WSTRING, value: D:\Mobistat\Log\], Port=Variant[type: TYPE_WSTRING, value: 21051], Principal=Variant[type: TYPE_WSTRING, value: krishnat/ip-10-0-0-186@MOBISTAT]}): +++++ enter +++++ Oct 12 17:50:52.989 ERROR 27 com.cloudera.impala.exceptions.ExceptionConverter.toSQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to obtain Principal Name for authentication ; java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to obtain Principal Name for authentication ; at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:224) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52) at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597) at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219) at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216) at workbench.db.DbDriver.connect(DbDriver.java:513) at workbench.db.ConnectionMgr.connect(ConnectionMgr.java:244) at workbench.db.ConnectionMgr.getConnection(ConnectionMgr.java:172) Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to obtain Principal Name for authentication ; ... 8 more Caused by: javax.security.auth.login.LoginException: Unable to obtain Principal Name for authentication at com.sun.security.auth.module.Krb5LoginModule.promptForName(Unknown Source) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source) at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at javax.security.auth.login.LoginContext.invoke(Unknown Source) at javax.security.auth.login.LoginContext.access$000(Unknown Source) at javax.security.auth.login.LoginContext$4.run(Unknown Source) at javax.security.auth.login.LoginContext$4.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(Unknown Source) at javax.security.auth.login.LoginContext.login(Unknown Source) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:113) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52) at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597) at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219) at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216) at workbench.db.DbDriver.connect(DbDriver.java:513) at workbench.db.ConnectionMgr.connect(ConnectionMgr.java:244) at workbench.db.ConnectionMgr.getConnection(ConnectionMgr.java:172) at workbench.gui.profiles.ConnectionGuiHelper$1.run(ConnectionGuiHelper.java:104) I checked very worker node, did not see any errors in the Impala Daemon Logs.
... View more
09-30-2016
11:41 AM
We have a 15 Node Kerborised Impala Cluster with a HAProxy. We have no issue using HUE to run queries. We are also able to use the ODBC Driver on a Windows Machine, authenticate with Kerberos and connect to the Impala via HA Proxy. However, when we try to connect to the Impala HA Proxy using SQL Workbench via JDBC Driver. We get the following error message: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to obtain Principal Name for authentication ; The connection string is: jdbc:impala://<PUBLIC IP ADDRESS>:21051;AuthMech=1;KrbRealm=<REALM>;KrbHostFQDN=<fqdn>;KrbServiceName=impala; We tried adding the Principal parameter, but it doesn't help. Any ideas, on how to get Impala JDBC to work from a windows machine using Kerberos?
... View more
09-14-2016
01:50 PM
We currently have CDH 5.7.2 Cluster, and want to upgrade to 5.8. Is there a documentation, that we can use to do this transistion?
... View more
08-24-2016
12:37 PM
All that begin said, when we go into production - we want to have a PUBLIC FQDN for the manager, edge nodes - i am not sure, how to go about setting it up with TLS - with all this internal ip conflicts going on. Any one else faced similar issues with setting up CLOUDERA on AWS?
... View more
08-24-2016
12:35 PM
My setup is on AWS. The AWS servers have a Internal IP, with a internal FQDN (ip-20-40-45.internal). For the purpose of TLS, We setup a ELASTIC PUBLIC IP, and assigned it a Public FQDN (say manager.example.com) When I generated the Self-Signed certificate, i used the new public FQDN ( manager.example.com) With the "Use TLS Encryption for Admin Console" - I was successful in running the admin console over SSL. I then proceeded to enabling "TLS" for the Agents. This is where i faced problems with my setup. The agents are communicating with the server using the INTERNAL IP, where as the certificate is for EXTERNAL IP. So I changed the CM IP in /etc/cloudera-smc-agent/config.ini to manager.example.com. It still didn't help, somewhere its still trying to connect to the INTERNAL IP. So I figured, why not generate the certificate usign the INTERNAL FQDN (ip-20-40-45.internal) - I repeated all the steps, and it works fine now. Now, all the agents are connecting to the Manager Service Server over TLS.
... View more
08-23-2016
11:56 AM
I am trying to setup Kerberos on CDH 5.7.2 - Cloudera Express for our development environment. I am following the steps in : https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_sg_intro_kerb.html Apart of the requirement, is to install TLS Level 1 , before enabling Kerberos. I am using a self signed certificate, so I followed the steps from this link (https://www.cloudera.com/documentation/enterprise/5-7-x/topics/sg_self_signed_tls.html#xd_583c10bfdbd) and then followed STEP 2,3 from (https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_sg_tls_browser.html#xd_583c10bfdbd326ba-7dae4aa6-147c30d0933--7a61) The management services do not start, I am not sure why its failing - any advice on how to troubleshoot this? P.S: If I un-tick Run Manager over TLS, the management services comes up. So the problem seems to be with the TLS setup.
... View more
08-15-2016
01:29 PM
Is it recommended to install HAProxy on the Gateway node, or should be it a dedicated server by itself? If so, is there a sepecific memory requirement? I would like to have this on AWS
... View more
04-07-2016
04:15 PM
2 Kudos
Figured out the issue. I had set hive.vectorized.execution.enabled = true; But Binary datatype is not supported by verctors. Once I disabled it, the query works.
... View more
04-07-2016
03:19 PM
create table gridList(Operator string, GridID int, MaxLat double, MinLat double, MaxLon double, MinLon double, MaxPoint binary, MINPoint binary)
STORED AS ORC; create table sedbWithPoly (
marketcode String, operator String, drivename String, fips int, SiteLocationID int, latitude double, longitude double, band string, channel string,
polygon1 binary, polygon2 binary, polygon3 binary,found boolean)
PARTITIONED BY (marketPart String)
STORED AS ORC; QUERY: SELECT *
FROM gridList g JOIN sedbWithPoly s
ON g.Operator = s.Operator
WHERE ST_Contains(s.polygon1, g.MaxPoint) OR ST_Contains(s.polygon1, g.MINPoint)
AND s.marketPart = 'Aberdeen WA';
... View more
