Created on 02-09-2019 01:16 AM - edited 09-16-2022 07:08 AM
HI all, been trying to get Hive/Druid/Kafka working (noob to both hive/druid)..'getting close I think' I have gotten to the point of being able load data via kafka->druid I have been able to create 'external druid' tables in hive, and even 'query', such that I can get accurate row counts. but when ever I actually query any columns..I get null back for all columns (except the __time).
This entire stack was setup via: HDP 3, onto 3 boxes.
CREATE EXTERNAL TABLE teststuff5 STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler' TBLPROPERTIES ("druid.datasource" = "teststuff5");
Queries are coming back like (we see we have a non null time value..and for some reason one other column is non null):
| teststuff5.__time | teststuff5.accepteddate | teststuff5.accountid | teststuff5.accountname | teststuff5.apiversion | teststuff5.carrierid | teststuff5.carriername | teststuff5.companyid | teststuff5.contentencoding | teststuff5.countrycode | teststuff5.countryname | teststuff5.delivereddate | teststuff5.destinationaddress | teststuff5.internalmessageid | teststuff5.liveoperatorlookup | teststuff5.messageid | teststuff5.messageoriginator | teststuff5.messageoriginatorton | teststuff5.messagestatus | teststuff5.messagetype | teststuff5.mtscore | teststuff5.parentmessageid | teststuff5.phonenumber | teststuff5.productid | teststuff5.productiddescription | teststuff5.productname | teststuff5.productsubtype | teststuff5.programid | teststuff5.remoteipaddress | teststuff5.remoteresponsecode | teststuff5.responsecode | teststuff5.responsecodedescription | teststuff5.sourceaddress | teststuff5.subaccount | teststuff5.updateddate | teststuff5.useragent | teststuff5.userdataheader | teststuff5.userdefined1 | teststuff5.userdefined2 | +----------------------------+--------------------------+-----------------------+-------------------------+------------------------+-----------------------+-------------------------+-----------------------+-----------------------------+-------------------------+-------------------------+---------------------------+--------------------------------+-------------------------------+--------------------------------+-----------------------+-------------------------------+----------------------------------+---------------------------+-------------------------+---------------------+-----------------------------+-------------------------+-----------------------+----------------------------------+-------------------------+----------------------------+-----------------------+-----------------------------+--------------------------------+--------------------------+-------------------------------------+---------------------------+-------------------------------------+-------------------------+-----------------------+----------------------------+--------------------------+--------------------------+ | 2019-02-05 18:53:00.0 UTC | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | ESA_SMS_DEVICE_NTERNAL | NULL | NULL | NULL | NULL | NULL | +----------------------------+--------------------------+-----------------------+-------------------------+------------------------+-----------------------+-------------------------+-----------------------+-----------------------------+-------------------------+-------------------------+---------------------------+--------------------------------+-------------------------------+--------------------------------+-----------------------+-------------------------------+----------------------------------+---------------------------+-------------------------+---------------------+-----------------------------+-------------------------+-----------------------+----------------------------------+-------------------------+----------------------------+-----------------------+-----------------------------+--------------------------------+--------------------------+-------------------------------------+---------------------------+-------------------------------------+-------------------------+-----------------------+----------------------------+--------------------------+--------------------------+
note: The same query on druid..shows all values..
Calling: : jdbc:hive2://wdc-tst-bdrd-001.openmarket.c> describe formatted teststuff5;
Gives:
INFO : Compiling command(queryId=hive_20190207234008_c536632e-547f-4843-96bc-e92b8577799d): describe formatted teststuff5 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, type:string, comment:from deserializer), FieldSchema(name:data_type, type:string, comment:from deserializer), FieldSchema(name:comment, type:string, comment:from deserializer)], properties:null) INFO : Completed compiling command(queryId=hive_20190207234008_c536632e-547f-4843-96bc-e92b8577799d); Time taken: 0.062 seconds INFO : Executing command(queryId=hive_20190207234008_c536632e-547f-4843-96bc-e92b8577799d): describe formatted teststuff5 INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20190207234008_c536632e-547f-4843-96bc-e92b8577799d); Time taken: 0.142 seconds INFO : OK +-------------------------------+----------------------------------------------------+----------------------------------------------------+ | col_name | data_type | comment | +-------------------------------+----------------------------------------------------+----------------------------------------------------+ | # col_name | data_type | comment | | __time | timestamp with local time zone | from deserializer | | accepteddate | string | from deserializer | | accountid | string | from deserializer | | accountname | string | from deserializer | | apiversion | string | from deserializer | | carrierid | string | from deserializer | | carriername | string | from deserializer | | companyid | string | from deserializer | | contentencoding | string | from deserializer | | countrycode | string | from deserializer | | countryname | string | from deserializer | | delivereddate | string | from deserializer | | destinationaddress | string | from deserializer | | internalmessageid | string | from deserializer | | liveoperatorlookup | string | from deserializer | | messageid | string | from deserializer | | messageoriginator | string | from deserializer | | messageoriginatorton | string | from deserializer | | messagestatus | string | from deserializer | | messagetype | string | from deserializer | | mtscore | string | from deserializer | | parentmessageid | string | from deserializer | | phonenumber | string | from deserializer | | productid | string | from deserializer | | productiddescription | string | from deserializer | | productname | string | from deserializer | | productsubtype | string | from deserializer | | programid | string | from deserializer | | remoteipaddress | string | from deserializer | | remoteresponsecode | string | from deserializer | | responsecode | string | from deserializer | | responsecodedescription | string | from deserializer | | sourceaddress | string | from deserializer | | subaccount | string | from deserializer | | updateddate | string | from deserializer | | useragent | string | from deserializer | | userdataheader | string | from deserializer | | userdefined1 | string | from deserializer | | userdefined2 | string | from deserializer | | | NULL | NULL | | # Detailed Table Information | NULL | NULL | | Database: | default | NULL | | OwnerType: | USER | NULL | | Owner: | admin | NULL | | CreateTime: | Thu Feb 07 23:06:21 UTC 2019 | NULL | | LastAccessTime: | UNKNOWN | NULL | | Retention: | 0 | NULL | | Location: | hdfs://wdc-tst-bdrd-001.openmarket.com:8020/warehouse/tablespace/external/hive/teststuff5 | NULL | | Table Type: | EXTERNAL_TABLE | NULL | | Table Parameters: | NULL | NULL | | | COLUMN_STATS_ACCURATE | {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"__time\":\"true\",\"accepteddate\":\"true\",\"accountid\":\"true\",\"accountname\":\"true\",\"apiversion\":\"true\",\"carrierid\":\"true\",\"carriername\":\"true\",\"companyid\":\"true\",\"contentencoding\":\"true\",\"countrycode\":\"true\",\"countryname\":\"true\",\"delivereddate\":\"true\",\"destinationaddress\":\"true\",\"internalmessageid\":\"true\",\"liveoperatorlookup\":\"true\",\"messageid\":\"true\",\"messageoriginator\":\"true\",\"messageoriginatorton\":\"true\",\"messagestatus\":\"true\",\"messagetype\":\"true\",\"mtscore\":\"true\",\"parentmessageid\":\"true\",\"phonenumber\":\"true\",\"productid\":\"true\",\"productiddescription\":\"true\",\"productname\":\"true\",\"productsubtype\":\"true\",\"programid\":\"true\",\"remoteipaddress\":\"true\",\"remoteresponsecode\":\"true\",\"responsecode\":\"true\",\"responsecodedescription\":\"true\",\"sourceaddress\":\"true\",\"subaccount\":\"true\",\"updateddate\":\"true\",\"useragent\":\"true\",\"userdataheader\":\"true\",\"userdefined1\":\"true\",\"userdefined2\":\"true\"}} | | | EXTERNAL | TRUE | | | bucketing_version | 2 | | | druid.datasource | teststuff5 | | | numFiles | 0 | | | numRows | 0 | | | rawDataSize | 0 | | | storage_handler | org.apache.hadoop.hive.druid.DruidStorageHandler | | | totalSize | 0 | | | transient_lastDdlTime | 1549580781 | | | NULL | NULL | | # Storage Information | NULL | NULL | | SerDe Library: | org.apache.hadoop.hive.druid.serde.DruidSerDe | NULL | | InputFormat: | null | NULL | | OutputFormat: | null | NULL | | Compressed: | No | NULL | | Num Buckets: | -1 | NULL | | Bucket Columns: | [] | NULL | | Sort Columns: | [] | NULL | | Storage Desc Params: | NULL | NULL | | | serialization.format | 1 | +-------------------------------+----------------------------------------------------+----------------------------------------------------+
hiveserver2 logging sample
2019-02-08T18:45:21,640 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:21,640 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:21,642 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: operation.OperationManager (:()) - Adding operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=3b771b6b-7ff7-49dc-b212-dfc6b77211dc]<br>2019-02-08T18:45:21,646 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ql.Driver (:()) - Compiling command(queryId=hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d): select * from teststuff5 limit 1<br>2019-02-08T18:45:21,706 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: lockmgr.DbTxnManager (:()) - Opened txnid:185<br>2019-02-08T18:45:21,707 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Starting Semantic Analysis<br>2019-02-08T18:45:21,708 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Completed phase 1 of Semantic Analysis<br>2019-02-08T18:45:21,708 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for source tables<br>2019-02-08T18:45:21,733 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for subqueries<br>2019-02-08T18:45:21,733 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for destination tables<br>2019-02-08T18:45:21,745 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ql.Context (:()) - New scratch dir is hdfs://wdc-tst-bdrd-001:8020/tmp/hive/admin/8dea5cfb-f504-4237-a3b4-fb19d739b792/hive_2019-02-08_18-45-21_696_7674740746834465900-2<br>2019-02-08T18:45:21,745 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Completed getting MetaData in Semantic Analysis<br>2019-02-08T18:45:21,896 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for source tables<br>2019-02-08T18:45:21,907 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for subqueries<br>2019-02-08T18:45:21,907 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Get metadata for destination tables<br>2019-02-08T18:45:21,908 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ql.Context (:()) - New scratch dir is hdfs://wdc-tst-bdrd-001:8020/tmp/hive/admin/8dea5cfb-f504-4237-a3b4-fb19d739b792/hive_2019-02-08_18-45-21_696_7674740746834465900-2<br>2019-02-08T18:45:21,912 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it doesn't exist: hdfs://wdc-tst-bdrd-001:8020/tmp/hive/admin/8dea5cfb-f504-4237-a3b4-fb19d739b792/hive_2019-02-08_18-45-21_696_7674740746834465900-2/-mr-10001/.hive-staging_hive_2019-02-08_18-45-21_696_7674740746834465900-2<br>2019-02-08T18:45:21,917 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - CBO Succeeded; optimized logical plan.<br>2019-02-08T18:45:21,918 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ppd.OpProcFactory (:()) - Processing for FS(2)<br>2019-02-08T18:45:21,918 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ppd.OpProcFactory (:()) - Processing for SEL(1)<br>2019-02-08T18:45:21,918 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ppd.OpProcFactory (:()) - Processing for TS(0)<br>2019-02-08T18:45:21,921 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: parse.CalcitePlanner (:()) - Completed plan generation<br>2019-02-08T18:45:21,921 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ql.Driver (:()) - Semantic Analysis Completed (retrial = false)<br>2019-02-08T18:45:21,921 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ql.Driver (:()) - Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:teststuff5.__time, type:timestamp with local time zone, comment:null), FieldSchema(name:teststuff5.accepteddate, type:string, comment:null), FieldSchema(name:teststuff5.accountid, type:string, comment:null), FieldSchema(name:teststuff5.accountname, type:string, comment:null), FieldSchema(name:teststuff5.apiversion, type:string, comment:null), FieldSchema(name:teststuff5.carrierid, type:string, comment:null), FieldSchema(name:teststuff5.carriername, type:string, comment:null), FieldSchema(name:teststuff5.companyid, type:string, comment:null), FieldSchema(name:teststuff5.contentencoding, type:string, comment:null), FieldSchema(name:teststuff5.countrycode, type:string, comment:null), FieldSchema(name:teststuff5.countryname, type:string, comment:null), FieldSchema(name:teststuff5.delivereddate, type:string, comment:null), FieldSchema(name:teststuff5.destinationaddress, type:string, comment:null), FieldSchema(name:teststuff5.internalmessageid, type:string, comment:null), FieldSchema(name:teststuff5.liveoperatorlookup, type:string, comment:null), FieldSchema(name:teststuff5.messageid, type:string, comment:null), FieldSchema(name:teststuff5.messageoriginator, type:string, comment:null), FieldSchema(name:teststuff5.messageoriginatorton, type:string, comment:null), FieldSchema(name:teststuff5.messagestatus, type:string, comment:null), FieldSchema(name:teststuff5.messagetype, type:string, comment:null), FieldSchema(name:teststuff5.mtscore, type:string, comment:null), FieldSchema(name:teststuff5.parentmessageid, type:string, comment:null), FieldSchema(name:teststuff5.phonenumber, type:string, comment:null), FieldSchema(name:teststuff5.productid, type:string, comment:null), FieldSchema(name:teststuff5.productiddescription, type:string, comment:null), FieldSchema(name:teststuff5.productname, type:string, comment:null), FieldSchema(name:teststuff5.productsubtype, type:string, comment:null), FieldSchema(name:teststuff5.programid, type:string, comment:null), FieldSchema(name:teststuff5.remoteipaddress, type:string, comment:null), FieldSchema(name:teststuff5.remoteresponsecode, type:string, comment:null), FieldSchema(name:teststuff5.responsecode, type:string, comment:null), FieldSchema(name:teststuff5.responsecodedescription, type:string, comment:null), FieldSchema(name:teststuff5.sourceaddress, type:string, comment:null), FieldSchema(name:teststuff5.subaccount, type:string, comment:null), FieldSchema(name:teststuff5.updateddate, type:string, comment:null), FieldSchema(name:teststuff5.useragent, type:string, comment:null), FieldSchema(name:teststuff5.userdataheader, type:string, comment:null), FieldSchema(name:teststuff5.userdefined1, type:string, comment:null), FieldSchema(name:teststuff5.userdefined2, type:string, comment:null)], properties:null)<br>2019-02-08T18:45:21,923 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: exec.TableScanOperator (:()) - Initializing operator TS[0]<br>2019-02-08T18:45:21,924 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: exec.SelectOperator (:()) - Initializing operator SEL[1]<br>2019-02-08T18:45:21,924 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: exec.SelectOperator (:()) - SELECT struct<__time:timestamp with local time zone,accepteddate:string,accountid:string,accountname:string,apiversion:string,carrierid:string,carriername:string,companyid:string,contentencoding:string,countrycode:string,countryname:string,delivereddate:string,destinationaddress:string,internalmessageid:string,liveoperatorlookup:string,messageid:string,messageoriginator:string,messageoriginatorton:string,messagestatus:string,messagetype:string,mtscore:string,parentmessageid:string,phonenumber:string,productid:string,productiddescription:string,productname:string,productsubtype:string,programid:string,remoteipaddress:string,remoteresponsecode:string,responsecode:string,responsecodedescription:string,sourceaddress:string,subaccount:string,updateddate:string,useragent:string,userdataheader:string,userdefined1:string,userdefined2:string><br>2019-02-08T18:45:21,924 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: exec.ListSinkOperator (:()) - Initializing operator LIST_SINK[3]<br>2019-02-08T18:45:21,925 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: ql.Driver (:()) - Completed compiling command(queryId=hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d); Time taken: 0.279 seconds<br>2019-02-08T18:45:21,925 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:21,927 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:21,931 INFO [HiveServer2-Background-Pool: Thread-4925]: reexec.ReExecDriver (:()) - Execution #1 of query<br>2019-02-08T18:45:21,931 INFO [HiveServer2-Background-Pool: Thread-4925]: lockmgr.DbTxnManager (:()) - Setting lock request transaction to txnid:185 for queryId=hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d<br>2019-02-08T18:45:22,009 INFO [HiveServer2-Background-Pool: Thread-4925]: ql.Driver (:()) - Executing command(queryId=hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d): select * from teststuff5 limit 1<br>2019-02-08T18:45:22,010 INFO [HiveServer2-Background-Pool: Thread-4925]: hooks.HiveProtoLoggingHook (:()) - Received pre-hook notification for: hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d<br>2019-02-08T18:45:22,016 INFO [HiveServer2-Background-Pool: Thread-4925]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,023 INFO [HiveServer2-Background-Pool: Thread-4925]: hooks.HiveProtoLoggingHook (:()) - Received post-hook notification for: hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d<br>2019-02-08T18:45:22,023 INFO [HiveServer2-Background-Pool: Thread-4925]: ql.Driver (:()) - Completed executing command(queryId=hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d); Time taken: 0.014 seconds<br>2019-02-08T18:45:22,023 INFO [HiveServer2-Background-Pool: Thread-4925]: ql.Driver (:()) - OK<br>2019-02-08T18:45:22,023 INFO [HiveServer2-Background-Pool: Thread-4925]: lockmgr.DbTxnManager (:()) - Stopped heartbeat for query: hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d<br>2019-02-08T18:45:22,064 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,064 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,064 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,065 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,152 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,152 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,152 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,152 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,177 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,177 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,177 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,177 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,182 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,182 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,186 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: pool.ChannelResourceFactory (:()) - Generating: http://wdc-tst-bdrd-001:8082<br>2019-02-08T18:45:22,211 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,210 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: exec.TableScanOperator (:()) - RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_TS_0:1, <br>2019-02-08T18:45:22,210 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: exec.SelectOperator (:()) - RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_SEL_1:1, <br>2019-02-08T18:45:22,210 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: exec.ListSinkOperator (:()) - RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_LIST_SINK_3:1, <br>2019-02-08T18:45:22,211 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,235 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,235 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,235 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,235 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,238 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,238 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,238 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,239 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,240 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,241 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,241 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,241 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,245 INFO [HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,245 INFO [HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Updating thread name to 8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127<br>2019-02-08T18:45:22,245 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: operation.OperationManager (:()) - Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=3b771b6b-7ff7-49dc-b212-dfc6b77211dc]<br>2019-02-08T18:45:22,245 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: operation.OperationManager (:()) - Removed queryId: hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=3b771b6b-7ff7-49dc-b212-dfc6b77211dc] with tag: null<br>2019-02-08T18:45:22,262 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: operation.Operation (:()) - Closing operation log /tmp/hive/operation_logs/8dea5cfb-f504-4237-a3b4-fb19d739b792/hive_20190208184521_4a573dee-6487-4e4e-a72f-83b343aed10d without delay<br>2019-02-08T18:45:22,262 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: conf.HiveConf (HiveConf.java:getLogIdVar(5130)) - Using the default value passed in for log id: 8dea5cfb-f504-4237-a3b4-fb19d739b792<br>2019-02-08T18:45:22,262 INFO [8dea5cfb-f504-4237-a3b4-fb19d739b792 HiveServer2-Handler-Pool: Thread-127]: session.SessionState (:()) - Resetting thread name to HiveServer2-Handler-Pool: Thread-127
Created 02-09-2019 02:00 AM
I bet your column names have upper cases in Druid tables, thus Hive will query the wrong columns since it is always down-casing the names