Sad to hear that 😞
I tried to create the same flow and it works as expected.
Here is what i tried..
i have create a table in hive bucketed,partitioned,orc format,transactional.
hive ddl:-
hive> create table default.tmp(id string,name string)
partitioned by(ts timestamp)
clustered by (id)
into 2 buckets
stored as orc
TBLPROPERTIES ("transactional"="true");hive> desc tmp; +--------------------------+-----------------------+-----------------------+--+ | col_name | data_type | comment | +--------------------------+-----------------------+-----------------------+--+ | id | string | | | name | string | | | ts | timestamp | | | | NULL | NULL | | # Partition Information | NULL | NULL | | # col_name | data_type | comment | | | NULL | NULL | | ts | timestamp | | +--------------------------+-----------------------+-----------------------+--+
i have created the tmp table with id,name columns as string types and partitioned by ts as timestamp type.
Now PutHiveStreaming Processor Configs:-
select data from tmp table now:-
hive> select * from tmp; +-----+-------+--------------------------+--+ | id | name | ts | +-----+-------+--------------------------+--+ | 1 | hdf | 2015-01-02 20:23:20.0 | | 1 | hdf | 2015-01-02 20:23:20.0 | | 1 | hdf | 2018-03-12 19:11:51.217 | | 1 | hdf | 2018-03-12 19:11:51.24 | +-----+-------+--------------------------+--+
hive> show partitions tmp; +---------------------------------+--+ | partition | +---------------------------------+--+ | ts=2015-01-02 20%3A23%3A20.0 | | ts=2018-03-12 19%3A11%3A51.217 | | ts=2018-03-12 19%3A11%3A51.24 | +---------------------------------+--+
For testing purpose use select hiveql processor with below configs:-
HiveQL Select Query
select int(1) id,string('hdf')name,timestamp(current_timestamp)ts
Output Format
Avro
Configs:-
Test Flow:-
try this flow once and make sure you are able to load using PutHiveStreaming processor into hive.once you are able to load the data into hive then try with your scenario.
If issue still exists then share your hive create table statement and 10 records of sample data so that we can recreate and help you out ASAP.
In addition if you don't want to create partitions then create table statement would be
hive> create table default.tmp(id string,name string,ts string) --with ts as timestamp type also works
clustered by (id)
into 2 buckets
stored as orc
TBLPROPERTIES ("transactional"="true");
You have to change the data types in your create table statement as table used by put hive streaming processor expects all the datatypes would be strings(except for timestamp..etc types).
use the below create table statement
CREATE TABLE default.purchase_acct_orc (
acct_num STRING,
pur_id STRING,
pur_det_id STRING,
product_pur_product_code STRING,
prod_amt STRING,
accttype_acct_type_code STRING,
accttype_acct_status_code STRING,
emp_emp_code STRING,
plaza_plaza_id STRING,
purstat_pur_status_code STRING)
PARTITIONED BY (pur_trans_date TIMESTAMP)
CLUSTERED BY(acct_num) INTO 5 BUCKETS
STORED AS ORC
TBLPROPERTIES ("transactional"="true");Then start the put hive streaming processor.
As i tried with same create table statement as above and the data that you have provided in the comment and i'm able to load the data into the table.
Flow:-
Select Hive Ql configs:-
HiveQL Select Query
select string(16680934) ACCT_NUM,string(747032990) PUR_ID,string(656564856) PUR_DET_ID,string(31) PRODUCT_PUR_PRODUCT_CODE,string(50) PROD_AMT,string('2015-01-01 08:12:03.0') PUR_TRANS_DATE,string(1) ACCTTYPE_ACCT_TYPE_CODE,string(01) ACCTTYPE_ACCT_STATUS_CODE,string(9999) EMP_EMP_CODE,string(009500 )PLAZA_PLAZA_ID,string(10) PURSTAT_PUR_STATUS_CODE
Output Format
Avro
Puthivestreaming Configs:-
Output:-
select * from default.purchase_acct_orc; +-----------+------------+-------------+---------------------------+-----------+--------------------------+----------------------------+---------------+-----------------+--------------------------+------------------------+--+ | acct_num | pur_id | pur_det_id | product_pur_product_code | prod_amt | accttype_acct_type_code | accttype_acct_status_code | emp_emp_code | plaza_plaza_id | purstat_pur_status_code | pur_trans_date | +-----------+------------+-------------+---------------------------+-----------+--------------------------+----------------------------+---------------+-----------------+--------------------------+------------------------+--+ | 16680934 | 747032990 | 656564856 | 31 | 50 | 1 | 1 | 9999 | 9500 | 10 | 2015-01-01 08:12:03.0 | +-----------+------------+-------------+---------------------------+-----------+--------------------------+----------------------------+---------------+-----------------+--------------------------+------------------------+--+
Let us know if you have any issues..!!
