Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Streaming MutatorCoordinator complex column

Highlighted

Streaming MutatorCoordinator complex column

New Contributor

Hello,

i'm newbie with Hadoop and Hive. I'm using Hive 2.3.0 and i'm working to build ingestion data from Kafka Stream to Hadhoop.

I have create a very simple table

CREATE TABLE test(id int, tipo string,name string, consumo double,attributes map<String,String>) PARTITIONED BY (orgid string, endpoint string ) clustered by (id) into 4 buckets  STORED as ORC tblproperties ("transactional"="true");

and i have started to build some queries, and all works fine. And it works fine also via jdbc:hive2 driver

INSERT INTO rse8 PARTITION (orgid = '888', endpoint='55') select 9,'VASCA','MOD',50,map('MARCA','SAMSUNG','CLASSE','A');

but i understand that i not suggest to use jdbc with kafka stream/storm and so. Then after 3 days i'm starting to use

//org.apache.hive.hcatalog.streaming.mutate.worker MutatorCoordinator 
MutatorCoordinator coordinator.....

coordinator.insert(part, new MutableRecord(updated.getObjectid(), updated.getParentID(), 20));

My problem now how i can do with complex type (map in my table example)?

+-----------+-------------+-------------+----------------+-------------------+--------------+-----------------+
| noemi.id | noemi.tipo | noemi.name | noemi.consumo | noemi.attributes | noemi.orgid | noemi.endpoint |
+-----------+-------------+-------------+----------------+-------------------+--------------+-----------------+
| NULL | 25000/1/2 | 25000 | 20.0 | NULL | flexvalley | prova |

the primitive data works but I don't understand why:

  1. my id is always null, i dont' understand very well RecordIdentifier maybe :-(
  2. how to send "maps" over coordinator?

I have no found much info (i have used hive test class to learn more) about this

Thank you in advance... it's very urgent for me i hope that you can help me how to fix.

Regards

Devis

my mutableRecord class is:

public class MutableRecord {    // Column 0

      public final RecordIdentifier id;    // Column 1      public final Text tipo;    // Column 2     dd public final Text name;    // Column 3        public final double consumo;    // Column 4       private final java.util.Map<String,String>  maps =new HashMap();    public MutableRecord(String tipo, String name, double consumo) {        this.tipo = new Text(tipo);        this.name = new Text(name);        this.consumo = consumo;        this.maps.put("xxx","xxx");        id=new RecordIdentifier();        id.setRowId(0);    }}
1 REPLY 1

Re: Streaming MutatorCoordinator complex column

Explorer

Did you find a solution for this?

Don't have an account?
Coming from Hortonworks? Activate your account here