About vtpcnk

vtpcnk · ‎07-10-2019

In Hue, how to bind user to particular queue - without having to specify it each time you login to Hue. The user when he logs into Hue, from thereafter any job he runs should be run on a specific queue. User doesn't want to specify the queue manually every time he logs into Hue. Appreciate any insights as to how this can be achieved.

vtpcnk · ‎06-27-2019

We have a user who wants to execute multiple pyspark sessions on the edge node. But past 13 sessions, new sessions simply hang apparently with the message : could not bind on port 4040 etc. These are just blank sessions and there is no job executed. They are simply testing the connections. They tried specifying the ports and could execute more than 13 session simultaneously. But they don't want to do this. So are new sessions hanging due to resource crunch? The edge node has 138 cores. I had the user try : pyspark --master local [4] So with the above command he could create more than 13 sessions. But is the above command restricting spark to run only on the local host? Will the job thus executed not scale across the cluster? Likewise can below command be used to execute spark across the cluster with specified cores : pyspark --master=yarn --deploy-mode=cluster --executor-cores 4 Appreciate the insights.

vtpcnk · ‎03-11-2019

@Kuldeep Kulkarni, there are many lines with "input data" in the page you referred - not sure which ones to ignore. Should I ignore the sections for datasets/input events/output events - that will leave only the workflow section. Is that right? Can't I use the coordinator from your shell action example? But in that I don't see : "<app-path>${workflowAppUri}</app-path>" Appreciate the clarification.

vtpcnk · ‎03-08-2019

@Kuldeep Kulkarni , does your example : https://community.hortonworks.com/articles/27497/oozie-coordinator-and-based-on-input-data-events.html set the job to run once a day? If not, can you please let me know how to do that? I want to run a job once daily. Thanks.

vtpcnk · ‎03-08-2019

BTW now I use cloudera. So I guess it has to be manual. Thanks.

vtpcnk · ‎03-08-2019

Hi Kuldeep, thanks so much for the clarification. I will try to do as per your instructions and let you know how it went. Thanks again.

vtpcnk · ‎03-08-2019

@Kuldeep Kulkarni, I created python action based on https://community.hortonworks.com/content/supportkb/151119/how-to-run-a-python-script-using-oozie-shell-actio.html But how do I integrate coordinator.xml with that? I tried creating the file but it is not executing as per that. Is there somewhere in job.properties or workflow.xml that you mention coordinator.xml? Appreciate the feedback.

vtpcnk · ‎03-06-2019

yes, I checked the nodes and found the output in one of them. I reran to make sure. So the script is not really needed on the Linux box? all of them - job.properties, the shellscript, workflow.xml, coordinator.xml? They need to be only in hdfs? Also next, how to execute a python code from oozie? Also I want it to run daily. Appreciate the insights.

vtpcnk · ‎03-06-2019

>this file should be created locally on the node manager where your shell script was run I have the file locally on a node. It is on this node that I execute the oozie command. Are you saying that the shell script should be in hdfs? And that yarn will execute this shell script on the node manager on which this job was run? Rather than on the host from which I run the oozie command? Appreciate the feedback.

vtpcnk · ‎03-06-2019

@Kuldeep Kulkarni , any insights pls? Appreciate the help.

vtpcnk · ‎03-05-2019

In the yarn log for the job I can see : ================================================================= >>> Invoking Shell command line now >> Exit code of the Shell command 0 <<< Invocation of Shell command completed <<< <<< Invocation of Main class completed <<< Oozie Launcher, capturing output data: =======================

vtpcnk · ‎03-05-2019

I created a similar job as in : https://community.hortonworks.com/content/supportkb/48985/how-to-setup-oozie-shell-action.html On execution the yarn log shows as success. But I can't see the /tmp/output file being created anywhere. I checked on the local linux host as well as hdfs. One question I have is in job.properties - oozie.coord.application.path=${nameNode}/user/root/apps/shell Should the configuration files (job.properties, coordinator.xml etc) reside in the above directory? I have them there. Not sure what is happening. Appreciate the insights.

vtpcnk · ‎02-19-2019

We have a setup where flume reads from kafka and writes into hdfs. We have a requirement where we want flume to read data from kafka for only a particular day - so we want to filter data by timestamp. So for example : we want to read data only for February 17th 2019. The data has within it the date in the format : "2019-02-17" I tried the below in the my flume configuration but it did not work : flume_agent.sources.kafka1.interceptors = regex flume_agent.sources.kafka1.interceptors.regex.type = regex_filter flume_agent.sources.kafka1.interceptors.regex.regex = 2019-02-17 flume_flat_agent.sources.kafka1.interceptors.regex.includeEvents = true Appreciate any insights as to how I can achieve this.

vtpcnk · ‎01-08-2019

Is it not possible to use the avro serializer with hive sink with flume? Like : r_hive.sinks.sink1.type = hive rsa_hive.sinks.sink1.hive.metastore = thrift://master2:9083 r_hive.sinks.sink1.hive.database = menw r_hive.sinks.sink1.hive.table = flume_hive r_hive.sinks.sink1.serializer = org.apache.flume.sink.hdfs.AvroEventSerializer$Builder r_hive.sinks.sink1.serializer.schemaURL = hdfs://host:8020/tmp/avroschemas-new/mf.json I am seeing the error in flume log : 2019-01-07 16:19:18,360 ERROR org.apache.flume.node.AbstractConfigurationProvider: Sink sink1 has been removed due to an error during configuration java.lang.IllegalArgumentException: Unable to instantiate serializer: org.apache.flume.sink.hdfs.AvroEventSerializer$Builder on sink: sink1 at org.apache.flume.sink.hive.HiveSink.createSerializer(HiveSink.java:228) Caused by: java.lang.ClassCastException: org.apache.flume.sink.hdfs.AvroEventSerializer$Builder cannot be cast to org.apache.flume.sink.hive.HiveEventSerializer

vtpcnk · ‎01-04-2019

I created the hive to hbase mapping with avro based on : https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration#HBaseIntegration-AvroDataStoredinHBaseColumns.1 hbase(main):002:0> scan "menw:test2" ROW COLUMN+CELL defaultrewcrwxd-1549-4177-86 column=me_data:pCol, timestamp=1546545400898, value=.reanda42398fsdjsfd432098\rwefds. rwe-rwerewhjsfd 2018-12-28-15.24.29.896\x08SUCC\x1rwejsfdasfd432f\x02e\x00\x06dfs\x00\x00 defaultrewcxnbz-1649-4188-86 column=me_data:pCol, timestamp=1546544320898, value=.reanda45398ffdssfd432098\rwbfds. rwe-rwerewhjsfd 2018-12-28-15.24.29.896\x08SUCC\x1rwsdffdasfd432f\x02e\x00\x06dfs\x00\x00 hive> CREATE EXTERNAL TABLE test_hbase_avro > ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = ":key,me_data:id", > "me_data.id.serialization.type" = "avro", > "me_data.id.avro.schema.url" = "hdfs://host:8020/tmp/avroschemas-new/table.json") > TBLPROPERTIES ( > "hbase.table.name" = "menw:test2", > "hbase.mapred.output.outputtable" = "menw:test2", > "hbase.struct.autogenerate"="true"); OK Time taken: 1.352 seconds But select from the table failed with below error : hive> select * from test_hbase_avro; OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating medata_pcol Time taken: 0.214 seconds Appreciate any insights to resolve the issue.

vtpcnk · ‎01-02-2019

I have data coming into flume which I want to go into three columns in a column family in a hbase table. The data pattern is like : A000052019-01-02:13:14:260000000SG First six digits are id. next 19 digits are date time. Last 9 digits are code. I need to know what to put in the flume configuration parameter r_hbase.sinks.sink1.serializer.regex - as below : r_hbase.sinks.sink1.table = mnew:r_test r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer=org.apache.flume.sink.hbase.RegexHbaseEventSerializer r_hbase.sinks.sink1.serializer.regex= r_hbase.sinks.sink1.serializer.colNames=me_data:id,me_data:date,me_data:code Appreciate the help.

vtpcnk · ‎12-28-2018

So we can use regex for flume/hbase only with delimited or fixed length data? Appreciate the clarification.

vtpcnk · ‎12-28-2018

So we can use regex for flume/hbase only with delimited or fixed length data? We cannot use regex with data without delimitation or variable lengths? Appreciate the clarification.

vtpcnk · ‎12-20-2018

this is the flume configuration hbase sink part : r_hbase.sinks.sink1.channel = channel1 r_hbase.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink r_hbase.sinks.sink1.table = mev:rtable r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer r_hbase.sinks.sink1.serializer.schemaURL = hdfs://host.com:8020/tmp/avroschemas-new/rtable.json r_hbase.sinks.sink1.serializer.columns = col1,col2,col3,col4,col5,col6,col7,col8,col9,col10

vtpcnk · ‎12-20-2018

We have data coming into hdfs through flume. This data is serially encoded. We use the avro schema to decode it. Now we want to put this data from flume into a hbase table instead of hdfs. We use below parameter to do serial decoding based on avro schema : ragent.sinks.sink1.serializer.schemaURL = hdfs://host.com:8020/tmp/avroschemas-new/rtable.json Will this same argument work when the target is hbase instead of hdfs? along with hbase table, hbase table column family etc? Actually for some reason the data is not going in properly into hbase, and I am not sure if this avro schema is the issue. Appreciate the insights.

vtpcnk · ‎12-20-2018

hi @Naveen any response on this please? Thanks.

vtpcnk · ‎12-19-2018

@Naveen my hbase table is created only with a single column family. create 'mbev:hb_test' , 'me_data' Then data is fed into the hbase table through flume. This data originally was sent to a hive table with 10 columns - but for small files issue, i am redirecting the same data to the above hbase table. my flume config has the following lines : r_hbase.sinks.sink1.table = mbev:hb_test r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer.columns =col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 So I can see data coming into the hbase table. Data is showing like : hbase(main):008:0> scan "mbev:hb_test" ROW COLUMN+CELL default0998c6b9-2fa9 column=me_data:pCol, timestamp=1545242255268, value=.z3knt -122e-1536-0o42ef7fb dErc90GqYg5a3n-zTQ\a04NQA32018-12-19-12.57.30.000123\x09FY a3e YY\q1V10006002079317\x01M\x00\x09KEF\x00\x00 So how do I create the hive table to see the data in the hbase table. I tried as we did above initially with only "":key,me_data:id", but don't see the data in the hive table. I also tried : create external table tmp_test4 (col1 string,col2 string,col3 string,col4 string,col5 string,col6 string,col7 string,col8 string,col9 string,col10 string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,me_data:col1,me_data:col2,me_data:col3,me_data:col4,me_data:col5,me_data:col6,me_data:col7,me_data:col8,me_data:col9,me_data:col10") TBLPROPERTIES("hbase.table.name" = "mbev:hb_test"); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.hbase.HBaseSerDe: columns has 10 elements while hbase.columns. mapping has 11 elements (counting the key if implicit)) Appreciate your help.

vtpcnk · ‎12-18-2018

btw : the create table shows two columns : create external table hv_test (id string, idate string) so how can this statement show only 1 column : WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,me_data:id") don't we need to likewise map idate?

vtpcnk · ‎12-18-2018

I have flume putting data into hbase. the data is like data for 4 columns. So can i create the table in hbase like below : create "table1", "col1", "col2", "col3", "col4" In the flume configuration, is this the way to set values for these three parameters : rsa_hbase.sinks.sink1.table = table1 rsa_hbase.sinks.sink1.columnFamily = col1,col2,col3,col4 rsa_hbase.sinks.sink1.column = col1,col2,col3,col4 rsa_hbase.sinks.sink1.serializer.columns = col1,col2,col3,col4 I am basically doing this as a solution for the small files issue - trying to use hbase for it. I will eventually be mapping the hbase table to a hive table. Appreciate the insights.

vtpcnk · ‎12-18-2018

In my hbase table actually data for 10 columns lands from flume. In the flume configuration I have defined : host1.sinks.sink1.serializer.columns = col1, col2 ... In HBASE the table is only defined as above : create 'mbev:hb_test' , 'me_data' So how do I define the hive table for the respective data for the 10 columns?

vtpcnk · ‎12-18-2018

Naveen, thanks so much.

vtpcnk · ‎12-17-2018

I created a table in hbase. Created a table in hive mapping it to the table in hbase. Inserted a row in the hbase table. But hive table doesn't show any rows. Any idea what I am doing wrong? Appreciate the feedback. hbase(main):019:0> create 'mbev:hb_test' , 'me_data' 0 row(s) in 1.2670 seconds hbase(main):006:0> put "mbev:hb_test",'1',"me_data:id",'1' 0 row(s) in 0.1700 seconds hbase(main):007:0> scan "mbev:hb_test" ROW COLUMN+CELL 1 column=me_data:id, timestamp=1545064141017, value=1 1 row(s) in 0.0590 seconds hive>create external table hv_test (id string, idate string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,me_data:idate") TBLPROPERTIES("hbase.table.name" = "mbev:hb_test"); hive> select * from hv_test ; OK Time taken: 0.23 seconds

vtpcnk · ‎12-14-2018

I have this below where clause - all below columns are defined as string in the table : WHERE a.code IN ('abc','cde') AND a.code = b.code AND create_ts > date_sub(from_unixtime(unix_timestamp()),60) AND create_dt > date_sub(from_unixtime(unix_timestamp()),60) AND number = '00000000044671' Can I know what all can be improved upon in the above where clause. Can it be better coded? Can some conversion be used for the date time for faster processing? Something to make the join more efficient? Or even will defining columns as datetime or integer, will that make it run faster? Whatever can be done to make it run faster - appreciate the insights.

vtpcnk · ‎12-04-2018

We had a Hive table which has its "location" in a hdfs directory. The location directory has a bunch of small data files which represents the data of the table. Data keeps coming into the location directory and so numerous small files are created all the time. But for performance we want to merge all these small files into a larger file on a periodic basis. Whats the best way to do this? I hear that Hive itself has a merge option? Appreciate the insights.

vtpcnk · ‎11-30-2018

I tried the below but it didn't work : hive> create view testview as select * from test1 where id = "{$hiveconf:id}"; OK Time taken: 0.13 seconds hive> set id=1; hive> select * from testview; Above query did not return any rows.

Online	Offline
Last Visited	‎05-28-2019 10:43 AM

Date Registered	‎11-24-2015 05:10 AM
Last Visited	‎05-28-2019 10:43 AM
Total Messages Posted	217
Total Kudos Received	10

Hue - binding user to queue

pyspark multiple sessions

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

oozie shell action

flume - filtering data

flume - hive sink - avro

hive to hbase mapping with avro - issue

regex pattern for flume-hbase

flume hbase regex

flume hbase regex

Re: flume with hbase with avro

flume with hbase with avro

Re: hive on hbase - not showing data

Re: hive on hbase - not showing data

Re: hive on hbase - not showing data

flume with hbase

Re: hive on hbase - not showing data

Re: hive on hbase - not showing data

hive on hbase - not showing data

hive query optimization scope

Hive : merging small data files into a big file

Re: hive view parameter and union all