BTW files with ".tmp" extension could be under any subdirectory under "/backups".   Thanks. ... View more

funnily nothing turned up in the log. But we got this alert : Content: The health test result for HTTPFS_SCM_HEALTH has become bad: This role's process exited. This role is supposed to be started.   We are using cloudera 5.15.1 on RHEL. ... View more

@Kuldeep Kulkarni, there are many lines with "input data" in the page you referred - not sure which ones to ignore. Should I ignore the sections for datasets/input events/output events - that will leave only the workflow section. Is that right? Can't I use the coordinator from your shell action example? But in that I don't see : "<app-path>${workflowAppUri}</app-path>" Appreciate the clarification. ... View more

@Kuldeep Kulkarni , does your example : https://community.hortonworks.com/articles/27497/oozie-coordinator-and-based-on-input-data-events.html set the job to run once a day? If not, can you please let me know how to do that? I want to run a job once daily. Thanks. ... View more

BTW now I use cloudera. So I guess it has to be manual. Thanks. ... View more

Hi Kuldeep, thanks so much for the clarification. I will try to do as per your instructions and let you know how it went. Thanks again. ... View more

@Kuldeep Kulkarni, I created python action based on https://community.hortonworks.com/content/supportkb/151119/how-to-run-a-python-script-using-oozie-shell-actio.html But how do I integrate coordinator.xml with that? I tried creating the file but it is not executing as per that. Is there somewhere in job.properties or workflow.xml that you mention coordinator.xml? Appreciate the feedback. ... View more

yes, I checked the nodes and found the output in one of them. I reran to make sure. So the script is not really needed on the Linux box? all of them - job.properties, the shellscript, workflow.xml, coordinator.xml? They need to be only in hdfs? Also next, how to execute a python code from oozie? Also I want it to run daily. Appreciate the insights. ... View more

>this file should be created locally on the node manager where your shell script was run I have the file locally on a node. It is on this node that I execute the oozie command. Are you saying that the shell script should be in hdfs? And that yarn will execute this shell script on the node manager on which this job was run? Rather than on the host from which I run the oozie command? Appreciate the feedback. ... View more

@Kuldeep Kulkarni , any insights pls? Appreciate the help. ... View more

In the yarn log for the job I can see : ================================================================= >>> Invoking Shell command line now >> Exit code of the Shell command 0 <<< Invocation of Shell command completed <<< <<< Invocation of Main class completed <<< Oozie Launcher, capturing output data: ======================= ... View more

this is the flume configuration hbase sink part : r_hbase.sinks.sink1.channel = channel1 r_hbase.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink r_hbase.sinks.sink1.table = mev:rtable r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer r_hbase.sinks.sink1.serializer.schemaURL = hdfs://host.com:8020/tmp/avroschemas-new/rtable.json r_hbase.sinks.sink1.serializer.columns = col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 ... View more

hi @Naveen any response on this please? Thanks. ... View more

@Naveen my hbase table is created only with a single column family. create 'mbev:hb_test' , 'me_data' Then data is fed into the hbase table through flume. This data originally was sent to a hive table with 10 columns - but for small files issue, i am redirecting the same data to the above hbase table. my flume config has the following lines : r_hbase.sinks.sink1.table = mbev:hb_test r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer.columns =col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 So I can see data coming into the hbase table. Data is showing like : hbase(main):008:0> scan "mbev:hb_test" ROW COLUMN+CELL default0998c6b9-2fa9 column=me_data:pCol, timestamp=1545242255268, value=.z3knt -122e-1536-0o42ef7fb dErc90GqYg5a3n-zTQ\a04NQA32018-12-19-12.57.30.000123\x09FY a3e YY\q1V10006002079317\x01M\x00\x09KEF\x00\x00 So how do I create the hive table to see the data in the hbase table. I tried as we did above initially with only "":key,me_data:id", but don't see the data in the hive table. I also tried : create external table tmp_test4 (col1 string,col2 string,col3 string,col4 string,col5 string,col6 string,col7 string,col8 string,col9 string,col10 string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,me_data:col1,me_data:col2,me_data:col3,me_data:col4,me_data:col5,me_data:col6,me_data:col7,me_data:col8,me_data:col9,me_data:col10") TBLPROPERTIES("hbase.table.name" = "mbev:hb_test"); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.hbase.HBaseSerDe: columns has 10 elements while hbase.columns. mapping has 11 elements (counting the key if implicit)) Appreciate your help. ... View more

btw : the create table shows two columns : create external table hv_test (id string, idate string) so how can this statement show only 1 column : WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,me_data:id") don't we need to likewise map idate? ... View more

In my hbase table actually data for 10 columns lands from flume. In the flume configuration I have defined : host1.sinks.sink1.serializer.columns = col1, col2 ... In HBASE the table is only defined as above : create 'mbev:hb_test' , 'me_data' So how do I define the hive table for the respective data for the 10 columns? ... View more

Naveen, thanks so much. ... View more

I tried the below but it didn't work : hive> create view testview as select * from test1 where id = "{$hiveconf:id}"; OK Time taken: 0.13 seconds hive> set id=1; hive> select * from testview; Above query did not return any rows. ... View more

ok, this can be done simply as : partitioned by (yr string, mth string). tks. ... View more

I tried this but wouldn't work : create table test_part_bkt_tbl (id string, cd string, dttm string) partitioned by (yr string) clustered by (month(dttm)) into 12 buckets; ... View more

if i partition a table by year - can i further bucket it by month? so the idea is the year will be the top level and months will be at a level beneath it. so the directory structure would be : 2018 -> 1, 2, 3 ... 12 2019 -> 1, 2, 3 ... 12 Is this what bucketing is about? Or should i be doing this someway with partitions itself? Appreciate the insights. ... View more