Kudu backup performance

vtpcnk · ‎04-15-2021

Hi, we are trying to backup a kudu table as below : spark2-submit --principal <user> --keytab <keytab> --master yarn --deploy-mode cluster --queue <queue> --executor-memory 12G --executor-cores 4 --driver-memory 4G --driver-cores 1 --class org.apache.kudu.backup.KuduBackup kudu-backup2_2.11-1.13.0.7.1.5.0-257.jar --kuduMasterAddresses $KUDU_MASTERS --rootPath hdfs:///backups --forceFull true impala::<table> And it is super slow. Any suggestions on how to make it run faster? Appreciate the feedback.

vtpcnk · ‎04-06-2021

BTW files with ".tmp" extension could be under any subdirectory under "/backups". Thanks.

vtpcnk · ‎04-06-2021

Hi, In BDR HDFS replication I want to exclude all files which end with ".tmp" in the directory "/backups/" Appreciate it if somebody could give the expression for this to add in BDR "Add Exclusion". Thanks

vtpcnk · ‎04-23-2020

funnily nothing turned up in the log. But we got this alert : Content: The health test result for HTTPFS_SCM_HEALTH has become bad: This role's process exited. This role is supposed to be started. We are using cloudera 5.15.1 on RHEL.

vtpcnk · ‎04-22-2020

We are experiencing out of memory errors with httpfs. This is happening when users use Hue to access a particular larger folder in hdfs. We upped "java heap size of httpfs" to 1 gb - but still facing issue. There is also a "java client heap size" parameter - will upping that help in our case? Appreciate the insights.

vtpcnk · ‎04-16-2020

We are trying to implement alerting in our cluster and alerting is setup in Cloudera Manager. So when I stop a service in Cloudera Manager, an alert is sent to my email. Because for some reason I hear that if you stop the service from CM, it is not the same as it crashing on its own. Especially with regards Canary Alerts, which we will not get if we stop a service through Cloudera Manager. So will I not get Canary Alerts for a service if the service is stopped through Cloudera Manager? Also I would like to know how to stop a service manually through Cloudera API. I would appreciate it if some forum member could give the command to stop - say Oozie or HBase - through Cloudera Manager API. Appreciate the help.

vtpcnk · ‎07-10-2019

In Hue, how to bind user to particular queue - without having to specify it each time you login to Hue. The user when he logs into Hue, from thereafter any job he runs should be run on a specific queue. User doesn't want to specify the queue manually every time he logs into Hue. Appreciate any insights as to how this can be achieved.

vtpcnk · ‎06-27-2019

We have a user who wants to execute multiple pyspark sessions on the edge node. But past 13 sessions, new sessions simply hang apparently with the message : could not bind on port 4040 etc. These are just blank sessions and there is no job executed. They are simply testing the connections. They tried specifying the ports and could execute more than 13 session simultaneously. But they don't want to do this. So are new sessions hanging due to resource crunch? The edge node has 138 cores. I had the user try : pyspark --master local [4] So with the above command he could create more than 13 sessions. But is the above command restricting spark to run only on the local host? Will the job thus executed not scale across the cluster? Likewise can below command be used to execute spark across the cluster with specified cores : pyspark --master=yarn --deploy-mode=cluster --executor-cores 4 Appreciate the insights.

vtpcnk · ‎03-11-2019

@Kuldeep Kulkarni, there are many lines with "input data" in the page you referred - not sure which ones to ignore. Should I ignore the sections for datasets/input events/output events - that will leave only the workflow section. Is that right? Can't I use the coordinator from your shell action example? But in that I don't see : "<app-path>${workflowAppUri}</app-path>" Appreciate the clarification.

vtpcnk · ‎03-08-2019

@Kuldeep Kulkarni , does your example : https://community.hortonworks.com/articles/27497/oozie-coordinator-and-based-on-input-data-events.html set the job to run once a day? If not, can you please let me know how to do that? I want to run a job once daily. Thanks.

vtpcnk · ‎03-08-2019

BTW now I use cloudera. So I guess it has to be manual. Thanks.

vtpcnk · ‎03-08-2019

Hi Kuldeep, thanks so much for the clarification. I will try to do as per your instructions and let you know how it went. Thanks again.

vtpcnk · ‎03-08-2019

@Kuldeep Kulkarni, I created python action based on https://community.hortonworks.com/content/supportkb/151119/how-to-run-a-python-script-using-oozie-shell-actio.html But how do I integrate coordinator.xml with that? I tried creating the file but it is not executing as per that. Is there somewhere in job.properties or workflow.xml that you mention coordinator.xml? Appreciate the feedback.

vtpcnk · ‎03-06-2019

yes, I checked the nodes and found the output in one of them. I reran to make sure. So the script is not really needed on the Linux box? all of them - job.properties, the shellscript, workflow.xml, coordinator.xml? They need to be only in hdfs? Also next, how to execute a python code from oozie? Also I want it to run daily. Appreciate the insights.

vtpcnk · ‎03-06-2019

>this file should be created locally on the node manager where your shell script was run I have the file locally on a node. It is on this node that I execute the oozie command. Are you saying that the shell script should be in hdfs? And that yarn will execute this shell script on the node manager on which this job was run? Rather than on the host from which I run the oozie command? Appreciate the feedback.

vtpcnk · ‎03-06-2019

@Kuldeep Kulkarni , any insights pls? Appreciate the help.

vtpcnk · ‎03-05-2019

In the yarn log for the job I can see : ================================================================= >>> Invoking Shell command line now >> Exit code of the Shell command 0 <<< Invocation of Shell command completed <<< <<< Invocation of Main class completed <<< Oozie Launcher, capturing output data: =======================

vtpcnk · ‎03-05-2019

I created a similar job as in : https://community.hortonworks.com/content/supportkb/48985/how-to-setup-oozie-shell-action.html On execution the yarn log shows as success. But I can't see the /tmp/output file being created anywhere. I checked on the local linux host as well as hdfs. One question I have is in job.properties - oozie.coord.application.path=${nameNode}/user/root/apps/shell Should the configuration files (job.properties, coordinator.xml etc) reside in the above directory? I have them there. Not sure what is happening. Appreciate the insights.

vtpcnk · ‎02-19-2019

We have a setup where flume reads from kafka and writes into hdfs. We have a requirement where we want flume to read data from kafka for only a particular day - so we want to filter data by timestamp. So for example : we want to read data only for February 17th 2019. The data has within it the date in the format : "2019-02-17" I tried the below in the my flume configuration but it did not work : flume_agent.sources.kafka1.interceptors = regex flume_agent.sources.kafka1.interceptors.regex.type = regex_filter flume_agent.sources.kafka1.interceptors.regex.regex = 2019-02-17 flume_flat_agent.sources.kafka1.interceptors.regex.includeEvents = true Appreciate any insights as to how I can achieve this.

vtpcnk · ‎01-08-2019

Is it not possible to use the avro serializer with hive sink with flume? Like : r_hive.sinks.sink1.type = hive rsa_hive.sinks.sink1.hive.metastore = thrift://master2:9083 r_hive.sinks.sink1.hive.database = menw r_hive.sinks.sink1.hive.table = flume_hive r_hive.sinks.sink1.serializer = org.apache.flume.sink.hdfs.AvroEventSerializer$Builder r_hive.sinks.sink1.serializer.schemaURL = hdfs://host:8020/tmp/avroschemas-new/mf.json I am seeing the error in flume log : 2019-01-07 16:19:18,360 ERROR org.apache.flume.node.AbstractConfigurationProvider: Sink sink1 has been removed due to an error during configuration java.lang.IllegalArgumentException: Unable to instantiate serializer: org.apache.flume.sink.hdfs.AvroEventSerializer$Builder on sink: sink1 at org.apache.flume.sink.hive.HiveSink.createSerializer(HiveSink.java:228) Caused by: java.lang.ClassCastException: org.apache.flume.sink.hdfs.AvroEventSerializer$Builder cannot be cast to org.apache.flume.sink.hive.HiveEventSerializer

vtpcnk · ‎01-04-2019

I created the hive to hbase mapping with avro based on : https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration#HBaseIntegration-AvroDataStoredinHBaseColumns.1 hbase(main):002:0> scan "menw:test2" ROW COLUMN+CELL defaultrewcrwxd-1549-4177-86 column=me_data:pCol, timestamp=1546545400898, value=.reanda42398fsdjsfd432098\rwefds. rwe-rwerewhjsfd 2018-12-28-15.24.29.896\x08SUCC\x1rwejsfdasfd432f\x02e\x00\x06dfs\x00\x00 defaultrewcxnbz-1649-4188-86 column=me_data:pCol, timestamp=1546544320898, value=.reanda45398ffdssfd432098\rwbfds. rwe-rwerewhjsfd 2018-12-28-15.24.29.896\x08SUCC\x1rwsdffdasfd432f\x02e\x00\x06dfs\x00\x00 hive> CREATE EXTERNAL TABLE test_hbase_avro > ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = ":key,me_data:id", > "me_data.id.serialization.type" = "avro", > "me_data.id.avro.schema.url" = "hdfs://host:8020/tmp/avroschemas-new/table.json") > TBLPROPERTIES ( > "hbase.table.name" = "menw:test2", > "hbase.mapred.output.outputtable" = "menw:test2", > "hbase.struct.autogenerate"="true"); OK Time taken: 1.352 seconds But select from the table failed with below error : hive> select * from test_hbase_avro; OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating medata_pcol Time taken: 0.214 seconds Appreciate any insights to resolve the issue.

vtpcnk · ‎01-02-2019

I have data coming into flume which I want to go into three columns in a column family in a hbase table. The data pattern is like : A000052019-01-02:13:14:260000000SG First six digits are id. next 19 digits are date time. Last 9 digits are code. I need to know what to put in the flume configuration parameter r_hbase.sinks.sink1.serializer.regex - as below : r_hbase.sinks.sink1.table = mnew:r_test r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer=org.apache.flume.sink.hbase.RegexHbaseEventSerializer r_hbase.sinks.sink1.serializer.regex= r_hbase.sinks.sink1.serializer.colNames=me_data:id,me_data:date,me_data:code Appreciate the help.

vtpcnk · ‎12-28-2018

So we can use regex for flume/hbase only with delimited or fixed length data? Appreciate the clarification.

vtpcnk · ‎12-28-2018

So we can use regex for flume/hbase only with delimited or fixed length data? We cannot use regex with data without delimitation or variable lengths? Appreciate the clarification.

vtpcnk · ‎12-20-2018

this is the flume configuration hbase sink part : r_hbase.sinks.sink1.channel = channel1 r_hbase.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink r_hbase.sinks.sink1.table = mev:rtable r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer r_hbase.sinks.sink1.serializer.schemaURL = hdfs://host.com:8020/tmp/avroschemas-new/rtable.json r_hbase.sinks.sink1.serializer.columns = col1,col2,col3,col4,col5,col6,col7,col8,col9,col10

vtpcnk · ‎12-20-2018

We have data coming into hdfs through flume. This data is serially encoded. We use the avro schema to decode it. Now we want to put this data from flume into a hbase table instead of hdfs. We use below parameter to do serial decoding based on avro schema : ragent.sinks.sink1.serializer.schemaURL = hdfs://host.com:8020/tmp/avroschemas-new/rtable.json Will this same argument work when the target is hbase instead of hdfs? along with hbase table, hbase table column family etc? Actually for some reason the data is not going in properly into hbase, and I am not sure if this avro schema is the issue. Appreciate the insights.

vtpcnk · ‎12-20-2018

hi @Naveen any response on this please? Thanks.

vtpcnk · ‎12-19-2018

@Naveen my hbase table is created only with a single column family. create 'mbev:hb_test' , 'me_data' Then data is fed into the hbase table through flume. This data originally was sent to a hive table with 10 columns - but for small files issue, i am redirecting the same data to the above hbase table. my flume config has the following lines : r_hbase.sinks.sink1.table = mbev:hb_test r_hbase.sinks.sink1.columnFamily = me_data r_hbase.sinks.sink1.serializer.columns =col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 So I can see data coming into the hbase table. Data is showing like : hbase(main):008:0> scan "mbev:hb_test" ROW COLUMN+CELL default0998c6b9-2fa9 column=me_data:pCol, timestamp=1545242255268, value=.z3knt -122e-1536-0o42ef7fb dErc90GqYg5a3n-zTQ\a04NQA32018-12-19-12.57.30.000123\x09FY a3e YY\q1V10006002079317\x01M\x00\x09KEF\x00\x00 So how do I create the hive table to see the data in the hbase table. I tried as we did above initially with only "":key,me_data:id", but don't see the data in the hive table. I also tried : create external table tmp_test4 (col1 string,col2 string,col3 string,col4 string,col5 string,col6 string,col7 string,col8 string,col9 string,col10 string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,me_data:col1,me_data:col2,me_data:col3,me_data:col4,me_data:col5,me_data:col6,me_data:col7,me_data:col8,me_data:col9,me_data:col10") TBLPROPERTIES("hbase.table.name" = "mbev:hb_test"); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.hbase.HBaseSerDe: columns has 10 elements while hbase.columns. mapping has 11 elements (counting the key if implicit)) Appreciate your help.

vtpcnk · ‎12-18-2018

btw : the create table shows two columns : create external table hv_test (id string, idate string) so how can this statement show only 1 column : WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,me_data:id") don't we need to likewise map idate?

vtpcnk · ‎12-18-2018

I have flume putting data into hbase. the data is like data for 4 columns. So can i create the table in hbase like below : create "table1", "col1", "col2", "col3", "col4" In the flume configuration, is this the way to set values for these three parameters : rsa_hbase.sinks.sink1.table = table1 rsa_hbase.sinks.sink1.columnFamily = col1,col2,col3,col4 rsa_hbase.sinks.sink1.column = col1,col2,col3,col4 rsa_hbase.sinks.sink1.serializer.columns = col1,col2,col3,col4 I am basically doing this as a solution for the small files issue - trying to use hbase for it. I will eventually be mapping the hbase table to a hive table. Appreciate the insights.

Online	Offline
Last Visited	‎05-05-2021 09:04 AM

Date Registered	‎11-24-2015 05:10 AM
Last Visited	‎05-05-2021 09:04 AM
Total Messages Posted	223
Total Kudos Received	10

Kudu backup performance

Re: BDR - HDFS - exclusion

BDR - HDFS - exclusion

Re: hdfs httpfs out of memory error

hdfs httpfs out of memory error

Cloudera Manager alerting and API

Hue - binding user to queue

pyspark multiple sessions

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

Re: oozie shell action

oozie shell action

flume - filtering data

flume - hive sink - avro

hive to hbase mapping with avro - issue

regex pattern for flume-hbase

flume hbase regex

flume hbase regex

Re: flume with hbase with avro

flume with hbase with avro

Re: hive on hbase - not showing data

Re: hive on hbase - not showing data

Re: hive on hbase - not showing data

flume with hbase