Re: How to Load data from hdfs Multi level directo...

helmi_khalifa · ‎01-17-2021

Hi @vjain , To configure the BuckeCache in the descripption there is a two JVM properties. Which one to use please? : HBASE_OPTS or HBASE_REGIONSERVER_OPTS In the hbase-env.sh file for each RegionServer, or in the hbase-env.sh file supplied to Ambari, set the -XX:MaxDirectMemorySize argument for HBASE_REGIONSERVER_OPTS to the amount of direct memory you wish to allocate to HBase. In the configuration for the example discussed above, the value would be 241664m . ( -XX:MaxDirectMemorySize accepts a number followed by a unit indicator; m indicates megabytes.) HBASE_OPTS="$HBASE_OPTS -XX:MaxDirectMemorySize=241664m" Thanks, Helmi KHALIFA

helmi_khalifa · ‎02-22-2020

Hi, I am facing the same problem. Do you find a solution to your problem ? Best, Helmi Khalifa

helmi_khalifa · ‎11-14-2019

hi @avengers If it works for you, would you be kind enough to accept the answer please ? Best, Helmi KHALIFA

helmi_khalifa · ‎11-08-2019

Hi @avengers , U will need to share variables between two zeppelin interpreters and i dont think that we can do it between spark and sparkSQL. I find an easier way by using sqlContext inside the same interpreter %spark: %spark val df = spark.read.format("csv").option("header", "true") .option("inferSchema", "true").load("/somefile.csv") df.createOrReplaceTempView("csvTable"); val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) val resultat = sqlContext.sql("select * from csvTable lt join hiveTable rt on lt.col = rt.col") resultat.show() I tried it and it works ! Best, Helmi KHALIFA

helmi_khalifa · ‎11-06-2019

Hi @av , Here the links for the Hive and Spark interpreter doc's : https://zeppelin.apache.org/docs/0.8.2/interpreter/hive.html https://zeppelin.apache.org/docs/0.8.2/interpreter/spark.html Best, Helmi KHALIFA

helmi_khalifa · ‎11-05-2019

Hi @Rak ; here the script : CREATE EXTERNAL TABLE IF NOT EXISTS sample_date (sc_code string, ddate timestamp, co_code DECIMAL, high DECIMAL, low DECIMAL, open DECIMAL, close DECIMAL, volume DECIMAL, no_trades DECIMAL, net_turnov DECIMAL, dmcap DECIMAL, return DECIMAL, factor DECIMAL, ttmpe DECIMAL, yepe DECIMAL, flag string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/lab/itim/ccbd/helmi/sampleDate' tblproperties('skip.header.line.count'='1'); ALTER TABLE sample_date SET SERDEPROPERTIES ("timestamp.formats"="MM/DD/YYYY"); Could you accept the answer please ? Best, Helmi KHALIFA

helmi_khalifa · ‎11-04-2019

Hi @Ra You have to change the name and column type as youscan see in red below : sc_code string ddate date co_code double high double low double open double close double volume double no_trades double net_turnov double dmcap double return double factor double ttmpe double yepe double flag string I tried it and it works well for me. Best, Helmi KHALIFA

helmi_khalifa · ‎10-31-2019

Hi @Rak , Can you show us some sample the 5 first rows of your csv file, please ? Best, Helmi KHALIFA

helmi_khalifa · ‎10-31-2019

Hi @saivenkatg55 More details please? screenshots of Yarn UI ? Best, Helmi KHALIFA

helmi_khalifa · ‎10-25-2019

Hi @RNN The best solution is to convert the Monthes to integers like: -Oct- => -10- -Dec- =>-12- So that is what i tested as you can see my file below: $ hdfs dfs -cat /lab/helmi/test_timestamp_MM.txt 1,2019-10-14 20:00:01.027898 2,2019-12-10 21:00:01.023 3,2019-11-25 20:00:01.03 4,2019-01-06 20:00:01.123 Create a Hive table : hive> CREATE EXTERNAL TABLE ttime(id int, t string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/lab/helmi/'; hive> select * from ttime; OK 1 2019-10-14 20:00:01.027898 2 2019-12-10 21:00:01.023 3 2019-11-25 20:00:01.03 4 2019-01-06 20:00:01.123 Time taken: 0.566 seconds, Fetched: 4 row(s) Finally i created another table with the right format: hive> create table mytime as select id, from_utc_timestamp(date_format(t,'yyyy-MM-dd HH:mm:ss.SSSSSS'),'UTC') as datetime from ttime; Best, Helmi KHALIFA

helmi_khalifa · ‎10-24-2019

Could you try on another table? Looks like you have an encoding problem with these caracteres. Best, Helmi KHALIFA

helmi_khalifa · ‎10-24-2019

I just tried it and it works for me as you can see in the print screen below: Are you sure that the table is not empty ? Best, Helmi

helmi_khalifa · ‎10-24-2019

I am not sure that you cant use it as your verison of hbase is 2.1.0 > 2.0.3 right? could you try it and share the result of RIT ? Best, Helmi

helmi_khalifa · ‎10-24-2019

Hi @Algrach , Could you try this and tell me if it works ? create view if not exists tdv.test_rus as select * from tdv.test_t_rus where c1 = '<the value>'; Best, Helmi KHALIFA

helmi_khalifa · ‎10-24-2019

Hi @sachith , Use QueryRecord processor and configure/enable Reader/Writer controller services Add custom sql query as new property to the processor QueryRecord configs: select id,age, CASE WHEN name='' THEN 'abc' end name from FLowfile The output flowfile from QueryRecord processor will have your desired result id,name,age 1,sachith,29 2,abc,17 Best, Helmi KHALIFA

helmi_khalifa · ‎10-24-2019

Hi @Li ; Did you try hbck1 before ? it is deprecated and if you tried it it seems that it complicate the situation with hbase 2.x https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2 Best, Helmi KHALIFA

helmi_khalifa · ‎09-24-2019

HI @hadoopguy Yes there is an impact you will have longer processing time and the operations will be queued. You have to carefully handle the timeout in your jobs. Best, @helmi_khalifa

helmi_khalifa · ‎09-24-2019

Hi Suresh, There is no command but you can easily find the information on the HBase Web UI. http://host:16010/master-status#baseStats Best, Helmi KHALIFA

helmi_khalifa · ‎06-20-2019

Hi rename the file gateway.jks mv /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks.bck when you start the know instance it will create a new certificate. Best, Helmi KHALIFA

helmi_khalifa · ‎06-20-2019

Hi rename the file gateway.jks mv /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks /var/lib/knox/data-2.6.4.0-91/security/keystores/gateway.jks.bck when you start the know instance it will create a new certificate. Best, Helmi KHALIFA

helmi_khalifa · ‎05-19-2019

Hi, I am encountring slow opertaions problems on HBase cluster. Some queries took more than One minute. When i checked the RegionServers logs i found too frequent major compactions every day on the same Column Family and same regions although we have hbase default parameters (Major compaction once a week)? Any idea how to solve this ? Thanks. Helmi KHALIFA

helmi_khalifa · ‎05-17-2019

Hi Sanket, Did you solve this problem? I am encountring the same problem.. many Major compaction the same day in the same Column Family and the same server? Thanke Helmi KHALIFA

helmi_khalifa · ‎02-25-2019

Hi Geoffrey, Thank you for your answer. I checked the pdf but there are manu differences between yours and mine as it is hdp 2.6.5 Vs hdp 3.1.0.0-78 Thanks, Helmi

helmi_khalifa · ‎02-25-2019

Hi, I installed a cluster HDP-3.1.0.0 (3.1.0.0-78) with HDFS 3.1.1 And I can't execute even an hdfs dfs -ls / Did any one encountred the same problems , please ? Here the errors : /usr/hdp/3.1.0.0-78//hadoop-hdfs/bin/hdfs.distro: line 239: hadoop_abs: command not found /usr/hdp/3.1.0.0-78//hadoop-hdfs/bin/hdfs.distro: line 248: hadoop_need_reexec: command not found /usr/hdp/3.1.0.0-78//hadoop-hdfs/bin/hdfs.distro: line 256: hadoop_verify_user_perm: command not found /usr/hdp/3.1.0.0-78//hadoop-hdfs/bin/hdfs.distro: line 267: hadoop_add_client_opts: command not found /usr/hdp/3.1.0.0-78//hadoop-hdfs/bin/hdfs.distro: line 274: hadoop_subcommand_opts: command not found /usr/hdp/3.1.0.0-78//hadoop-hdfs/bin/hdfs.distro: line 277: hadoop_generic_java_subcmd_handler: command not found Best, Helmi KHALIFA

helmi_khalifa · ‎12-20-2018

hi Muji, Great job 🙂 just missing a ',' after : B_df("_c1").cast(StringType).as("S_STORE_ID") // Assign column names to the Region dataframe val storeDF = B_df.select( B_df("_c0").cast(IntegerType).as("S_STORE_SK"), B_df("_c1").cast(StringType).as("S_STORE_ID"), B_df("_c5").cast(StringType).as("S_STORE_NAME") )

helmi_khalifa · ‎08-20-2018

Hi Neeraj, Allowing read and wright to all users to Poenix SYSTEM tables is not really secure. Is there any solution to avoid it? Thanks Helmi

helmi_khalifa · ‎07-16-2018

Hi, I execute an inner join on 2 tables of 10 Millions records each and i have an outofmemory issue. The columns are both indexed. phoenix.query.maxServerCacheBytes=2147483648 phoenix.query.maxGlobalMemoryPercentage=35 Error: Encountered exception in sub plan [0] execution. (state=,code=0) java.sql.SQLException: Encountered exception in sub plan [0] execution. at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:201) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:145) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:140) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:281) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:266) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:265) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1444) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:808) at sqlline.SqlLine.begin(SqlLine.java:681) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) at java.io.DataOutputStream.writeInt(DataOutputStream.java:200) at org.apache.phoenix.util.TupleUtil.write(TupleUtil.java:152) at org.apache.phoenix.join.HashCacheClient.serialize(HashCacheClient.java:125) at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:85) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:387) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:169) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:165) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)

helmi_khalifa · ‎06-07-2018

Hi, We need to index some columns on hbase tables. Any recommandations or best practises using the secondary-index , please? Thanks Helmi KHALIFA

helmi_khalifa · ‎03-06-2018

Hi, i wanna use hbase rsgroups too but it seems that it is not enough mature..? I am using hbase 1.1.2 and i tested the rsgroup properties and it works. Did you still have the same problem, yet? BR, Helmi Khalifa

helmi_khalifa · ‎01-30-2018

thank you @Josh Elser 🙂

Online	Offline
Last Visited	‎01-20-2022 03:59 PM

Date Registered	‎08-05-2016 07:49 AM
Last Visited	‎01-20-2022 03:59 PM
Total Messages Posted	52
Total Kudos Received	1

Re: How to Load data from hdfs Multi level directo...

Re: Optimizing HBase I/O for Large Scale Hadoop Im...

Re: What is expected behavior of spark.streaming.b...

Re: Can I use SparkSQL on a cluster using Hive on ...

Re: Can I use SparkSQL on a cluster using Hive on ...

Re: Can I use SparkSQL on a cluster using Hive on ...

Re: csv imported from local its not taking date va...

Re: csv imported from local its not taking date va...

Re: csv imported from local its not taking date va...

Re: yarn is not utilizing the cluster resources a...

Re: Hive - Convert Formatted String to Timestamp

Re: Hive Create View as select problem

Re: Hive Create View as select problem

Re: cdh6.3.X hbase version problems

Re: Hive Create View as select problem

Re: How to add a default value for a csv

Re: cdh6.3.X hbase version problems

Re: How to move all the regions of a region server...

Re: How to find number of regions per region serve...

Re: Knox Gateway Start fail

Re: Knox Gateway Start fail

Too frequent major compaction every day ?

Re: Too frequent major compaction

Re: problem executing ls with hdfs 3.1

problem executing ls with hdfs 3.1

Re: Specify Schema for CSV files with no header an...

Re: Apache Ranger and HBase

Inner join Phoenix query out of memory ?

What are the best practices for HBase secondary in...

Re: multi-tenancy HBase - grouping of Region Serve...

Re: HBase how to configure the table location ?