Member since
12-06-2016
40
Posts
5
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
719 | 01-03-2017 02:53 PM | |
966 | 12-29-2016 05:02 PM | |
3882 | 12-22-2016 06:34 PM |
11-16-2017
10:17 AM
Hi, I started my first Flume successfully and gather data from telnet port and put it into HDFS. Now I try to use env variable but it can't recognise the variable. log.txtcollect-log-hdfsconf.txtflume-envsh.txt Following the change done in the source configuration: *********************** a1.sources.r1.port = ${NC_PORT} *********************** Following the lines added to conf/flume-env.sh: *********************** export JAVA_HOME=/opt/jdk1.8.0_144
export NC_PORT=44444 *********************** Another point, it show me always a warning that it doesn't recognise the JAVA_HOME: Warning: JAVA_HOME is not set! Attached the agent conf file, env file and the log file. Thank you in advance. Regards, W-H
... View more
Labels:
- Labels:
-
Apache Flume
02-23-2017
12:36 PM
Hi, After enabling Hive LLAP and doing some property change https://hortonworks.com/hadoop-tutorial/interactive-sql-hadoop-hive-llap/ , Phoenix Query Server become unable to stay on, it shutdown after few seconds of restarting. It start successfully but it shutdown automatically after few seconds. Any idea please? Thank you.
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Phoenix
02-11-2017
11:39 PM
@mhegedus In fact the cluster is one single machine (centos 7). HDP 2.5 installed.
... View more
02-11-2017
09:35 PM
@mhegedus Hi, Thank you very much for your answer. When listing the tables in namespace i don't see the backup table hbase(main):001:0> list_namespace
NAMESPACE
default
hbase
2 row(s) in 0.6120 seconds
hbase(main):002:0> list_namespace_tables 'hbase'
TABLE
meta
namespace
2 row(s) in 0.1050 seconds I don't have another backed up table to use it to recreate backup table. The last chance was to recreate the backup table. but he tell me that the table already exist. Any help please? hbase(main):001:0> create 'hbase:backup', {NAME => 'meta', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'session', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
ERROR: Table already exists: hbase:backup!
... View more
02-11-2017
09:09 PM
@mqureshi Hi, please find here the link for the tutorial in HortonWorks website: http://hortonworks.com/hadoop-tutorial/introduction-apache-hbase-concepts-apache-phoenix-new-backup-restore-utility-hbase/#section_4
... View more
02-10-2017
02:34 PM
Hi, Trying to create a full hbase backup using the following command (Hello World Tutorial series, LAB 4): hbase@vds001:~$ hbase backup create full hdfs://MACHINE_NAME:8020/user/hbase/backup I get the following error: 2017-02-10 15:22:16,146 INFO [main] util.BackupClientUtil: Using existing backup root dir: hdfs://vds001.databridge.tn:8020/user/hbase/backup
Backup session finished. Status: FAILURE
2017-02-10 15:22:17,093 ERROR [main] util.AbstractHBaseTool: Error running command-line tool
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException): Failed 1 action: Table 'hbase:backup' was not found, got: driver_dangerous_event.: 1 time,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:234)
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:214)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1751)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1422)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1013)
at org.apache.hadoop.hbase.backup.impl.BackupSystemTable.updateBackupInfo(BackupSystemTable.java:127)
at org.apache.hadoop.hbase.backup.impl.BackupManager.updateBackupInfo(BackupManager.java:396)
at org.apache.hadoop.hbase.backup.master.FullTableBackupProcedure.beginBackup(FullTableBackupProcedure.java:130)
at org.apache.hadoop.hbase.backup.master.FullTableBackupProcedure.executeFromState(FullTableBackupProcedure.java:502)
at org.apache.hadoop.hbase.backup.master.FullTableBackupProcedure.executeFromState(FullTableBackupProcedure.java:69)
at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:107)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:443)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:934)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:736)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:689)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:73)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:416)
Any help please? Thank you in advance.
... View more
Labels:
- Labels:
-
Apache HBase
02-08-2017
09:25 PM
@mhegedus thank you very much 🙂
... View more
02-08-2017
03:30 PM
Hello, Trying to achieve LAB 4 in Hello World series, I'm facing following error when loading data into Hbase table hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, -Dimporttsv.columns="HBASE_ROW_KEY,events:driverId,events:driverName,events:eventTime,events:eventType,events:latitudeColumn,events:longitudeColumn,events:routeId,events:routeName,events:truckId" driver_dangerous_event hdfs://sandbox.hortonworks.com:/tmp/data.csv
SyntaxError: (hbase):8: syntax error, unexpected ','
hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, -Dimporttsv.columns="HBASE_ROW_KEY,events:driverId,events:driverName,events:eventTime,events:eventType,events:latitudeColumn,events:longitudeColumn,events:routeId,events:routeName,events:truckId" driver_dangerous_event hdfs://sandbox.hortonworks.com:/tmp/data.csv
^
It looks that it does'nt accept the separator syntax!! any idea please?
... View more
- Tags:
- Data Processing
- HBase
Labels:
- Labels:
-
Apache HBase
01-20-2017
09:23 PM
Hi Aldo, in HDFS. The parameter is called "NameNode Java heap size".
... View more
01-05-2017
10:39 AM
@Michael Young Hi, Long type is not permitted in Hive, trying to use float for riskfactor column, but error persist. any idea please? log-error.txt
... View more
01-04-2017
06:57 PM
@Michael Young Hi, Why it try to convert the data type? the statement is "create as select", then the result table columns data type should follow the origin table. Why converting the data type of this column? Also, double data type is supported by Hive and when trying the same query in Hive it works fine (create table riskfactor as select * from finalresults;)
... View more
01-04-2017
06:36 PM
1 Kudo
Hi, Trying to finalize lab 4 : Riskfactor Analysis with Spark. When executing the final instruction, I got the following error: %spark
hiveContext.sql("create table riskfactor as select * from finalresults") org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
stage 0.0 (TID 7, vds001.databridge.tn): java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to
org.apache.hadoop.io.LongWritable Enclosed all log records. Any idea please?spark-error.txt
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
01-04-2017
06:29 PM
Now it works fine thank you.
... View more
01-04-2017
05:34 PM
Hi, Trying to complete Lab 4 (Riskfactor Analysis with Spark), i got the following error for the last instruction: %spark
hiveContext.sql("create table riskfactor as select * from finalresults") <console>:28: error: not found: value hiveContext hiveContext.sql("create table riskfactor as select * from finalresults") ^ Any idea please? Regards,
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
-
Apache Zeppelin
01-04-2017
10:59 AM
1 Kudo
Hi, Trying to finalize lab 4 : Riskfactor Analysis with Spark. All works fine till final step, save data into riskfactor table with "create table as select" statement. I got a connection problem. Hive works fine. Any idea please? java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:189)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:258)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:281)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:328)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) following the last executed command : %spark hiveContext.sql("create table riskfactor as select * from finalresults")
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
-
Apache Zeppelin
01-03-2017
02:53 PM
@Jay SenSharma Hi, I added the IP@ of the VDS to my local hosts file and it's done 🙂 Thank you for your help.
... View more
01-03-2017
02:53 PM
@Jay SenSharma Hi, I added the IP@ of the VDS to my local hosts file and it's done 🙂 Thank you for your help.
... View more
01-03-2017
01:05 PM
@Jay SenSharma The current cluster is composed by one machine. Please note that the machine is a remote VDS (virtual dedicated server) that i can access only via ssh or web. my local machine is windows 10. Following the output of hostname commands root@vds001:~# hostname vds001 root@vds001:~# hostname -f vds001.databridge.tn following the output of the API call : http://197.12.8.49:8080/api/v1/clusters/DataBridge/hosts/ {
"href" : "http://197.12.8.49:8080/api/v1/clusters/DataBridge/hosts/",
"items" : [
{
"href" : "http://197.12.8.49:8080/api/v1/clusters/DataBridge/hosts/vds001.databridge.tn",
"Hosts" : {
"cluster_name" : "DataBridge",
"host_name" : "vds001.databridge.tn"
}
}
] } following the zeppellin UI call : http://197.12.8.49:9995 (i can't put the hostname instead of the ip@) it works --> should i configure the host file in my local machine? Regards,
... View more
01-03-2017
09:44 AM
Hi, When trying to access zeppelin view from ambari, i got the following error: hostname’s server DNS address could not be found. I can access it through separate page via port 9995. But I would know why it's impossible from ambari. Please find enclosed the screen shot.zeppelin-view-1.jpgzeppelin-view-2.jpg Ragards,
... View more
Labels:
- Labels:
-
Apache Zeppelin
12-29-2016
05:02 PM
Some errors in lab --> Pig Script must be as follow: a = LOAD 'geolocation' USING org.apache.hive.hcatalog.pig.HCatLoader(); b = FILTER a BY event != 'normal'; c = FOREACH b GENERATE driverid, (int) '1' as occurance; d = GROUP c BY driverid; e = FOREACH d GENERATE group as driverid, SUM(c.occurance) as totevents; g = LOAD 'drivermileage' using org.apache.hive.hcatalog.pig.HCatLoader(); h = join e by driverid, g by driverid; final_data = foreach h generate $0 as driverid, $1 as totevents, $3 as totmiles, (float) $3/$1 as riskfactor; store final_data into 'riskfactor' using org.apache.hive.hcatalog.pig.HCatStorer(); riskfactor table in Hive must be as follow: CREATE TABLE riskfactor (driverid string,totevents bigint,totmiles double,riskfactor float)
STORED AS ORC;
... View more
12-29-2016
10:51 AM
@milind pandit My problem is not linked to data type. Please find enclosed the entire log file.job-1482423183850-0021-logs.txt
... View more
12-29-2016
10:27 AM
@WeiHsiang Tseng, Hi, I'm facing the same problem. did you resolve it? Thanks.
... View more
12-29-2016
09:42 AM
Hi, using ambari 2.4.1.0 and HDP 2.5 I'm trying to execute the first lab instruction : a = LOAD 'geolocation' USING org.apache.hive.hcatalog.pig.HCatLoader(); I add the following argument to let Pig know the HCatLoader() class : -useHCatalog I get the following log : can any one help me to fix this? thanks. ls: cannot access /hadoop/yarn/local/usercache/admin/appcache/application_1482423183850_0022/container_1482423183850_0022_01_000002/hive.tar.gz/hive/lib/slf4j-api-*.jar: No such file or directory
ls: cannot access /hadoop/yarn/local/usercache/admin/appcache/application_1482423183850_0022/container_1482423183850_0022_01_000002/hive.tar.gz/hive/hcatalog/lib/*hbase-storage-handler-*.jar: No such file or directory
WARNING: Use "yarn jar" to launch YARN applications.
16/12/29 10:28:37 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
16/12/29 10:28:37 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
16/12/29 10:28:37 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2016-12-29 10:28:37,537 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0.2.5.3.0-37 (rexported) compiled Nov 30 2016, 02:28:11
2016-12-29 10:28:37,537 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/admin/appcache/application_1482423183850_0022/container_1482423183850_0022_01_000002/pig_1483003717522.log
2016-12-29 10:28:38,970 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found
2016-12-29 10:28:39,216 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://vds002.databridge.tn:8020
2016-12-29 10:28:41,059 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-9b551f9a-3393-4ab2-93ea-de21982a11cc
2016-12-29 10:28:42,237 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://vds002.databridge.tn:8188/ws/v1/timeline/
2016-12-29 10:28:42,704 [main] INFO org.apache.pig.backend.hadoop.PigATSClient - Created ATS Hook
2016-12-29 10:28:44,448 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2016-12-29 10:28:44,521 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://vds002.databridge.tn:9083
2016-12-29 10:28:44,588 [main] INFO hive.metastore - Connected to metastore.
2016-12-29 10:28:45,238 [main] INFO org.apache.pig.Main - Pig script completed in 8 seconds and 278 milliseconds (8278 ms)
... View more
- Tags:
- Data Processing
- Pig
Labels:
- Labels:
-
Apache Pig
12-22-2016
06:34 PM
2 Kudos
@Jay SenSharma the real problem is the namenode heap of memory. When History Server try to start, The memory usage of the NameNode climbs quickly to exceed the limit of 1 Gega byte (default configuration) and causes the service to fall. When changing max memory heap to 3 Gb it works fine. I installed previously ambari 2.4.0.1 and i don't seen this behaviour (2.4.2.0 same behaviour as 2.4.1.0). Do you know why?
... View more
12-21-2016
07:10 PM
@Jay SenSharma hostname -f : vds002.databridge.tn netstat -tnlpa | grep 50070 : nothing root@vds002:~# netstat -tnlpa | grep 50070 root@vds002:~# how to enable communication on this port. trying : firewall-cmd --add-port 50070/tcp --permanent but no effect.
... View more
12-21-2016
03:03 PM
Hi Jay, Thanks for the help. no files found under /var/log/hadoop-mapreduce/mapred the command doesn't works : curl: (7) Failed connect to vds002.databridge.tn:50070; Connection refused The cluster is one machine. The firewalld is disabled. any idea plz?
... View more
12-21-2016
10:04 AM
After installing fresh HDP 2.5.3 cluster (ambari 2.4.1.0), all sevices (default selection) installed successfully without any warning. When Starting services, History Server fail to start and make mapreduce fail also. curl: (52) Empty reply from server
100
... View more
Labels:
- Labels:
-
Apache Hadoop