Member since
10-27-2014
38
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1990 | 04-07-2015 01:13 AM | |
10489 | 12-18-2014 02:21 AM |
04-26-2015
09:08 PM
Thanks denloe, The command "hbase hbck -repair" solve 6 inconsistencies. And the step that delete znodes make this command working, i think, because i've been trying "hbase hbck -repair" before but it just stuck. HBase working fine now, thank you.
... View more
04-23-2015
03:20 AM
Thanks Gonzalo, I've been try to delete the "/hbase" folder ( just move it to /tmp), but when i restart Hbase, the hbase master can't start, as i remember is because of user authentication of "/" belong to HDFS, and i don't want to chown "/" to hbase. Even after i create a new "/hbase" and chow it to hbase:hbase, the hbase master still won't start unless i move back the old "/hbase" About znode in zookeeper, i really don't know much about it, i just know my ZooKeeper Znode Parent is "/hbase", do i just delete this folder or i have to delete something elsse ?
... View more
04-23-2015
02:27 AM
Ok, the Hbase master stop shutdown now after i delete this file hdfs://00master.mabu.com:8020/hbase/WALs/02slave.mabu.com,60020,1429168945778-splitting but now i got another problem in this topic: http://community.cloudera.com/t5/Storage-Random-Access-HDFS/HBase-Region-in-Transition/m-p/26707#M1176
... View more
04-23-2015
02:24 AM
Thanks for reply Gonzalo, I can't do anything with hbase right now (can't disable, drop table, can't event view sample table on hue), everything just stuct, so i just manually delete all table data on HDFS, just keep the default(sample) table, but it still won't work.
... View more
04-23-2015
12:51 AM
HBase keep having region in transition: Regions in Transition Region State RIT time (ms) 1588230740 hbase:meta,,1.1588230740 state=FAILED_OPEN, ts=Thu Apr 23 12:15:49 ICT 2015 (8924s ago), server=02slave.mabu.com,60020,1429765579823 8924009 Total number of Regions in Transition for more than 60000 milliseconds 1 Total number of Regions in Transition 1 I've try "sudo -u hbase hbase hbck -repair" and also "unassign ' hbase:meta,,1.1588230740'" but still can't fix the problem.
... View more
Labels:
04-20-2015
09:39 PM
Recently my cluster have an error in HBase due to the HBase master auto shutdown.
When i restart cluster, it work fine, but after about 10 minutes, it will shutdown again, even i didn't do anything with HBase, here is the log file:
11:14:09.706 AM WARN org.apache.hadoop.hbase.master.SplitLogManager
Error splitting /hbase/splitWAL/WALs%2F02slave.mabu.com%2C60020%2C1429168945778-splitting%2F02slave.mabu.com%252C60020%252C1429168945778.1429268912230.meta
11:14:09.707 AM WARN org.apache.hadoop.hbase.master.SplitLogManager
error while splitting logs in [hdfs://00master.mabu.com:8020/hbase/WALs/02slave.mabu.com,60020,1429168945778-splitting] installed = 1 but only 0 done
11:14:09.707 AM FATAL org.apache.hadoop.hbase.master.HMaster
Master server abort: loaded coprocessors are: []
11:14:09.707 AM FATAL org.apache.hadoop.hbase.master.HMaster
Unhandled exception. Starting shutdown.
java.io.IOException: error or interrupted while splitting logs in [hdfs://00master.mabu.com:8020/hbase/WALs/02slave.mabu.com,60020,1429168945778-splitting] Task = installed = 1 done = 0 error = 1
at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:360)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:416)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:308)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:299)
at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1085)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:881)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:613)
at java.lang.Thread.run(Thread.java:745)
11:14:09.864 AM ERROR org.apache.hadoop.hbase.master.HMasterCommandLine
Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2822)
... View more
Labels:
04-15-2015
09:58 PM
Oh, I finally do it, follow is my hql: SELECT id, part.lock, part.key FROM mytable EXTERNAL VIEW explode(parts) parttable AS part; many thanks chrisf !
... View more
04-15-2015
02:54 AM
Thanks for your reply chrisf, I've been trying to use LATERAL VIEW explode for week but still can't figure how to use it, can you give me an example from my first post. I also try json-serde in HiveContext, i can parse table, but can't querry although the querry work fine in Hive. EX: + In both Hive anh HiveContext, i can parse table: CREATE EXTERNAL TABLE data( parts array<struct<locks:STRING, key:STRING>> ) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' LOCATION '/user/hue/...'; + Then in Hive, i can use this: SELECT parts.locks FROM data; but it will return error in HiveContext. Looking forward for reply, thanks !
... View more
04-07-2015
01:27 AM
I have a simple JSON dataset as below. How do I query all parts.lock JSON: {
"id" : 1 ,
"name" : "A green door" ,
"price" : 12.50 ,
"tags" : [ "home" , "green" ],
"parts" : [
{
"lock" : "One lock" ,
"key" : "single key"
},
{
"lock" : "2 lock" ,
"key" : "2 key"
}
]
} Query: select id , name , price , parts . lock from product The point is if I use parts[0].lock it will return one row as below: { u 'price' : 12.5 , u 'id' : 1 , u '.lock' : { u 'lock' : u 'One lock' , u 'key' : u 'single key' }, u 'name' : u 'A green door' } But I want to return all the locks in the parts structure. Please help me with this, thanks !
... View more
04-07-2015
01:13 AM
Found my solution, i need to add 2 file: + db.hsqldb.properties + dn.hsqldb.script to the oozie job, then the job just work fine, still don't understand why because i don't need these 2 file when import.
... View more
03-11-2015
01:58 AM
Hi, first, i try to import a sample data from MySQL to HDFS, using oozie-sqoop workflow, evething OK. Then i try to export the result back to Mysql, the sqoop export command is OK Next, i use oozie-sqoop workflow and got the error : Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] I've try manny way: about file mysql-connector-java.....jar + I upload it to HDFS and add it to the file path. + I also upload it to /user/oozie/share/lib/lib_.../sqoop/ and also to /user/oozie/share/lib/sqoop/ and chmod 777 to it. + I also copy it to /opt/cloudera/parcels/CDH-5.3.2.../lib/sqoop/lib/ and to /var/lib/sqoop/ and chmod 777 too here is the job definition: <workflow-app name="sqoop_export" xmlns="uri:oozie:workflow:0.4"> <start to="export_potluck"/> <action name="export_potluck"> <sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <command>sqoop export --connect jdbc:mysql://192.168.6.10/mabu --username root --password 123456 --table potluck --export-dir /user/hue/mabu/test_e</command> <file>/user/hue/mabu/oozie/mysql-connector-java-5.1.34-bin.jar#mysql-connector-java-5.1.34-bin.jar</file> </sqoop> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app> and here is the job configuration: <configuration> <property> <name>hue-id-w</name> <value>31</value> </property> <property> <name>user.name</name> <value>hue</value> </property> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> <property> <name>mapreduce.job.user.name</name> <value>hue</value> </property> <property> <name>oozie.wf.rerun.failnodes</name> <value>false</value> </property> <property> <name>nameNode</name> <value>hdfs://00master.mabu.com:8020</value> </property> <property> <name>jobTracker</name> <value>00master.mabu.com:8032</value> </property> <property> <name>oozie.wf.application.path</name> <value>hdfs://00master.mabu.com:8020/user/hue/oozie/workspaces/_hue_-oozie-31-1425982013.65</value> </property> </configuration> Really appreciate the help, thanks !
... View more
Labels:
03-02-2015
08:11 PM
I've run an example hive coordinators. The job going well but here are some issues i meet: 1. I'm from VietNam, when i choose timezone = GMT+7, i get error, when searching for this error i found that it should be GMT+07:00, and problem is GMT+07:00 won't show up in the combo box. 2. So i choosing Asia/Ho_Chi_Minh instead, and another problem is i must choose the timezone base on Universaltime. Example: I want to run from 10:00 AM to 03:00 PM 3 March, i have to choose the time is from 03:00 AM to 08:00 AM 3 March base on UTC, why i can't edit like my own time even afte i choose Asia/Ho_Chi_Minh. 3. Final problem is when i run this coordinators: the timezone in the job info show up a complete different. Here is my oozie deffinion: <coordinator-app name="simple_1" frequency="0,5,10 * * * *" start="2015-03-03T03:00Z" end="2015-03-03T08:00Z" timezone="Asia/Ho_Chi_Minh" xmlns="uri:oozie:coordinator:0.2" > and here is the info of the first hour: Mon, 02 Mar 2015 19:00:00 Mon, 02 Mar 2015 19:05:00 Mon, 02 Mar 2015 19:10:00 Now, why it is 02 Mar 19:00 ?
... View more
Labels:
02-24-2015
04:44 AM
I also try with a different way without using sqoop shell: I use the following command: sqoop export --driver com.mysql.jdbc.Driver --connect jdbc:mysql://127.0.0.1/mabu --username root -P --table sample7 --export-dir /user/hue/mabu --input-fields-terminated-by '\t' --verbose And got this error: Error: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
... View more
02-24-2015
02:22 AM
Because Sqoop on He currnently unavailable so i try sqoop2 on terminal. I've succes create 2 link from hdfs-connector and generic-jdbc-connector (use com.mysql.jdbc.Driver). Then i create 2 job (1 import an example table from mysql to hdfs and 1 export file sample_07 from hdfs to mysql table). I start job 2 ok, but all 2 job return FAILED when i check their status. Here is my job config: Import job: schema name: mabu table name: potluck .... default ... output directory: /user/home/mabu extractors: 2 loaders: 2 Export job: Input directory: /user/hue/mabu schema name: mabu table name: sample7 ... default ... extractors: 2 loaders: 2
... View more
02-23-2015
07:03 PM
I've just install CDH 5.3.1 and Sqoop on Hue still don't work, is it still no fix in 5.3.1 ?
... View more
02-08-2015
07:55 PM
I've just install Flume service on my cluster using Cloudera Manager, 2 of my agents is working but i got 1 agent bad health with bad health : The Cloudera Manager Agent is not able to communicate with this role's web server. This is the error log: org.apache.flume.node.PollingPropertiesFileConfigurationProvider : Failed to load configuration data. Exception follows. org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource Can someone help me with this, i'm new to Flume. Regards, Tu Nguyen
... View more
02-02-2015
07:00 PM
Thanks for reply, I've increase the datanode heap size to 1Gb , and my datanode work well so far, but there is one more thing: I upload data (just using -put command) to my cluster (2736 folder with 200 file each folder (about 15kB each file) ) and my cluster go from 350k up to over 700k blocks each node, then the warning too many block prompted. I really don't understand why there are so many blocks because the total size of data is just about 5GB. Regards, Tu Nguyen
... View more
01-29-2015
05:45 AM
Thanks for reply, I got 3 datanodes, the one that shutdown is on master host, this is the information: 00master - block: 342823 - block pool used: 53,95GB (6,16%) 01slave - block: 346297 - block pool used: 54,38GB (12,46%) 02slave - block: 319262 - block pool used: 48,39GB (33,23%) and this is my heap setting DataNode Default Group / Resource Management : 186 MB DataNode Group 1 / Resource Management 348 MB Regards, Tu Nguyen
... View more
01-28-2015
01:23 AM
Hi, i'm using CDH5.3 i've got a cluster with 3 host: 1 master host have namenode & datanode, 2 host just have datanode, Everything run fine till recently when i run a hive Job, the datanode on the master shutdown and i got the error missing block & underreplicated blocks. Here is the error on the master's datanode: 3:35:09.545 PM ERROR org.apache.hadoop.hdfs.server.datanode.DirectoryScanner Error compiling report java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:545) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:422) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:403) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:359) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space 3:35:09.553 PM INFO org.apache.hadoop.hdfs.server.datanode.DataNode opWriteBlock BP-993220972-192.168.0.140-1413974566312:blk_1074414393_678864 received exception java.io.IOException: Premature EOF from inputStream 3:35:09.553 PM ERROR org.apache.hadoop.hdfs.server.datanode.DirectoryScanner Exception during DirectoryScanner execution - will continue next cycle java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:549) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:422) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:403) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:359) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:545) ... 10 more Caused by: java.lang.OutOfMemoryError: Java heap space 3:35:09.553 PM ERROR org.apache.hadoop.hdfs.server.datanode.DataNode 00master.mabu.com:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.6.10:48911 dst: /192.168.6.10:50010 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:468) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:772) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:724) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) at java.lang.Thread.run(Thread.java:745) Can someone help me to fix this ? Thanks !
... View more
01-24-2015
12:49 AM
Can someone help, i've been stuck here for a week ? Or you can just give me a link to a pig forum ? ...
... View more
01-21-2015
01:46 AM
Sorry about the post above this post, this is a complete one: Thanks joey, This look like the answer for me, but for the example in the link you give : A = load 'T';
B = group A by si
C = foreach B {
C1 = order A by d;
generate flatten(Stitch(C1, Over(C1.f, 'sum(float)')));
}
D = foreach C generate s, $9; I don't quite understand: i can't use GROUP BY and ORDER in this telematic data. So i try to run some exsample myselft: I just want to DUMP row x and another row with value is above x 1 line: EX: X: 2, 1, 3, 4 ... ,5 , 6. X': 0, 2, 1, 3, ...., 5 . B = LOAD '/user/hue/pig/drivers/drivers/1002/1.csv' using PigStorage(',') AS (x: float, y: float); C = FOREACH B GENERATE x , Over(x, lead, -1, 0, 1, 0); DUMP C; and got this error: <file script.pig, line 16, column 13> Failed to generate logical plan. Nested exception: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve Over using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.] I've already try this: B = LOAD '/user/hue/pig/drivers/drivers/1002/1.csv' using PigStorage(',') AS (x: float, y: float); C = FOREACH B GENERATE x , Over(B.x, lead, -1, 0, 1, 0); DUMP C; But still got the same error. Can you give me some more specific advise. Thanks !
... View more
01-21-2015
01:40 AM
Thanks joey. This look like the answer for me, but for the example in the link you give : A = load 'T';
B = group A by si
C = foreach B {
C1 = order A by d;
generate flatten(Stitch(C1, Over(C1.f, 'sum(float)')));
}
D = foreach C generate s, $9; , i don't quite understand: i can't use GROUP BY and ORDER in this telematic data. So i try to run some exsample myselft: I just want to DUMP row x and another row with value is above x 1 line: EX: X: 2, 1, 3, 4 ... B = LOAD '/user/hue/pig/drivers/drivers/1002/1.csv' using PigStorage(',') AS (x: float, y: float); C = FOREACH B GENERATE x , Over(x, lead, -1, 0, 1, 0); DUMP C; and got this error: <file script.pig, line 16, column 13> Failed to generate logical plan. Nested exception: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve Over using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.] I've already try this: Can you give me some more specific advise, I just need to
... View more
01-19-2015
08:13 PM
I have a telematic data like this: X Y 0.1 0.2 0.3 0.1 .... .... I need to calculate the distance and add it to the third row call Dis: X Y Dis 0.1 0.2 0 0.3 0.1 0.22 .... ..... .... How can i calculate this on Pig, i think i need to generate an index column, but still don't know how to do it ? And is there a way that doesn't require generate index column ??? Many thanks !
... View more
Labels:
01-07-2015
07:13 PM
I'm using CM 5.2, When i change the IP on my cluster , then restart CM, it wont detect my new IP address. Then i try to fix by edit /etc/cloudera-scm-agent/config.ini, but still don't work. TILL NOW, I STILL CAN ACCESS TO CLOUDERA MANAGER. Then, i try to update coudera manager to 5.3, hope that the update can auto reconfig the IP, this is when i got the whole new error, i can access to CM, but the HOME, CLUSTER and HOST tab wont show anything and i got this error: A server error has occurred. Send the following information to Cloudera. Path: http://192.168.6.10:7180/cmf/parcel/topLevelCount Version: Cloudera Express 5.3.0 (#166 built by jenkins on 20141218-0505 git: 9ec4939d0a7b563597da611c675725916369a60d) javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: Could not open connection at AbstractEntityManagerImpl.java line 1387 in org.hibernate.ejb.AbstractEntityManagerImpl convert()
... View more
12-18-2014
02:21 AM
Hello masfworld, I've found my solution here, hope this'll help you too: http://community.cloudera.com/t5/Cloudera-Search-Apache-SolrCloud/Solr-Server-not-starting/m-p/4839#M97
... View more
12-18-2014
12:26 AM
My CM version is 5.2.0, my cluster have 1 master and 4 node which contain 4 Impala daemon. I've been using Impala for month, everything is ok, but recently, when i try to connect Tableau to Impala using Impala ODBC, one of my impala daemon fail to start, the other 3 daemon still normal, here's the log: 2:30:56.906 PM INFO authentication.cc:990 Internal communication is not authenticated 2:30:56.906 PM INFO authentication.cc:1011 External communication is not authenticated 2:30:56.907 PM INFO init.cc:135 impalad version 2.0.0-cdh5 RELEASE (build ecf30af0b4d6e56ea80297df2189367ada6b7da7)
Built on Sat, 11 Oct 2014 14:19:47 PST 2:30:56.907 PM INFO init.cc:136 Using hostname: 01slave.mabu.com 2:30:56.907 PM INFO logging.cc:100 Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module=
--print_llvm_ir_instruction_count=false
--unopt_module=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=01slave.mabu.com
--keytab_file=
--krb5_conf=
--krb5_debug_file=
--mem_limit=698351616
--principal=
--log_filename=impalad
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--skip_lzo_version_check=false
--max_row_batches=0
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=00master.mabu.com
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=00master.mabu.com
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/var/lib/impala/udfs
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--scratch_dirs=/impala/impalad
--queue_wait_timeout_ms=60000
--default_pool_max_queued=50
--default_pool_max_requests=20
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=/run/cloudera-scm-agent/process/171-impala-IMPALAD/impala-conf/fair-scheduler.xml
--llama_site_path=/run/cloudera-scm-agent/process/171-impala-IMPALAD/impala-conf/llama-site.xml
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_mem_usage_interval=0
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=false
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_
... View more
12-17-2014
02:07 AM
Hello, i'm currently using CDH 5.2 I've a sample data like this : 0 18.6 36.1 53.7 86.5 but when i'm upload to hive with float type, i got this : 0.0 18.6000003815 36.0999984741 53.7000007629 86.5 I don't understand why 18.6 become 18.6000003815 , but 86.5 is still the same, every other variable is change except the .0 and .5 varible. Can someone explain to me, really appreciate the help, thanks !
... View more
11-24-2014
02:52 AM
Hi, I'm currently trying to enable Spark on Hue. I'm using Spark on Yarn with CDH 5.2. I've install Spark job server, the sbt > re-start command return this: Found 0 errors Found 10 warnings Finished in 1 ms [success] created: /tmp/spark-jobserver/akka-app/target/scalastyle-result.xml [info] Application job-server not yet started [info] Starting application job-server in the background ... job-server Starting spark.jobserver.JobServer.main() [success] Total time: 2 s, completed Nov 24, 2014 5:31:29 PM Problem is, i've change the blacklist option in hue.ini to empty but the desktop configuration tab in Hue still show spark in blacklist, sorry if this is on the wrong topic, i already post this question on hue topic but look like nobody support there. Now, i'm trying with Spark standalone in another guide : http://gethue.com/get-started-with-spark-deploy-spark-server-and-compute-pi-from-your-web-browser/ and get this error why running "./bin/server_package.sh demo" command: Exception encountered when attempting to run suite spark.jobserver.LocalContextSupervisorHaSpec: IO Exception: "java.io.FileNotFoundException: /tmp/spark-job-server-test/sqldao/data/75f50c43-1b57-4fc3-9407-54509287c121.lock.db (Permission denied)"; "/tmp/spark-job-server-test/sqldao/data/75f50c43-1b57-4fc3-9407-54509287c121.lock.db" [90031-170] *** ABORTED *** [info] org.h2.jdbc.JdbcSQLException: IO Exception: "java.io.FileNotFoundException: /tmp/spark-job-server-test/sqldao/data/75f50c43-1b57-4fc3-9407-54509287c121.lock.db (Permission denied)"; "/tmp/spark-job-server-test/sqldao/data/75f50c43-1b57-4fc3-9407-54509287c121.lock.db" [90031-170] [info] at org.h2.message.DbException.getJdbcSQLException(DbException.java:329) [info] at org.h2.message.DbException.get(DbException.java:158) [info] at org.h2.message.DbException.convertIOException(DbException.java:315) [info] at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:265) [info] at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:223) [info] at org.h2.store.FileLock.save(FileLock.java:198) [info] at org.h2.store.FileLock.lockFile(FileLock.java:334) [info] at org.h2.store.FileLock.lock(FileLock.java:129) [info] at org.h2.engine.Database.open(Database.java:541) [info] at org.h2.engine.Database.openDatabase(Database.java:221) [info] ... [info] Cause: java.io.FileNotFoundException: /tmp/spark-job-server-test/sqldao/data/75f50c43-1b57-4fc3-9407-54509287c121.lock.db (Permission denied) [info] at java.io.FileOutputStream.open(Native Method) [info] at java.io.FileOutputStream.<init>(FileOutputStream.java:212) [info] at java.io.FileOutputStream.<init>(FileOutputStream.java:131) [info] at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:257) [info] at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:223) [info] at org.h2.store.FileLock.save(FileLock.java:198) [info] at org.h2.store.FileLock.lockFile(FileLock.java:334) [info] at org.h2.store.FileLock.lock(FileLock.java:129) [info] at org.h2.engine.Database.open(Database.java:541) [info] at org.h2.engine.Database.openDatabase(Database.java:221) Exception encountered when attempting to run suite spark.jobserver.SparkWebUiActorSpec: Await termination timed out after [10 seconds] *** ABORTED *** [info] java.util.concurrent.TimeoutException: Await termination timed out after [10 seconds] [info] at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:759) [info] at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:729) [info] at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86) [info] at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86) [info] at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) [info] at scala.concurrent.Await$.ready(package.scala:86) [info] at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:598) [info] at ooyala.common.akka.AkkaTestUtils$.shutdownAndWait(AkkaTestUtils.scala:25) [info] at spark.jobserver.SparkWebUiActorSpec.afterAll(SparkWebUiActorSpec.scala:99) [info] at org.scalatest.BeforeAndAfterAll$class.afterAll(BeforeAndAfterAll.scala:183) - should be able to save one jar and get it back *** FAILED *** [info] java.io.FileNotFoundException: /tmp/spark-job-server-test/sqldao/data/test-appName0-2014-11-24T17:49:37.555+07:00.jar (Permission denied) [info] at java.io.FileOutputStream.open(Native Method) [info] at java.io.FileOutputStream.<init>(FileOutputStream.java:212) [info] at java.io.FileOutputStream.<init>(FileOutputStream.java:160) [info] at spark.jobserver.io.JobSqlDAO.cacheJar(JobSqlDAO.scala:236) [info] at spark.jobserver.io.JobSqlDAO.saveJar(JobSqlDAO.scala:148) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JobSqlDAOSpec.scala:95) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(JobSqlDAOSpec.scala:90) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(JobSqlDAOSpec.scala:90) [info] at org.scalatest.FunSpec$$anon$1.apply(FunSpec.scala:1600) [info] at org.scalatest.Suite$class.withFixture(Suite.scala:1974) [info] ... [info] - should be able to retrieve the jar file *** FAILED *** [info] java.util.NoSuchElementException: head of empty list [info] at scala.collection.immutable.Nil$.head(List.scala:337) [info] at scala.collection.immutable.Nil$.head(List.scala:334) [info] at spark.jobserver.io.JobSqlDAO$$anonfun$fetchJar$1.apply(JobSqlDAO.scala:210) [info] at spark.jobserver.io.JobSqlDAO$$anonfun$fetchJar$1.apply(JobSqlDAO.scala:202) [info] at scala.slick.backend.DatabaseComponent$DatabaseDef$class.withSession(DatabaseComponent.scala:31) [info] at scala.slick.jdbc.JdbcBackend$DatabaseFactoryDef$$anon$4.withSession(JdbcBackend.scala:61) [info] at spark.jobserver.io.JobSqlDAO.fetchJar(JobSqlDAO.scala:201) [info] at spark.jobserver.io.JobSqlDAO.fetchAndCacheJarFile(JobSqlDAO.scala:195) [info] at spark.jobserver.io.JobSqlDAO.retrieveJarFile(JobSqlDAO.scala:188) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$2$$anonfun$apply$mcV$sp$2.apply$mcV$sp(JobSqlDAOSpec.scala:111) [info] ... - should save a new JobInfo and get the same JobInfo *** FAILED *** [info] java.util.NoSuchElementException: head of empty list [info] at scala.collection.immutable.Nil$.head(List.scala:337) [info] at scala.collection.immutable.Nil$.head(List.scala:334) [info] at spark.jobserver.io.JobSqlDAO.spark$jobserver$io$JobSqlDAO$$queryJarId(JobSqlDAO.scala:230) [info] at spark.jobserver.io.JobSqlDAO$$anonfun$saveJobInfo$1.apply(JobSqlDAO.scala:279) [info] at spark.jobserver.io.JobSqlDAO$$anonfun$saveJobInfo$1.apply(JobSqlDAO.scala:276) [info] at scala.slick.backend.DatabaseComponent$DatabaseDef$class.withSession(DatabaseComponent.scala:31) [info] at scala.slick.jdbc.JdbcBackend$DatabaseFactoryDef$$anon$4.withSession(JdbcBackend.scala:61) [info] at spark.jobserver.io.JobSqlDAO.saveJobInfo(JobSqlDAO.scala:275) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$4$$anonfun$apply$mcV$sp$8.apply$mcV$sp(JobSqlDAOSpec.scala:176) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$4$$anonfun$apply$mcV$sp$8.apply(JobSqlDAOSpec.scala:174) [info] ... [info] - should be able to get previously saved JobInfo *** FAILED *** [info] Set() did not equal Set(test-id0) (JobSqlDAOSpec.scala:193) [info] - Save another new jobInfo, bring down DB, bring up DB, should JobInfos from DB *** FAILED *** [info] java.util.NoSuchElementException: head of empty list [info] at scala.collection.immutable.Nil$.head(List.scala:337) [info] at scala.collection.immutable.Nil$.head(List.scala:334) [info] at spark.jobserver.io.JobSqlDAO.spark$jobserver$io$JobSqlDAO$$queryJarId(JobSqlDAO.scala:230) [info] at spark.jobserver.io.JobSqlDAO$$anonfun$saveJobInfo$1.apply(JobSqlDAO.scala:279) [info] at spark.jobserver.io.JobSqlDAO$$anonfun$saveJobInfo$1.apply(JobSqlDAO.scala:276) [info] at scala.slick.backend.DatabaseComponent$DatabaseDef$class.withSession(DatabaseComponent.scala:31) [info] at scala.slick.jdbc.JdbcBackend$DatabaseFactoryDef$$anon$4.withSession(JdbcBackend.scala:61) [info] at spark.jobserver.io.JobSqlDAO.saveJobInfo(JobSqlDAO.scala:275) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$4$$anonfun$apply$mcV$sp$10.apply$mcV$sp(JobSqlDAOSpec.scala:204) [info] at spark.jobserver.io.JobSqlDAOSpec$$anonfun$4$$anonfun$apply$mcV$sp$10.apply(JobSqlDAOSpec.scala:197) [info] ... [info] - saving a JobInfo with the same jobId should update the JOBS table *** FAILED *** [info] 0 did not equal 2 (JobSqlDAOSpec.scala:231) [error] Failed: Total 123, Failed 8, Errors 0, Passed 113, Skipped 2 [error] Failed tests: [error] spark.jobserver.LocalContextSupervisorHaSpec [error] spark.jobserver.io.JobSqlDAOSpec [error] spark.jobserver.SparkWebUiActorSpec [error] (job-server/test:test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 121 s, completed Nov 24, 2014 5:49:47 PM Assembly failed Really need a help here. Thanks !
... View more
11-24-2014
02:35 AM
Thanks sowen, I can run the sbt command now ( after install sbt ).
... View more