I've been experimenting with the options for copying data from our (bare metal) cluster to S3. I found that something like this works: hive> create table aws.my_table > ( > `column1` string, > `column2` string, .... > `columnX` string) > row format delimited fields terminated by ',' > lines terminated by '\n' > stored as textfile > location 's3n://my_bucket/my_folder_path/'; hive> insert into table aws.my_table select * from source_db.source_table; But only if the source data set is pretty small. For a larger data set (10's of GB), it fails with errors like Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: n must be positive at ... I understand that pushing gigabytes (and eventually terabytes) of data to a remote server is going to be somewhat painful. So, I'm wondering what kind of customizations are available. Is there a way to specify compression or upload throttling, etc? Can anyone give me instruction on getting around the errors? ... View more

I want to copy some data from Hive tables on our (bare metal) cluster to a S3. I know that I can export the data out of HDFS to a CSV file and upload that to S3, but I'm guessing that there are better ways to accomplish this. Any ideas? ... View more

We installed Zeppelin along with our upgrade to 2.4.2 In Ambari, everything looks good, but when I try to actually interact with it, nothing seems to happen. When I look at the log file on the server, it shows the error below. Could this just be a permissions issue? INFO [2016-05-24 09:15:00,707] ({qtp895821303-5595} NotebookServer.java[onClose]:213) - Closed connection to 10.1.165.24 : 63750. (1001) null WARN [2016-05-24 09:15:01,104] ({qtp895821303-5592} SecurityRestApi.java[ticket]:79) - {"status":"OK","message":"","body":{"principal":"anonymous","ticket":"anonymous","roles":"[]"}} INFO [2016-05-24 09:15:01,154] ({qtp895821303-5592} NotebookServer.java[onOpen]:92) - New connection from 10.1.165.24 : 63761 ERROR [2016-05-24 09:15:13,897] ({qtp895821303-5593} NotebookServer.java[onMessage]:207) - Can't handle message org.apache.commons.vfs2.FileSystemException: Could not create folder "file:///usr/hdp/current/zeppelin-server/lib/notebook/2BNKRY7B6". at org.apache.commons.vfs2.provider.AbstractFileObject.createFolder(AbstractFileObject.java:999) at org.apache.zeppelin.notebook.repo.VFSNotebookRepo.save(VFSNotebookRepo.java:218) at org.apache.zeppelin.notebook.repo.NotebookRepoSync.save(NotebookRepoSync.java:144) at org.apache.zeppelin.notebook.Note.persist(Note.java:439) at org.apache.zeppelin.notebook.Notebook.createNote(Notebook.java:159) at org.apache.zeppelin.notebook.Notebook.createNote(Notebook.java:133) at org.apache.zeppelin.socket.NotebookServer.createNote(NotebookServer.java:497) at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:147) at org.apache.zeppelin.socket.NotebookSocket.onMessage(NotebookSocket.java:56) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameHandler.onFrame(WebSocketConnectionRFC6455.java:835) at org.eclipse.jetty.websocket.WebSocketParserRFC6455.parseNext(WebSocketParserRFC6455.java:349) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.handle(WebSocketConnectionRFC6455.java:225) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.commons.vfs2.FileSystemException: Could not create directory "/usr/hdp/current/zeppelin-server/lib/notebook/2BNKRY7B6". at org.apache.commons.vfs2.provider.local.LocalFile.doCreateFolder(LocalFile.java:153) at org.apache.commons.vfs2.provider.AbstractFileObject.createFolder(AbstractFileObject.java:988) ... View more