Hi Team, We are trying to access Azure BLOB storage from hdfs and somehow we are unable to do this at the moment . We have secured environment with proxies, so any outgoing traffic passes through the proxy, i already whitelisted the blob URL and i can access and upload files into BLOB storage from local linux system on the same machine where hadoop is installed. However when i try to access to azure BLOB storage with hdfs command, it just stucks and does not give any error. Following is the command and the output: hdfs dfs -ls wasbs://xxxx@xxxxxxxx.blob.core.windows.net/ it get stuck after these steps : 16/12/05 15:45:57 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 16/12/05 15:45:57 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s). 16/12/05 15:45:57 INFO impl.MetricsSystemImpl: azure-file-system metrics system started Even enabling debug: export HADOOP_ROOT_LOGGER=DEBUG,console 18/10/29 10:49:47 DEBUG util.Shell: setsid exited with exit code 0 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: i :: Ignore failures during copy ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: p [ARG] :: preserve status (rbugpcaxt)(replication, block-size, user, group, permission, checksum-type, ACL, XATTR, timestamps). If -p is specified with no <arg>, then preserves replication, block size, user, group, permission, checksum type and timestamps. raw.* xattrs are preserved when both the source and destination paths are in the /.reserved/raw hierarchy (HDFS only). raw.* xattrpreservation is independent of the -p flag. Refer to the DistCp documentation for more details. ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: update :: Update target, copying only missingfiles or directories ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: delete :: Delete from target, files missing in source ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: mapredSslConf [ARG] :: Configuration for ssl config file, to use with hftps://. Must be in the classpath. ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: numListstatusThreads [ARG] :: Number of threads to use for building file listing (max 40). ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: m [ARG] :: Max number of concurrent maps to use for copy ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: f [ARG] :: List of files that need to be copied ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: atomic :: Commit all changes or none ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: tmp [ARG] :: Intermediate work path to be used for atomic commit ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: log [ARG] :: Folder on DFS where distcp execution logs are saved ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: v :: Log additional info (path, size) in the SKIP/COPY log ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: strategy [ARG] :: Copy strategy to use. Default is dividing work based on file sizes ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: skipcrccheck :: Whether to skip CRC checks between source and target paths. ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: overwrite :: Choose to overwrite target files unconditionally, even if they exist. ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: append :: Reuse existing data in target files and append new data to them if possible ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: diff [ARG...] :: Use snapshot diff report to identify the difference between source and target ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: async :: Should distcp execution be blocking ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: filelimit [ARG] :: (Deprecated!) Limit number of files copied to <= n ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: sizelimit [ARG] :: (Deprecated!) Limit number of files copied to <= n bytes ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: bandwidth [ARG] :: Specify bandwidth per map in MB ] 18/10/29 10:49:47 DEBUG tools.OptionsParser: Adding option [ option: filters [ARG] :: The path to a file containing a list of strings for paths to be excluded from the copy. ] 18/10/29 10:49:47 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true 18/10/29 10:49:47 DEBUG security.Groups: Creating new Groups object 18/10/29 10:49:47 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 18/10/29 10:49:47 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library 18/10/29 10:49:47 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution 18/10/29 10:49:47 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping 18/10/29 10:49:48 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000 18/10/29 10:49:48 DEBUG security.UserGroupInformation: hadoop login 18/10/29 10:49:48 DEBUG security.UserGroupInformation: hadoop login commit 18/10/29 10:49:48 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: svc_hdfs 18/10/29 10:49:48 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: svc_hdfs" with name svc_hdfs 18/10/29 10:49:48 DEBUG security.UserGroupInformation: User entry: "svc_hdfs" 18/10/29 10:49:48 DEBUG security.UserGroupInformation: Assuming keytab is managed externally since logged in from subject. 18/10/29 10:49:48 DEBUG security.UserGroupInformation: UGI loginUser:svc_hdfs (auth:SIMPLE) 18/10/29 10:49:48 DEBUG gcs.GoogleHadoopFileSystemBase: GHFS version: 1.8.1.2.6.5.0-292 18/10/29 10:49:48 DEBUG configuration.ConfigurationUtils: ConfigurationUtils.locate(): base is null, name is hadoop-metrics2-azure-file-system.properties 18/10/29 10:49:48 DEBUG configuration.ConfigurationUtils: ConfigurationUtils.locate(): base is null, name is hadoop-metrics2.properties 18/10/29 10:49:48 DEBUG configuration.ConfigurationUtils: Loading configuration from the context classpath (hadoop-metrics2.properties) 18/10/29 10:49:48 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 18/10/29 10:49:48 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 18/10/29 10:49:48 INFO impl.MetricsSystemImpl: azure-file-system metrics system started 18/10/29 10:49:48 DEBUG azure.AzureNativeFileSystemStore: AzureNativeFileSystemStore init. Settings=8,false,90,{3000,3000,30000,30},{true,1.0,1.0} 18/10/29 10:49:48 DEBUG azure.AzureNativeFileSystemStore: Page blob directories: 18/10/29 10:49:48 DEBUG azure.AzureNativeFileSystemStore: Block blobs with compaction directories: 18/10/29 10:49:48 DEBUG azure.AzureNativeFileSystemStore: Atomic rename directories: /hbase 18/10/29 10:49:48 DEBUG azure.NativeAzureFileSystem: NativeAzureFileSystem. Initializing. 18/10/29 10:49:48 DEBUG azure.NativeAzureFileSystem: blockSize = 536870912 18/10/29 10:49:48 DEBUG azure.NativeAzureFileSystem: Getting the file status for wasbs:// /user 18/10/29 10:49:48 DEBUG azure.AzureNativeFileSystemStore: Retrieving metadata for user 18/10/29 10:49:48 DEBUG azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest: threadId=1, requestType=read , isFirstRequest=true, sleepDuration=0 Can anyone please help on this .Let me know if any additional configuration is required . Regards, Vishal ... View more

Hi , Can somebody help me on configuring ambari integration with multiple AD . example ; AD1 abc.hadoop.com AD2 abcd.hadoop.com AD1 abc.hadoop.com trust AD2 abcd.hadoop.com not vice versa . Regards, Vishal ... View more

Is integrating ADFS with Knoxsso for saml authentication possible ? Can somebody please share the steps. ... View more

Hi Team, We have to enable SMAl authentication for one of our client . Can somebody please guide us on this . Regards, Vishal ... View more

Hi Team, We have configured HDF 3.1.1 single node instance for POC . He we are trying to perform a small test : We have written a small python program to generate kafka message : from kafka import KafkaProducer from kafka.errors import KafkaError producer = KafkaProducer(bootstrap_servers=' hostname:6667') topic = "jsontest" producer.send(topic, b'test message') We want to consume this message thorough Nifi Consumekafka ,which is not working . nifi.app.log (type=HEARTBEAT, length=2086 bytes) from hdfdemo.c.corecompetetraining.internal:9090 in 1 millis 2018-05-15 07:51:19,808 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-05-15 07:51:19,804 and sent to hdfdemo.c.corecompetetraining.internal:9088 at 2018-05-15 07:51:19,808; send took 4 millis 2018-05-15 07:51:20,352 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:21,505 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:22,357 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:23,560 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:24,557 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 8392 nanos 2018-05-15 07:51:24,668 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:24,811 INFO [Process Cluster Protocol Request-8] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 8aec83a3-cfbc-43c2-918d-143d701d13a8 (type=HEARTBEAT, length=2086 bytes) from hdfdemo.c.corecompetetraining.internal:9090 in 1 millis 2018-05-15 07:51:24,815 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-05-15 07:51:24,809 and sent to hdfdemo.c.corecompetetraining.internal:9088 at 2018-05-15 07:51:24,815; send took 6 millis 2018-05-15 07:51:25,582 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:26,735 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:27,888 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:28,941 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:29,558 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 7009 nanos 2018-05-15 07:51:29,817 INFO [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Finished processing request fcae4257-6a34-4691-9427-530ed061faf1 (type=HEARTBEAT, length=2086 bytes) from hdfdemo.c.corecompetetraining.internal:9090 in 1 millis 2018-05-15 07:51:29,819 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-05-15 07:51:29,815 and sent to hdfdemo.c.corecompetetraining.internal:9088 at 2018-05-15 07:51:29,819; send took 3 millis 2018-05-15 07:51:29,944 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:30,846 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:31,999 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available nifi-user.log 2018-05-15 08:06:11,913 INFO [NiFi Web Server-387] org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET http://hostname:9090/nifi-api/flow/status (source ip: ) Please help on this . Regards, Vishal ... View more

REATE TABLE IF NOT EXISTS hbase_hive_table(key string, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:json") TBLPROPERTIES ("hbase.table.name" = "hbase_hive_table"); Getting log thread is interrupted, since query is done! Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'Error' ':' 'Error' (state=42000,code=40000) org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'Error' ':' 'Error' at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:277) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:263) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:303) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:244) at org.apache.hive.beeline.Commands.execute(Commands.java:871) at org.apache.hive.beeline.Commands.sql(Commands.java:729) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1000) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:835) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:793) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:493) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:476) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'Error' ':' 'Error' at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:324) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:148) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:228) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:264) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:479) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:466) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:509) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1377) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:0 cannot recognize input near 'Error' ':' 'Error' at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:214) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:171) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1224) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1218) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146) ... 15 more ... View more

CREATE TABLE IF NOT EXISTS hbase_hive_table(key string, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:json") TBLPROPERTIES ("hbase.table.name" = "hbase_hive_table"); Error: Getting log thread is interrupted, since query is done! Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'Error' ':' 'Error' (state=42000,code=40000) org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'Error' ':' 'Error' at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:277) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:263) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:303) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:244) at org.apache.hive.beeline.Commands.execute(Commands.java:871) at org.apache.hive.beeline.Commands.sql(Commands.java:729) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1000) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:835) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:793) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:493) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:476) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'Error' ':' 'Error' at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:324) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:148) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:228) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:264) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:479) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:466) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:509) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1377) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:0 cannot recognize input near 'Error' ':' 'Error' at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:214) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:171) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1224) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1218) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146) ... 15 more ... View more

Hi  ,   I have installed cloudera CDH4.7.1 A 3 cluster.I am trying to load data from hive to oracle database using Oracle loader for hadoop.I have integrated hadoop with Oracle Data Integrater ODI .For trigering OLH from ODI we have knowledge module  File-Hive to Oracle ( OLH-OSCH ).   I am getting following error: 5/01/08 00:20:47 INFO loader.OraLoader: Oracle Loader for Hadoop Release 3.0.0 - Production   Copyright (c) 2011, 2014, Oracle and/or its affiliates. All rights reserved.     15/01/08 00:20:47 INFO loader.OraLoader: Built-Against: hadoop-2.2.0-cdh5.0.0-beta-2 hive-0.12.0-cdh5.0.0-beta-2 avro-1.7.3 jackson-1.8.8 15/01/08 00:20:48 INFO utils.OraLoaderConf: property "oracle.hadoop.loader.loaderMapFile" is deprecated; use "oracle.hadoop.loader.loaderMap.*" instead 15/01/08 00:21:03 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: I_SAMPLE_TRG is not partitioned 15/01/08 00:21:03 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided 15/01/08 00:21:03 INFO loader.OraLoader: Reduce tasks set to 0 because of no partitioning or sorting. Loading will be done in the map phase. 15/01/08 00:21:04 INFO output.DBOutputFormat: Setting map tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat 15/01/08 00:21:08 WARN loader.OraLoader: Sampler error: the number of reduce tasks must be greater than one; the configured value is 0 . Job will continue without sampled information. 15/01/08 00:21:08 INFO loader.OraLoader: Sampling time=0D:0h:0m:0s:79ms (79 ms) 15/01/08 00:21:08 INFO loader.OraLoader: Submitting OraLoader job OraLoader 15/01/08 00:21:15 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 15/01/08 00:21:16 WARN conf.HiveConf: hive-site.xml not found on CLASSPATH 15/01/08 00:21:18 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. 15/01/08 00:21:18 INFO hive.metastore: Trying to connect to metastore with URI thrift://172.16.12.217:10000 15/01/08 00:21:18 INFO hive.metastore: Connected to metastore. 15/01/08 00:21:38 INFO mapred.JobClient: Cleaning up the staging area hdfs://n1.bias.com:8020/user/oracle/.staging/job_201501072220_0005 15/01/08 00:21:38 ERROR security.UserGroupInformation: PriviledgedActionException as:oracle (auth:SIMPLE) cause:java.io.IOException: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out java.io.IOException: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at oracle.hadoop.loader.lib.input.HiveToAvroInputFormat.getSplits(HiveToAvroInputFormat.java:451) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1107) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1124) at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:178) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1023) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976) at org.apache.hadoop.mapreduce.Job.submit(Job.java:582) at oracle.hadoop.loader.OraLoader.monitorAndPrintJob(OraLoader.java:652) at oracle.hadoop.loader.OraLoader.run(OraLoader.java:580) at oracle.hadoop.loader.OraLoader.run(OraLoader.java:412) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at oracle.hadoop.loader.OraLoader.main(OraLoader.java:940) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:879) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:865) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:788) at oracle.hadoop.loader.lib.input.HiveToAvroInputFormat.getSplits(HiveToAvroInputFormat.java:259) ... 21 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)   please reply as I have client demo 14 jan 2015. Thanks&Regards Vishal   ... View more

Hi All,   I am running a 4 node cluster CDH4 latest version managed and installed by cloudera manager trail version.I want to configure flume to do some twitter analysis or some system log analysis , inorder to see how the data is getting loaded into HDFS.I don't want to use mapreduce concept .Please guide me to perform this operation.   Thanks&Regards Vishal singh Email id :vishalsingh6193@gmail.com ... View more