Hello, what is the version of the stack and Ambari are you using? You can attach the hiveserver2 log or report error displayed on Ambari? ... View more

My first try is on CentOS7 (single node). Then I try to other cluster (CentOS7 multinode) and have same error. On Azure with CentOS7 (openlogic) not have error. ... View more

Hi guys, I had the same problem with ambari 2.2.1.1 and upgrade from 2.3.4.0 to 2.4.0.0 (equal to @Sebastian Zimmermann), I solved it by going to edit the dabatase of Ambari First, I added the version 2.4.0.0 via Ambari->stack and version, I follow hortonworks guide at http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.0/bk_upgrading_Ambari/content/_upgrading_HDP_register_target_version.html Then I analyzed the tables of ambaridb on mysql:: mysql> select * from clusters; | cluster_id | resource_id | cluster_info | cluster_name | provisioning_state | security_type | desired_cluster_state | desired_stack_id | | 2 | 4 | | nb_cluster | INSTALLED | NONE | | 4 | 1 row in set (0,00 sec) mysql> select * from stack; +----------+------------+-----------------+ | stack_id | stack_name | stack_version | +----------+------------+-----------------+ | 8 | HDP | 2.0 | | 2 | HDP | 2.0.6 | | 1 | HDP | 2.0.6.GlusterFS | | 7 | HDP | 2.1 | | 3 | HDP | 2.1.GlusterFS | | 5 | HDP | 2.2 | | 4 | HDP | 2.3 | | 6 | HDP | 2.3.GlusterFS | | 9 | HDP | 2.4 | +----------+------------+-----------------+ mysql> select * from repo_version; | repo_version_id | stack_id | version | display_name | repositories | | 1 | 4 | 2.3.4.0-3485 | HDP-2.3.4.0-3485 | [{"repositories":[{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/updates/2.3.4.0","Repositories/repo_name":"HDP","Repositories/repo_id":"HDP-2.3"},{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/suse11sp3","Repositories/repo_name":"HDP-UTILS","Repositories/repo_id":"HDP-UTILS-1.1.0.20"}],"OperatingSystems/os_type":"suse11"},{"repositories":[{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.3.4.0","Repositories/repo_name":"HDP","Repositories/repo_id":"HDP-2.3"},{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos7","Repositories/repo_name":"HDP-UTILS","Repositories/repo_id":"HDP-UTILS-1.1.0.20"}],"OperatingSystems/os_type":"redhat7"},{"repositories":[{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/debian7/2.x/updates/2.3.4.0","Repositories/repo_name":"HDP","Repositories/repo_id":"HDP-2.3"},{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/debian6","Repositories/repo_name":"HDP-UTILS","Repositories/repo_id":"HDP-UTILS-1.1.0.20"}],"OperatingSystems/os_type":"debian7"},{"repositories":[{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0","Repositories/repo_name":"HDP","Repositories/repo_id":"HDP-2.3"},{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6","Repositories/repo_name":"HDP-UTILS","Repositories/repo_id":"HDP-UTILS-1.1.0.20"}],"OperatingSystems/os_type":"redhat6"},{"repositories":[{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.3.4.0","Repositories/repo_name":"HDP","Repositories/repo_id":"HDP-2.3"},{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu12","Repositories/repo_name":"HDP-UTILS","Repositories/repo_id":"HDP-UTILS-1.1.0.20"}],"OperatingSystems/os_type":"ubuntu14"},{"repositories":[{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/ubuntu12/2.x/updates/2.3.4.0","Repositories/repo_name":"HDP","Repositories/repo_id":"HDP-2.3"},{"Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu12","Repositories/repo_name":"HDP-UTILS","Repositories/repo_id":"HDP-UTILS-1.1.0.20"}],"OperatingSystems/os_type":"ubuntu12"}] | | 2 | 9 | 2.4.0.0 | HDP-2.4.0.0 | [{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/debian7/2.x/updates/2.4.0.0/","Repositories/repo_name":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/debian7","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"debian7"},{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.0.0","Repositories/repo_name":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"redhat6"},{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.0.0","Repositories/repo_name":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos7","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"redhat7"},{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/updates/2.4.0.0/","Repositories/repo_name":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/suse11sp3","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"suse11"},{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/ubuntu12/2.x/updates/2.4.0.0/","Repositories/repo_name":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu12","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"ubuntu12"},{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.4.0.0/","Repositories/repo_name":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"ubuntu14"}] | 2 rows in set (0,00 sec) mysql> select * from cluster_version; +----+-----------------+------------+---------+---------------+---------------+------------+ | id | repo_version_id | cluster_id | state | start_time | end_time | user_name | +----+-----------------+------------+---------+---------------+---------------+------------+ | 1 | 1 | 2 | CURRENT | 1452865243126 | 1452865243194 | _anonymous | +----+-----------------+------------+---------+---------------+---------------+------------+ 1 row in set (0,00 sec) So I added the following information for manually add the stack 2.4 (care to use correct value of repo_version_id and cluster_id): mysql> insert into cluster_version (id,repo_version_id,cluster_id,state,start_time,end_time,user_name) value (2,2,2,'INSTALLED',1452865243126,1459763540455,'xxxxxx'); Query OK, 1 row affected (0,10 sec) mysql> select * from cluster_version; +----+-----------------+------------+-----------+---------------+---------------+------------+ | id | repo_version_id | cluster_id | state | start_time | end_time | user_name | +----+-----------------+------------+-----------+---------------+---------------+------------+ | 1 | 1 | 2 | CURRENT | 1452865243126 | 1452865243194 | _anonymous | | 2 | 2 | 2 | INSTALLED | 1452865243126 | 1459763540455 | xxxxxx | +----+-----------------+------------+-----------+---------------+---------------+------------+ 2 rows in set (0,00 sec) I think we can put the field were in a state NOT INSTALL, but could not find the correct syntax. When I refresh page appeared to me the stack: Because the packages is not installed, i think is MUST click Reinstall Packages before click Perform Upgrade. The Install Packages took place without problems. After install packages state of host change in installed, but if try to upgrade I can't select rolling or express upgrade and can't click on proceed. Any idea for resolve the issue? ... View more

nothing. I reinstall all, but have same error (No FileSystem for scheme: hdfs) list of jar on flume windows: apacheds-i18n-2.0.0-M15.jar apacheds-kerberos-codec-2.0.0-M15.jar api-asn1-api-1.0.0-M20.jar api-util-1.0.0-M20.jar asm-3.2.jar avro-1.7.4.jar aws-java-sdk-1.7.4.jar azure-storage-2.2.0.jar commons-beanutils-1.7.0.jar commons-beanutils-core-1.8.0.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.2.jar commons-compress-1.4.1.jar commons-configuration-1.6.jar commons-digester-1.8.jar commons-httpclient-3.1.jar commons-io-2.4.jar commons-lang-2.6.jar commons-logging-1.1.3.jar commons-math3-3.1.1.jar commons-net-3.1.jar curator-client-2.7.1.jar curator-framework-2.7.1.jar curator-recipes-2.7.1.jar flume-avro-source-1.6.0.jar flume-dataset-sink-1.6.0.jar flume-file-channel-1.6.0.jar flume-hdfs-sink-1.6.0.jar flume-hive-sink-1.6.0.jar flume-irc-sink-1.6.0.jar flume-jdbc-channel-1.6.0.jar flume-jms-source-1.6.0.jar flume-kafka-channel-1.6.0.jar flume-kafka-source-1.6.0.jar flume-ng-auth-1.6.0.jar flume-ng-configuration-1.6.0.jar flume-ng-core-1.6.0.jar flume-ng-elasticsearch-sink-1.6.0.jar flume-ng-embedded-agent-1.6.0.jar flume-ng-hbase-sink-1.6.0.jar flume-ng-kafka-sink-1.6.0.jar flume-ng-log4jappender-1.6.0.jar flume-ng-morphline-solr-sink-1.6.0.jar flume-ng-node-1.6.0.jar flume-ng-sdk-1.6.0.jar flume-scribe-source-1.6.0.jar flume-spillable-memory-channel-1.6.0.jar flume-thrift-source-1.6.0.jar flume-tools-1.6.0.jar flume-twitter-source-1.6.0.jar gson-2.2.4.jar guava-11.0.2.jar hadoop-annotations-2.7.1.2.3.4.0-3485.jar hadoop-auth-2.7.1.2.3.4.0-3485.jar hadoop-aws-2.7.1.2.3.4.0-3485.jar hadoop-azure-2.7.1.2.3.4.0-3485.jar hadoop-common-2.7.1.2.3.4.0-3485-tests.jar hadoop-common-2.7.1.2.3.4.0-3485.jar hadoop-nfs-2.7.1.2.3.4.0-3485.jar hamcrest-core-1.3.jar htrace-core-3.1.0-incubating.jar httpclient-4.2.5.jar httpcore-4.2.5.jar jackson-annotations-2.2.3.jar jackson-core-2.2.3.jar jackson-core-asl-1.9.13.jar jackson-databind-2.2.3.jar jackson-jaxrs-1.9.13.jar jackson-mapper-asl-1.9.13.jar jackson-xc-1.9.13.jar java-xmlbuilder-0.4.jar jaxb-api-2.2.2.jar jaxb-impl-2.2.3-1.jar jersey-core-1.9.jar jersey-json-1.9.jar jersey-server-1.9.jar jets3t-0.9.0.jar jettison-1.1.jar jetty-6.1.26.hwx.jar jetty-util-6.1.26.hwx.jar jsch-0.1.42.jar jsp-api-2.1.jar jsr305-3.0.0.jar junit-4.11.jar log4j-1.2.17.jar microsoft-windowsazure-storage-sdk-0.6.0.jar mockito-all-1.8.5.jar netty-3.6.2.Final.jar paranamer-2.3.jar protobuf-java-2.5.0.jar ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar servlet-api-2.5.jar slf4j-api-1.7.10.jar slf4j-log4j12-1.7.10.jar snappy-java-1.0.4.1.jar stax-api-1.0-2.jar xmlenc-0.52.jar xz-1.0.jar zookeeper-3.4.6.2.3.4.0-3485.jar List of jar of cluster on azure: activation-1.1.jar apacheds-i18n-2.0.0-M15.jar apacheds-kerberos-codec-2.0.0-M15.jar api-asn1-api-1.0.0-M20.jar api-util-1.0.0-M20.jar asm-3.2.jar avro-1.7.4.jar aws-java-sdk-1.7.4.jar azure-storage-2.2.0.jar commons-beanutils-1.7.0.jar commons-beanutils-core-1.8.0.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.2.jar commons-compress-1.4.1.jar commons-configuration-1.6.jar commons-digester-1.8.jar commons-httpclient-3.1.jar commons-io-2.4.jar commons-lang-2.6.jar commons-logging-1.1.3.jar commons-math3-3.1.1.jar commons-net-3.1.jar curator-client-2.7.1.jar curator-framework-2.7.1.jar curator-recipes-2.7.1.jar gson-2.2.4.jar guava-11.0.2.jar hamcrest-core-1.3.jar htrace-core-3.1.0-incubating.jar httpclient-4.2.5.jar httpcore-4.2.5.jar jackson-annotations-2.2.3.jar jackson-core-2.2.3.jar jackson-core-asl-1.9.13.jar jackson-databind-2.2.3.jar jackson-jaxrs-1.9.13.jar jackson-mapper-asl-1.9.13.jar jackson-xc-1.9.13.jar java-xmlbuilder-0.4.jar jaxb-api-2.2.2.jar jaxb-impl-2.2.3-1.jar jersey-core-1.9.jar jersey-json-1.9.jar jersey-server-1.9.jar jets3t-0.9.0.jar jettison-1.1.jar jetty-6.1.26.hwx.jar jetty-util-6.1.26.hwx.jar jsch-0.1.42.jar jsp-api-2.1.jar jsr305-3.0.0.jar junit-4.11.jar log4j-1.2.17.jar microsoft-windowsazure-storage-sdk-0.6.0.jar mockito-all-1.8.5.jar native netty-3.6.2.Final.jar ojdbc6.jar paranamer-2.3.jar protobuf-java-2.5.0.jar ranger-hdfs-plugin-impl ranger-hdfs-plugin-shim-0.5.0.2.3.4.0-3485.jar ranger-plugin-classloader-0.5.0.2.3.4.0-3485.jar ranger-yarn-plugin-impl ranger-yarn-plugin-shim-0.5.0.2.3.4.0-3485.jar servlet-api-2.5.jar slf4j-api-1.7.10.jar slf4j-log4j12-1.7.10.jar snappy-java-1.0.4.1.jar stax-api-1.0-2.jar xmlenc-0.52.jar xz-1.0.jar zookeeper-3.4.6.2.3.4.0-3485.jar for last attempt I put on flume/lib directory all jar of hadoop cluster. No have idea what is the problem. Thanks for patience and help. ... View more

I copy jar from azure :) btw now I erase all and restart config, hope find the problem ... View more

I try and agent start correctly. Now i used cluster stored on azure. I download jar file and hdfs conf client from cluster (put on /conf dir of flume). When sink start have this error: (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)] HDFS IO error java.io.IOException: No FileSystem for scheme: hdfs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2644) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) I try to add this part on core-site.xml (resolve this error on sandbox) <property> <name>fs.file.impl</name> <value>org.apache.hadoop.fs.LocalFileSystem</value> <description>The FileSystem for file: uris.</description> </property> <property> <name>fs.hdfs.impl</name> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> <description>The FileSystem for hdfs: uris.</description> </property> But have this error: 2016-03-17 16:25:29,380 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) ... 16 more 2016-03-17 16:25:29,429 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) ... 16 more Thanks for help ... View more

Thank you, I try this solution today. I just have to start agent with java, with windows I can't started with bin\flume-ng command. ... View more

Hi, thanks for suggestion. I copy all jar missing file (I think), but when sink start I have this warning, and on hdfs write some tmp empty file. error-sink.txt ... View more

Hello everyone, I am writing to report the steps I followed. For test I'm using Windows 10 as a client machine, which runs on the sandbox with VirtualBox 2.4. I added the hosts file: 127.0.0.1 sandbox.hortonworks.com I was inspired by these guide because with binary Apache have more error and other difficulties: http://mapredit.blogspot.it/2012/07/run-flume-13x-on-windows.html - Installed jdk and maven - Set the environment variable - Compiled with maven - Unpacking the tar in c:\flume - Created the flume configuration file syslog-agent.sources = Syslog syslog-agent.channels = MemoryChannel-1 syslog-agent.sinks = Console syslog-agent.sources.Syslog.type = syslogTcp syslog-agent.sources.Syslog.port = 5140 syslog-agent.sources.Syslog.channels = MemoryChannel-1 syslog-agent.sinks.Console.channel = MemoryChannel-1 syslog-agent.sinks.Console.type = logger syslog-agent.channels.MemoryChannel-1.type = memory - Launched the client: java -Xmx20m -Dlog4j.configuration=file:///%CD%\conf\log4j.properties -cp "c:\flume\lib\*" org.apache.flume.node.Application -f c:\flume\conf\syslog-agent.conf -n syslog-agent - Try to send syslog message, and have this response: 2016-03-15 17:50:42,215 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{host=host, Severity=7, Facility=1, priority=15, timestamp=1458042030000} body: 66 6F 6F 5B 33 34 35 5D 3A 20 61 20 73 79 73 6C foo[345]: a sysl } Works (yeah), then I try write log on HDFS (my final goal). Change conf file in: syslog-agent.sources = Syslog syslog-agent.channels = MemoryChannel-1 syslog-agent.sinks = HDFS-LAB syslog-agent.sources.Syslog.type = syslogTcp syslog-agent.sources.Syslog.port = 5140 syslog-agent.sources.Syslog.channels = MemoryChannel-1 syslog-agent.sinks.HDFS-LAB.channel = MemoryChannel-1 syslog-agent.sinks.HDFS-LAB.type = hdfs syslog-agent.sinks.HDFS-LAB.hdfs.path = hdfs://sandbox.hortonworks.com:8020/apps/flume/%y-%m-%d/%H%M/%S syslog-agent.sinks.HDFS-LAB.hdfs.file.Prefix = syslogfiles- syslog-agent.sinks.HDFS-LAB.hdfs.round = true syslog-agent.sinks.HDFS-LAB.hdfs.roundValue = 10 syslog-agent.sinks.HDFS-LAB.hdfs.roundUnit = second syslog-agent.channels.MemoryChannel-1.type = memory Start agen, ERROR: 2016-03-15 18:05:50,322 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)] Failed to start agent because dependencies were not found in classpath. Error follows. java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 12 more The list of jar on c:\flume\lib: 15/03/2016 12:04 346.729 apache-log4j-extras-1.1.jar 15/03/2016 12:04 18.031 async-1.4.0.jar 15/03/2016 12:04 1.138.911 asynchbase-1.5.0.jar 15/03/2016 11:06 303.139 avro-1.7.4.jar 15/03/2016 11:06 187.840 avro-ipc-1.7.4.jar 15/03/2016 12:02 41.123 commons-cli-1.2.jar 15/03/2016 11:06 263.865 commons-codec-1.8.jar 15/03/2016 11:06 588.337 commons-collections-3.2.2.jar 15/03/2016 11:06 241.367 commons-compress-1.4.1.jar 15/03/2016 12:03 160.519 commons-dbcp-1.4.jar 15/03/2016 12:02 163.151 commons-io-2.1.jar 15/03/2016 12:10 267.634 commons-jexl-2.1.1.jar 15/03/2016 11:06 279.193 commons-lang-2.5.jar 15/03/2016 11:06 60.686 commons-logging-1.1.1.jar 15/03/2016 12:03 96.221 commons-pool-1.5.4.jar 15/03/2016 12:04 68.866 curator-client-2.6.0.jar 15/03/2016 12:04 185.245 curator-framework-2.6.0.jar 15/03/2016 12:04 248.171 curator-recipes-2.6.0.jar 15/03/2016 12:03 3.103.132 derby-10.11.1.1.jar 15/03/2016 12:12 20.623 flume-avro-source-1.7.0-SNAPSHOT.jar 15/03/2016 12:11 38.276 flume-dataset-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:03 295.717 flume-file-channel-1.7.0-SNAPSHOT.jar 15/03/2016 12:03 67.897 flume-hdfs-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 43.114 flume-hive-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:03 13.672 flume-irc-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:03 53.603 flume-jdbc-channel-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 27.199 flume-jms-source-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 21.662 flume-kafka-channel-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 15.948 flume-kafka-source-1.7.0-SNAPSHOT.jar 15/03/2016 12:02 26.369 flume-ng-auth-1.7.0-SNAPSHOT.jar 15/03/2016 12:01 56.785 flume-ng-configuration-1.7.0-SNAPSHOT.jar 15/03/2016 12:02 381.642 flume-ng-core-1.7.0-SNAPSHOT.jar 15/03/2016 12:05 37.966 flume-ng-elasticsearch-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:04 20.620 flume-ng-embedded-agent-1.7.0-SNAPSHOT.jar 15/03/2016 12:04 53.204 flume-ng-hbase-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:09 15.267 flume-ng-kafka-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 16.668 flume-ng-log4jappender-1.7.0-SNAPSHOT.jar 15/03/2016 12:08 35.263 flume-ng-morphline-solr-sink-1.7.0-SNAPSHOT.jar 15/03/2016 12:04 37.514 flume-ng-node-1.7.0-SNAPSHOT.jar 15/03/2016 12:01 120.730 flume-ng-sdk-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 44.551 flume-scribe-source-1.7.0-SNAPSHOT.jar 15/03/2016 12:03 22.533 flume-spillable-memory-channel-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 32.807 flume-taildir-source-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 56.552 flume-thrift-source-1.7.0-SNAPSHOT.jar 15/03/2016 12:13 18.917 flume-tools-1.7.0-SNAPSHOT.jar 15/03/2016 12:12 14.944 flume-twitter-source-1.7.0-SNAPSHOT.jar 15/03/2016 12:02 189.285 gson-2.2.2.jar 15/03/2016 12:01 1.648.200 guava-11.0.2.jar 15/03/2016 11:06 427.021 httpclient-4.2.1.jar 15/03/2016 11:06 181.201 httpcore-4.1.3.jar 15/03/2016 12:03 132.202 irclib-1.10.jar 15/03/2016 12:07 35.058 jackson-annotations-2.3.0.jar 15/03/2016 12:07 197.986 jackson-core-2.3.1.jar 15/03/2016 11:06 228.268 jackson-core-asl-1.9.3.jar 15/03/2016 12:07 914.311 jackson-databind-2.3.1.jar 15/03/2016 11:06 773.019 jackson-mapper-asl-1.9.3.jar 15/03/2016 11:06 539.912 jetty-6.1.26.jar 15/03/2016 11:06 177.131 jetty-util-6.1.26.jar 15/03/2016 12:02 570.478 joda-time-2.1.jar 15/03/2016 12:08 53.244 jopt-simple-3.2.jar 15/03/2016 12:01 33.015 jsr305-1.3.9.jar 15/03/2016 12:09 3.514.920 kafka_2.10-0.8.1.1.jar 15/03/2016 12:10 2.141.463 kite-data-core-1.0.0.jar 15/03/2016 12:10 2.020.522 kite-data-hbase-1.0.0.jar 15/03/2016 12:10 1.799.126 kite-data-hive-1.0.0.jar 15/03/2016 12:07 1.764.982 kite-hadoop-compatibility-1.0.0.jar 15/03/2016 11:06 347.531 libthrift-0.9.0.jar 15/03/2016 11:06 489.884 log4j-1.2.17.jar 15/03/2016 12:03 390.675 mapdb-0.9.9.jar 15/03/2016 12:04 82.123 metrics-core-2.2.0.jar 15/03/2016 12:02 644.934 mina-core-2.0.4.jar 15/03/2016 11:06 1.132.988 netty-3.5.12.Final.jar 15/03/2016 12:10 19.827 opencsv-2.3.jar 15/03/2016 11:06 29.555 paranamer-2.3.jar 15/03/2016 12:07 41.943 parquet-avro-1.4.1.jar 15/03/2016 12:07 724.377 parquet-column-1.4.1.jar 15/03/2016 12:07 11.368 parquet-common-1.4.1.jar 15/03/2016 12:07 272.946 parquet-encoding-1.4.1.jar 15/03/2016 12:07 471.470 parquet-format-2.0.0.jar 15/03/2016 12:07 10.385 parquet-generator-1.4.1.jar 15/03/2016 12:07 152.325 parquet-hadoop-1.4.1.jar 15/03/2016 12:10 2.764.448 parquet-hive-bundle-1.4.1.jar 15/03/2016 12:07 1.029.033 parquet-jackson-1.4.1.jar 15/03/2016 12:02 533.455 protobuf-java-2.5.0.jar 15/03/2016 12:09 7.137.903 scala-library-2.10.1.jar 15/03/2016 12:12 276.420 serializer-2.7.2.jar 15/03/2016 12:02 133.240 servlet-api-2.5-20110124.jar 15/03/2016 11:06 25.496 slf4j-api-1.6.1.jar 15/03/2016 11:06 9.753 slf4j-log4j12-1.6.1.jar 15/03/2016 11:06 410.710 snappy-java-1.1.0.jar 15/03/2016 12:12 284.077 twitter4j-core-3.0.3.jar 15/03/2016 12:12 27.698 twitter4j-media-support-3.0.3.jar 15/03/2016 12:12 56.307 twitter4j-stream-3.0.3.jar 15/03/2016 11:06 449.505 velocity-1.7.jar 15/03/2016 12:12 3.154.938 xalan-2.7.2.jar 15/03/2016 12:12 1.229.125 xercesImpl-2.9.1.jar 15/03/2016 12:12 194.354 xml-apis-1.3.04.jar 15/03/2016 11:06 94.672 xz-1.0.jar 15/03/2016 12:08 64.009 zkclient-0.3.jar I tried to take the jar out of the sandbox (hadoop-common-2.7.1.2.4.0.0-169.jar) and it runs nicely. with no problem, when send syslog message, and have this response: 2016-03-15 18:00:19,906 (hdfs-HDFS-LAB-call-runner-0) [ERROR - org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:385)] Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable C:\flume\bin\winutils.exe in the Hadoop binaries. Solved with this guide: https://github.com/spring-projects/spring-hadoop/wiki/Using-a-Windows-client-together-with-a-Linux-cluster I downloaded and placed under the folder c:\ flume\bin - Launched agent java -Xmx20m -Dlog4j.configuration=file:///%CD%\conf\log4j.properties -Dhadoop.home.dir=C:\flume -cp "c:\flume\lib\*" org.apache.flume.node.Application -f c:\flume\conf\syslog-agent-hdfs.conf -n syslog-agent No problem for start. c:\flume>java -Xmx20m -Dlog4j.configuration=file:///%CD%\conf\log4j.properties -Dhadoop.home.dir=C:\flume -cp "c:\flume\lib\*" org.apache.flume.node.Application -f c:\flume\conf\syslog-agent-hdfs.conf -n syslog-agent 2016-03-15 18:11:19,563 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting 2016-03-15 18:11:19,573 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:78)] Configuration provider started 2016-03-15 18:11:19,576 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:c:\flume\conf\syslog-agent-hdfs.conf for changes 2016-03-15 18:11:19,595 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:c:\flume\conf\syslog-agent-hdfs.conf 2016-03-15 18:11:19,610 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:HDFS-LAB 2016-03-15 18:11:19,618 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1021)] Created context for HDFS-LAB: hdfs.round 2016-03-15 18:11:19,625 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:HDFS-LAB 2016-03-15 18:11:19,631 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:HDFS-LAB 2016-03-15 18:11:19,636 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:HDFS-LAB 2016-03-15 18:11:19,644 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:HDFS-LAB 2016-03-15 18:11:19,652 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:931)] Added sinks: HDFS-LAB Agent: syslog-agent 2016-03-15 18:11:19,659 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:HDFS-LAB 2016-03-15 18:11:19,664 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:HDFS-LAB 2016-03-15 18:11:19,673 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:314)] Starting validation of configuration for agent: syslog-agent, initial-configuration: AgentConfiguration[syslog-agent] SOURCES: {Syslog={ parameters:{channels=MemoryChannel-1, port=5140, type=syslogTcp} }} CHANNELS: {MemoryChannel-1={ parameters:{type=memory} }} SINKS: {HDFS-LAB={ parameters:{hdfs.path=hdfs://sandbox.hortonworks.com:8020/apps/flume/%y-%m-%d/%H%M/%S, hdfs.file.Prefix=syslogfiles-, hdfs.round=true, channel=MemoryChannel-1, type=hdfs, hdfs.roundValue=10, hdfs.roundUnit=second} }} 2016-03-15 18:11:19,694 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:469)] Created channel MemoryChannel-1 2016-03-15 18:11:19,704 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:675)] Creating sink: HDFS-LAB using HDFS 2016-03-15 18:11:19,716 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:372)] Post validation configuration for syslog-agent AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[syslog-agent] SOURCES: {Syslog={ parameters:{channels=MemoryChannel-1, port=5140, type=syslogTcp} }} CHANNELS: {MemoryChannel-1={ parameters:{type=memory} }} SINKS: {HDFS-LAB={ parameters:{hdfs.path=hdfs://sandbox.hortonworks.com:8020/apps/flume/%y-%m-%d/%H%M/%S, hdfs.file.Prefix=syslogfiles-, hdfs.round=true, channel=MemoryChannel-1, type=hdfs, hdfs.roundValue=10, hdfs.roundUnit=second} }} 2016-03-15 18:11:19,735 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:136)] Channels:MemoryChannel-1 2016-03-15 18:11:19,743 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:137)] Sinks HDFS-LAB 2016-03-15 18:11:19,747 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:138)] Sources Syslog 2016-03-15 18:11:19,752 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:141)] Post-validation flume configuration contains configuration for agents: [syslog-agent] 2016-03-15 18:11:19,759 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:145)] Creating channels 2016-03-15 18:11:19,768 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel MemoryChannel-1 type memory 2016-03-15 18:11:19,779 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)] Created channel MemoryChannel-1 2016-03-15 18:11:19,786 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source Syslog, type syslogTcp 2016-03-15 18:11:19,802 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: HDFS-LAB, type: hdfs 2016-03-15 18:11:19,815 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel MemoryChannel-1 connected to [Syslog, HDFS-LAB] 2016-03-15 18:11:19,828 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{Syslog=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogTcpSource{name:Syslog,state:IDLE} }} sinkRunners:{HDFS-LAB=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3f476562 counterGroup:{ name:null counters:{} } }} channels:{MemoryChannel-1=org.apache.flume.channel.MemoryChannel{name: MemoryChannel-1}} } 2016-03-15 18:11:19,855 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel MemoryChannel-1 2016-03-15 18:11:19,908 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: CHANNEL, name: MemoryChannel-1: Successfully registered new MBean. 2016-03-15 18:11:19,918 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: CHANNEL, name: MemoryChannel-1 started 2016-03-15 18:11:19,927 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink HDFS-LAB 2016-03-15 18:11:19,935 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source Syslog 2016-03-15 18:11:19,938 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SINK, name: HDFS-LAB: Successfully registered new MBean. 2016-03-15 18:11:19,958 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SINK, name: HDFS-LAB started 2016-03-15 18:11:19,971 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting 2016-03-15 18:11:20,054 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.SyslogTcpSource.start(SyslogTcpSource.java:123)] Syslog TCP Source starting... 2016-03-15 18:11:49,947 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:c:\flume\conf\syslog-agent-hdfs.conf for changes When try to send syslog message, and have this response: 2016-03-15 18:18:33,672 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:234)] Creating hdfs://sandbox.hortonworks.com:8020/apps/flume/16-03-15/1240/30/FlumeData.1458062313441.tmp 2016-03-15 18:18:33,715 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36) at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:120) at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:236) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2812) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 18 more Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36) at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:120) at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:236) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2812) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 18 more At this point I think is missing some jar, you have idea what it is? Thank you ... View more