Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Errors while running Kite CDK examples

Errors while running Kite CDK examples

Explorer

H

 

I am running the kite sdk examples on quickstart vm 5.1.  Building the maven package was successful for logging directory but when trying to create dataset directory errors are displayed. Here is detailed information of logs. can anyone help me how to resolve this issue.

 

------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.2:java (default-cli) on project logging: An exception occured while executing the Java class. null: InvocationTargetException: Unable to create metadata directory:hdfs://quickstart.cloudera:8020/tmp/data/events/.metadata for dataset:events: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "quickstart.cloudera/127.0.0.1"; destination host is: "quickstart.cloudera":8020; -> [Help 1]

 

one more question, is it mandatory to run the examples on 4.4 version of VM? If so, where can  I download it from. I also see later versions.

 

If it runs on 4.4 alone, why come it is not compatable with newer versions?

 

Thanks,

Asmath.

11 REPLIES 11

Re: Errors while running Kite CDK examples

Contributor

Hi Azzu!

 

The examples in Kite 0.16 and earlier are only compatible with CDH4. You can use any version of CDH that is 4.4 or later.

 

For our next release, 0.17, we're switching to CDH5 for the examples. If you want to start working with them now you can use the snapshot version of the examples:

 

https://github.com/kite-sdk/kite-examples/tree/snapshot

 

I'm still in the process of fixing up some of the examples so you may have better luck with my current working branch:

 

https://github.com/joey/kite-examples/tree/cdk-656-cdh5-qs

 

In either case, you need to be sure to clone the Kite repository and build a snapshot version of it as the examples snapshot relies on the snapshot version of Kite.

 

Let me know if you need some more help.

 

-Joey

Re: Errors while running Kite CDK examples

Explorer

Thanks Joey for your quick response. I will try to run the examples now in CDH4.

Re: Errors while running Kite CDK examples

Explorer

Hi,

 

I am getting error while running loggiing application. Looks like the data is not getting added to HDFS due to exception with flume's security account.

 

04 Oct 2014 20:35:21,427 ERROR [hdfs-sink-1-call-runner-6] (org.apache.hadoop.security.UserGroupInformation.doAs:1441)  - PriviledgedActionException as:cloudera (auth:PROXY) via flume (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: flume is not allowed to impersonate cloudera
04 Oct 2014 20:35:21,427 WARN  [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:438)  - HDFS IO error
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: flume is not allowed to impersonate cloudera
    at org.apache.hadoop.ipc.Client.call(Client.java:1238)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at $Proxy13.create(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at $Proxy13.create(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:193)
    at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1376)
    at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1395)
    at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1255)
    at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1212)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:276)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:265)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:888)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:869)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:768)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:757)
    at org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:80)
    at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:228)
    at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:221)
    at org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:589)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
    at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:158)
    at org.apache.flume.sink.hdfs.BucketWriter.access$800(BucketWriter.java:57)
    at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:586)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

 

I followed all the steps mentioned for executing the logging application. Added default flume agent and added entry in core-site.xml to allow all users and groups related to flume. Am I missing anything else to run this example?

 

Thanks,

Azzu

Re: Errors while running Kite CDK examples

Contributor
The error looks like it's not picking up the core-site changes.

Can you send me what you added to core-site.xml? Make sure that the
changes are made on the configuration the NN is using, typically
/etc/hadoop/conf/core-site.xml on the NN server or in Cloudera Manager
if you're using CM. Also, make sure the NN was restarted after the
configuration was updated.

-Joey

Re: Errors while running Kite CDK examples

Explorer

Hi Joyce,

 

I tried all the possible ways to resolve this issue but finally gave up. Here are the few things I want to bring to your notice.

 

1. Cloudera user is not having access to modify core-site.xml so I changed the persmissions by loggig into root account and executed below command

chmod 777 /etc/hadoop/conf/core-site.xml 

 

Updated file after changing permissions.

 

<!--Autogenerated by Cloudera CM on 2014-06-01T23:18:54.093Z--><configuration><property><name>fs.defaultFS</name><value>hdfs://localhost.localdomain:8020</value></property><property><name>fs.trash.interval</name><value>1</value></property><property><name>io.file.buffer.size</name><value>65536</value></property><property><name>io.compression.codecs</name><value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value></property><property><name>hadoop.security.authentication</name><value>simple</value></property><property><name>hadoop.rpc.protection</name><value>authentication</value></property><property><name>hadoop.security.auth_to_local</name><value>DEFAULT</value></property><property><name>hadoop.proxyuser.flume.groups</name><value>*</value></property><property><name>hadoop.proxyuser.flume.hosts</name><value>*</value></property></configuration>

 

2. Not able to change the system time with the command sudo ntpdate pool.ntp.org . Temporarily changed it by disabling the optons.

 

3. By Default etc/hosts file has entry as 127.0.0.1 localhost.localdomain localhost . Do I need to change it to 127.0.0.1 localhost localhost.localdomain

 

4. mvn package of logging examples was successful. /tmp/data/events is successfully created and APP java is priniting the log statements on console but cannot add them to HDFS.

 

Verified the log file for more information. Here it is.

 

2014-10-05 13:46:06,602 WARN org.apache.flume.sink.hdfs.HDFSEventSink: HDFS IO error
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: flume is not allowed to impersonate cloudera
    at org.apache.hadoop.ipc.Client.call(Client.java:1238)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)

 

Note: I restarted cluster,flume and then vm machine after doing changes still no luck

 

Few more questions?

 

Is VM time and PC time is causing the issue as time is different?

Is it because of cloudera not having edit access to folders even though it is sudo user?

 

 

Thanks,

Azzu

Highlighted

Re: Errors while running Kite CDK examples

Explorer

Hi,

 

Can anyone help me with this issue.

 

Thanks,

Azzu

Re: Errors while running Kite CDK examples

Explorer

Hi,

 

Can anyone help me with this issue.

 

Thanks,

Azzu

Re: Errors while running Kite CDK examples

Contributor

Sorry for the delay. Since you're running the examples on CDH4, you need to make sure that you make the proxy configuration change in the CM UI.

 

Which version of the CDH4 VM are you using?

 

-Joey

Re: Errors while running Kite CDK examples

Explorer

Hi,

 

I am using the below version.

 

[cloudera@localhost target]$ hadoop version
Hadoop 2.0.0-cdh4.7.0
Subversion git://rhel64-6-0-mk3.jenkins.cloudera.com.121.29.172.in-addr.arpa/data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.7.0/src/hadoop-common-project/hadoop-common -r 8e266e052e423af592871e2dfe09d54c03f6a0e8
Compiled by jenkins on Wed May 28 10:11:59 PDT 2014
From source with checksum f60207d0daa9f943f253cc8932d598c8
This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar

 

Thanks,

Azzu


@joey wrote:

Sorry for the delay. Since you're running the examples on CDH4, you need to make sure that you make the proxy configuration change in the CM UI.

 

Which version of the CDH4 VM are you using?

 

-Joey



@joey wrote:

Sorry for the delay. Since you're running the examples on CDH4, you need to make sure that you make the proxy configuration change in the CM UI.

 

Which version of the CDH4 VM are you using?

 

-Joey




Don't have an account?
Coming from Hortonworks? Activate your account here