Reply
Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: How does speed layer updates model?

I don't think the problem was ever quite clear. Can you reply with
answers to some of the same questions above?
My guess is you'll find the model is not being built and there is an
error in your batch layer logs.

I can probably add some more logging here and there about how many
messages are being received to maybe show more progress in the logs.
Explorer
Posts: 18
Registered: ‎01-06-2016

Re: How does speed layer updates model?

Hi thanks for the quick response.

 

To answer some of your questions above: I do have a speed layer running. I can see activity on the input topic immediately after I ingest through the serving layer, and I also see activity on the update topic roughly 5 mins later. 

 

I am inclined to believe that the batch layer is successfully building the model, as I don't see any error in the output logs and the files are available on HDFS when processing completes.

 

Unfortunately, I don't see any activity on the driver UI on port 4041. No jobs seem to be added. 

I did discover the following errors today which maybe you could shed some light on.

When I start the serving layer, I see the following message in the Kafka logs:

 

[2016-01-07 16:00:38,391] ERROR Closing socket for /127.0.0.1 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)


I also see the following on my hadoop cluster jobs. The SpeedLayer job seems to hang in the ACCEPTED state.

 

application_1452182244662_0002dylanaOryxSpeedLayer-ALSExampleSPARKdefaultThu, 07 Jan 2016 15:59:13 GMTN/AACCEPTEDUNDEFINED UNASSIGNED
application_1452182244662_0001dylanaOryxBatchLayer-ALSExampleSPARKdefaultThu, 07 Jan 2016 15:58:13 GMTN/ARUNNINGUNDEFINED ApplicationMaster

 

 

 

 

Explorer
Posts: 18
Registered: ‎01-06-2016

Re: How does speed layer updates model?

Also, I did see an error in the SpeedLayer Yarn jobs similar to the one below:

 

INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
ERROR yarn.ApplicationMaster: Failed to connect to driver at xxxx   

 

Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: How does speed layer updates model?

Yeah, what this tells me is that the speed layer app never actually
started on YARN. Is it possible there aren't enough resources to run
it?
Explorer
Posts: 18
Registered: ‎01-06-2016

Re: How does speed layer updates model?

Yea I think that was part of the issue. I've moved to running on a new machine with more resources, and I can now see the results being offered by the serving layer.

 

However, I still don't see any action on the speed layer. When I ingest some data it  always takes 5 mins before any action is taken (which is on the batch layer). I also still don't see any jobs at 4041, and the job is still marked as "ACCEPTED" on the cluster. There is definitely still about 2GB RAM still available on the machine so I think it must be something else. 

Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: How does speed layer updates model?

If you see results at the Serving Layer, then the update topic sounds
like it's being populated correctly, and therefore the input topic as
well. If you see no jobs then somehow the executors aren't running. I
doubt it's a config problem since you'd then see different errors like
no topic found.

If the executors are definitely running, note you won't see the speed
layer do anything unless there's input. Are you sending input *after*
it's running? it also won't do anything until it has a model. So you
want to test by seeing what happens after the model is published.

You can look at the logs to see what it thinks is happening Some of
those may be useful here.
Explorer
Posts: 18
Registered: ‎01-06-2016

Re: How does speed layer updates model?

I recently just came back to this. I believe the issue is with a lack of resources available. I can run the batch layer and get results through the serving layer. If I shut then shut down the batch layer and start the speed layer, I can feed more data and get further results through the serving layer. Ideally I will be moving to a machine with more resources soon, so I am happy that I can get Oryx running for now. Thanks for help. 

I have another question regarding a custom application. I am looking at the example app on Github (https://github.com/OryxProject/oryx/tree/master/app/example). I can't seem to get it to compile. Has the recent update (2.1.2) broken this example? 

 

error:

The type ExampleSpeedModelManager must implement the inherited abstract method SpeedModelManager<String,String,String>.close()

Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: How does speed layer updates model?

No, the example is compiled along with everything else and has always passed ...
https://travis-ci.org/OryxProject/oryx/branches

What are you seeing?
Explorer
Posts: 18
Registered: ‎01-06-2016

Re: How does speed layer updates model?

Apologies, stupid error on my side. Compling fine now. 

Explorer
Posts: 18
Registered: ‎01-06-2016

Re: How does speed layer updates model?

srowen, I'm looking into using/customising the ALS recommendation example. Is there a link that describes  the system (inputs/outputs, data flow etc) or could you give a brief explanation of the system here? Thanks

Announcements