Yeah, you can't do that. The results are a function of all the data. I'd argue that's a good thing, even in your case. You can learn about items and users from interactions with items in any category. You would have to otherwise build separate models, yes.