As long as the networks are routable, this should be functional to not co-locate HDFS and HBase (ZooKeeper gets scary, consider all of the following to apply only to HDFS and HBase).
I believe writes will only be slower with respect to the underlying network connection. Each write will still be bound by the sync of the slowest of the three datanodes hosting the replicas for the block, so, I don't think you'll pay a much larger penalty here.
Reads, however, will likely be noticeably slower. HBase largely expects to take advantage of a feature in HDFS known as "Short Circuit Reads". This features takes advantage of the local resources that the Datanode and the RegionServer share with a shared memory segment and a Unix domain socket. This avoids a TCP socket when the local RegionServer can read data from the local DataNode. I'm sure there are performance numbers out there floating around (I don't recall specifics), but this is a noticeable performance gain when short-circuit reads can be used (and is why many metrics often consider RegionServer locality to the Regions' data it is hosting).
