I resorted to using an Avro schema with the RecordsetWriter in order to coerce the values into numbers.  So even though the documentation says toNumber's return type is "Number", it doesn't get expressed that way.  But the schema fixes it so its all good.