There is currently no way for a newly created directory in HDFS to set its group from the primary group of the creating user automatically. Instead, it always follows the rule quoted in the question: the group is the group of the parent directory.
One way I've handled this in the past is first to create an intermediate directory and then explicitly change its group to the user's primary group, using chmod on the shell or setOwner in the Java APIs. Then, additional files and directories created by the process would use this as the destination directory. For example, a MapReduce job could specify its output directory under this intermediate directory, and then the output files created by that MapReduce job would have the desired group.