Detection of multiple QTL with epistatic effects under a mixed inheritance model in an outbred population