### Data

The dataset used in this paper was simulated for the QTLMAS 2010 workshop (Poznań, Poland). A pedigree consisting of 3226 individuals in 5 generations (*F*_{0} - *F*_{4}) was simulated, where *F*_{0} contains 5 males and 15 females. Each female was mated once and gave birth to about 30 progeny. Two traits were simulated, where one is quantitative (QT), and the other is binary (BT). Young individuals in *F*_{4} (individuals 2327 to 3226) had no phenotypic records. The genome was assumed to be about 5 × 10^{8} bp long, consisting of 5 chromosomes, each of which contained about 1 × 10^{8} bp. Each individual was genotyped for 10031 biallelic SNPs in the genome.

### Models

DHGLM provides a unified analysis for both QTL mapping and genomic breeding value estimation. Similar to BayesA, the data are modeled on two levels, i.e. both the phenotypic mean and the variance are modeled with random effects. For a quantitative trait, the phenotype **y** (*n* × 1 vector) is postulated as a random effect model

**y** = **X** β + **Zg** + **e** (1)

where **g** ~ N(**0**, diag(**λ**)) are the SNP effects, **λ** = (λ_{1}, λ_{2},..., λ_{m})′ are the variances of the SNP effects, and the residuals **e** ~ *N*(0, *σ*^{2}**I**). The fixed effects β included an intercept and the sex effect in our application to reduce the residual errors. The SNP variances **λ** are modeled as

log **λ** = **1** *a* + **b** (2)

with an intercept *a* and normally distributed random effects **b**. The *genomic estimated breeding value* (GEBV) for individual *i* is computed as
. QTL can be scanned using the marker-specific variances **λ**. For a binary trait, the mean of **y**, is modeled by the same linear predictor **X** β + **Zg** through a logit link function.

For the marker-specific variances, the correlated random effects, **b**, follow a multivariate normal distribution with a mean of zero and a variance-covariance matrix
, where *m* is the number of SNPs and *k*, *l* are the SNP indices. When *ρ* = 1, all the SNPs have a constant variance (GLMM); when *ρ* = 0, the SNPs are assumed to be independent (DHGLM); and for 0 <*ρ* < 1, the correlation between two SNPs is a monomial function of *ρ*, which is referred to as the *smoothed* DHGLM [10]. We propose the use of smoothed DHGLMs since it reduces the noise in marker-specific variance estimates and highlights the signals of QTL. *ρ*, regarded as a spatial correlation parameter, was chosen to be 0.9 in this paper, which nicely shrank the SNPs with zero effect.

The overall phenotypic variance can be expressed as

where
is the variance of **z** *.*_{
j
} (the *j*-th column of **Z**) across individuals. These variance values can be directly calculated from the data. The contribution (heritability) of a particular SNP is expressed by
[4].

### Fitting algorithm

According to the extended likelihood principle, inference of the random SNP effects **g** should be drawn through the *h*-likelihood, fixed effects β through the marginal likelihood, and variance components **λ**, *σ*^{2} and
through the adjusted profile likelihood [11]. However, for efficient estimation, we propose to initialize variance components and iterate the following steps until convergence [7],

• Solve the following WLS problem for

and

**ĝ**,

Where
and
. The subscript *M* stands for ‘mean’.

• Update *σ*^{2} by fitting the deviance residuals
using an intercept-only gamma GLM and prior weight **w**_{
M
} = (**1** – **q**_{
M
})/2, where
are the residuals of (4), and
are the diagonal elements of
The subscript 1 and 2 stand for individuals (1 to *n*) and SNPs (*n* + 1 to *n* + *m*), respectively.

• Solve the following WLS problem for

*â* and

,

where
,
, **z** = log **λ** + (**d**_{M2} – **λ**)/**λ** is linearized **λ** in a gamma GLM with a log link, and **L** satisfies **LL**′ = **A**. The subscript *D* stands for ‘dispersion’.

• Update
by fitting the deviance residuals
using an intercept-only gamma GLM and prior weight **w**_{
D
} = (**1** – **q**_{
D
})*/*2, where ê_{
D
} are the last *m* residuals of (5), and **q**_{
D
} are the last *m* diagonal elements of
.