- Open Access
Comparison of analyses of the QTLMAS XIV common dataset. I: genomic selection
BMC Proceedings volume 5, Article number: S1 (2011)
For the XIV QTLMAS workshop, a dataset for traits with complex genetic architecture has been simulated and released for analyses by participants. One of the tasks was to estimate direct genomic values for individuals without phenotypes. The aim of this paper was to compare results of different approaches used by the participants to calculate direct genomic values for quantitative trait (QT) and binary trait (BT).
Participants applied 26 approaches for QT and 15 approaches for BT. Accuracy for QT was between 0.26 and 0.89 for males and between 0.31 and 0.89 for females, and for BT ranged from 0.27 to 0.85. For QT, percentage of lost response to selection varied from 8% to 83%, whereas for BT the loss was between 15% and 71%.
Bayesian model averaging methods predicted breeding values slightly better than GBLUP in a simulated data set. The methods utilizing genomic information performed better than traditional pedigree based BLUP analyses. Bivariate analyses was slightly advantageous over single trait for the same method. None of the methods estimated the non-additivity of QTL affecting the QT, which may be one of the constrains in accuracy observed in real data.
An idea of genomic selection (GS) has been presented nearly a decade ago  and since that time it has been applied to plant  and animal breeding . Together with an increased availability of dense marker assays, implementation of GS in breeding programs has become more popular [4, 5] stimulating development of methods to estimate genomic breeding values.
Genetic basis of a phenotypic trait – its genetic architecture - is often complex. A particular trait may be, for example, controlled by many genes with small effects or by several major genes. Genes that control one trait may also control other trait(s), i.e. they are pleiotropic and the traits are genetically correlated. A gene variant may have an effect when it is inherited from a parent of one sex but not from the other (i.e. imprinting) or its effect will be present only when several alleles are in a particular combination (i.e. epistasis or haplotype effect). Genomic selection opens new opportunities in the analyses of complex traits.
A number of approaches have been developed to obtain direct genomic values (DGV) or genomic-enhanced breeding values (GEBV) [1, 6]. Because number of markers is usually greater than number of genotyped individuals, predictions of individual genes are based on Bayesian model averaging, penalized regression, dimension reduction methods and algorithmic machine learning methods. .Several Bayesian models have been developed to model effects of individual loci. They differ in number of characteristics, including a prior distribution of number of QTL, their effects and assumption of homogeneity or heterogeneity of QTL variance (Table 1). Ridge regression (RR) and spatial regression are two types of panelized estimation, which assume homogenous variance across all markers. Double hierarchical generalized linear models (DHGLM) estimates marker-specific variances and can be solved by the iteratively weighted least squares. Partial Least Square Regression (PLSR) is an extension of the principal component analyses (PCA): the most systematic variations in marker data are decomposed into a small number of latent variables (principal components). This method reduces the dimensionality of the problem utilizing existing correlations between SNP . GBLUP is an alternative, which treats the markers as a source of information on relatedness among individuals and models the sum of all QTL instead of individual loci. Some variants of GBLUP use preselected SNP to build relationship matrix for particular trait . Some authors apply machine learning approaches (boosting, support vector), with hope that these methods better account for interaction between QTL .
For the XIV QTLMAS workshop, a dataset for traits with complex genetic architecture has been simulated and released for analyses by participants . One of the tasks was to estimate DGV for individuals without phenotypes. The aim of this paper was to evaluate and compare results of different approaches used by the participants to calculate DGV.
Simulated, four-generation pedigree consisted of 3,226 individuals, descended from 20 founders, each mating resulted in 30 offspring. The last generation consisted of 900 young individuals with no progeny and no phenotypes. All 3,226 individuals had 100 Mb long genomes consisting of 5 chromosomes. In total, 37 out of 10,1031 single nucleotide polymorphism (SNP) markers were assumed to be QTL of which two had major effects. One of the simulated traits was a quantitative trait (QT) and the other one was a binary trait (BT). Heritability for QT, due to imprinting, was higher for males (0.52) than for females (0.39). True breeding values (TBV) for QT were calculated as a summation of effects of 30 additive QTL, haplotype effects (QTL pairs 31-32 and 33-34) and effects of imprinted QTL (for males only). Heritability for BT was 0.48. TBV for BT were calculated as a summation of effects of 22 additive QTL. Simulated pedigree, genome, marker and phenotypic data were made available for analyses. More detailed description of simulation can be found in  and the simulated dataset is available at http://jay.up.poznan.pl/qtlmas2010/dataset.html.
Methods used by participants to estimate genomic breeding values
Eleven groups submitted their estimates of DGV. Participants applied several methods and often different variants of same method [8, 10–17]. In total, they applied 26 approaches for QT and 15 approaches for BT (Table 2 and 3). The QT was analyzed by 11 groups, whereas BT by 6 groups. Ten groups used univariate models and two groups applied bivariate models. Bayesian models were used by five groups, machine learning was applied by a single group, and eight groups tested other methods.
Five criteria were used to compare the applied genomic selection methods: (1) Accuracy being the Pearson correlation between true breeding values (TBV) and DGV. (2) Bias of estimates calculated as the linear regression coefficient (TBV = M + b*DGV +E) (unbiased estimates are expected to have regression coefficient of 1), (3) mean square difference (MSD) between TBV and DGV, (4) % of shared ID when selecting top 10% (45 males and 45 females) based on DGV vs. TBV, and (5) selection loss from selecting on DGV instead of TBV as a proportion of response using TBV. Due to a presence of imprinting, the average genetic values for males and females were different, and therefore, accuracies and regression coefficients for these groups were calculated separately.
For QT, the accuracy was between 0.26 and 0.89 for males and between 0.31 and 0.89 for females (Table 2). Most of the approaches using Bayesian model averaging methods performed slightly better (average accuracy 0.68) than other methods. GBLUP yielded an average accuracy of 0.61 (after exclusion of the least accurate case of GBLUP). Traditional pedigree BLUP ignoring genomic data was only about half as accurate as the best approaches. No substantial differences between bivariate and univariate analyses were found. For BT, accuracy level was, generally, higher than for QT and ranged from 0.27 to 0.85 (Table 3). Similarly to the QT, Bayesian approaches were somewhat superior to other methods. The highest accuracy was reached by bivariate BayesC approach. Unlike QT, for BT bivariate analyses were considerably more accurate than univariate ones.
For QT, regression coefficients ranged from 0.19 to 9.05 for males and from 0.15 to 7.31 for females (Table 2). Similarly to the previous criterion, the best performing approaches were Bayesian methods. For BT, regression coefficients ranged from 0.61 for some of GBLUP applications to 0.77 for bivariate BayesC (Table 3). Again for this trait bivariate analyses appeared to be better than univariate, which was not the case for QT.
Mean square difference (MSD)
For QT, MSD for most of the approaches ranged from 42 to 63 (Table 2). Higher MSD were observed for one of the machine learning techniques - boosting – (280.7), one case of GBLUP analyses (87.0) and PLSR (76.4). These approaches were inferior in comparison to pedigree BLUP that yielded MSD of 66.4-66.8. For BT, MSD ranged from 0.26 for BayesC bivariate to 1.20 for PLSR. These results indicate that BayesC bivariate was superior to remaining methods (Table 3).
For QT, percentage of ID shared between the groups of young individuals selected on TBV and DGV varied substantially and ranged from 16% (PLSR) to 77% (Table 2). The best three methods were: BayesB (74%-77%), BayesCpi (77%) and TA-BLUP (72%-73%). For pedigree BLUP only 17% (univariate) or 23% (bivariate) ID were shared with true top individuals. For BT, similar range of variation of shared ID was observed (from 20% with PLSR to 64% with bivariate BayesC) (Table 3). The best three approaches were: bivariate BayesC (64%), bivariate GBLUP (60%), and bivariate BayesA (60%). For pedigree BLUP only 30% (univariate) or 28% (bivariate) ID were shared.
High percentage of shared ID, generally, was associated with low level of loss in genetic gain. For QT, percentage of lost response to selection when 10% are selected based on DGV instead of TBV varied from low (8%) to very high (83%). Pedigree BLUP resulted in 62% to 79% of loss, whereas approaches using genomic information, in general, resulted in smaller loss. Most of analyses using BayesB as well as BayesCpi and TA-BLUP appeared to be superior to other methods and caused only 8% to 9% of loss. Percentage of loss for most of the GBLUP approaches was close to 20%. For BT, loss in response to selection was, usually, at higher level. The smallest observed loss was 15% for bivariate BayesB and the biggest was 71% for PLSR. Pedigree BLUP caused from 74% to 79% of loss. Bivariate analyses were superior to univariate for both traits.
When phenotypes for young individuals are not available, the approaches that use genomic information had superior performance compared to the methods that were based solely on pedigree information. Use of genomic information, therefore, led to improved breeding value estimation, which was also found by others [1, 18–24]. The traits simulated for the XIV QTL-MAS workshop differed with respect to complexity and a number of QTL. Effects of simulated QTL were unequal, some QTL had large effects and most of the other QTL had small effects on the simulated traits. All Bayesian model averaging methods had similar accuracy. These models were, furthermore, expected to achieve higher accuracy than GBLUP because of relatively small number of QTL . GBLUP, however, was expected to capitalize on genetic relationships between training and validation sets . We have found that these two groups of methods yielded similar accuracies. Bastiaansen et al.  who analyzed results of the previous QTLMAS workshop also reached a similar conclusion. Lack of apparent advantage in terms of accuracy of a single method across a range of traits was also shown in other simulation studies  and in real data, e.g.  and .
When the same approach is used by different researchers, similar results are expected, which was not always the case in our comparison. One GBLUP implementation, for example, was about 0.20 less accurate than other GBLUP analyses. This suggests that the methods may be very sensitive to data preparation and that their implementations may vary in performance.
Bivariate analyses, in general, performed better than univariate analyses for the same approach. This was expected as the two simulated traits were indeed genetically correlated. Differences between univariate and bivariate analyses were especially apparent for BT, for which phenotypes carry less information. More complex approaches, requiring initial estimation of marker effects or use of machine learning techniques applied to QT provided similar or inferior results to simpler methods.
Bayesian model averaging methods predicted breeding values slightly better than GBLUP in a simulated data set, where traits had complex genetic architecture (epitasis, pleiotropy, and imprinting) and were affected by relatively small number of QTL. The methods utilizing genomic information performed better than traditional pedigree based BLUP analyses. Bivariate analyses were slightly advantageous over single trait for the same method. None of the methods estimated the non-additivity of QTL affecting the QT, which may be one of the constrains in accuracy observed in real data.
Meuwissen THE, Hayes BJ, Goddard ME: Prediction of Total Genetic Value Using Genome-Wide Dense Marker Maps. Genetics. 2001, 157 (4): 1819-1829.
Rafalski A: Applications of single nucleotide polymorphisms in crop genetics. Current Opinion in Plant Biology. 2002, 5 (2): 94-100. 10.1016/S1369-5266(02)00240-6.
Hayes BJ, Bowman PJ, Chamberlain AJ, Goddard ME: Invited review: Genomic selection in dairy cattle: Progress and challenges. J Dairy Sci. 2009, 92 (2): 433-443. 10.3168/jds.2008-1646.
Rothschild MF, Plastow GS: Impact of genomics on animal agriculture and opportunities for animal health. Trends in Biotechnology. 2008, 26 (1): 21-25. 10.1016/j.tibtech.2007.10.001.
Dekkers JCM: Commercial application of marker- and gene-assisted selection in livestock: Strategies and lessons. J Anim Sci. 2004, 82 (13_suppl): E313-328.
Xu S: Estimating polygenic effects using markers of the entire genome. Genetics. 2003, 163 (2): 789-
Bjornstad A, Westad F, Martens H: Analysis of genetic marker-phenotype relationships by jack-knifed partial least squares regression (PLSR). Hereditas. 2004, 141 (2): 149-165. 10.1111/j.1601-5223.2004.01816.x.
Ogutu JO, Piepho H-P, Schulz-Streeck T: A comparison of random forests, boosting and support vector machines for genomic selection with SNP markers. BMC Proceedings. 2011, 5 (Suppl 3): S11-10.1186/1753-6561-5-S3-S11.
Szydlowski M, Paczynska P: QTL-MAS 2010: Simulated Dataset. BMC Proceedings. 2011, 5 (Suppl 3): S4-10.1186/1753-6561-5-S3-S4.
Calus MPL, Mulder HA, Veerkamp RF: Estimating genomic breeding values and detecting QTL using univariate and bivariate models. BMC Proceedings. 2011, 5 (Suppl 3): S5-10.1186/1753-6561-5-S3-S5.
Cleveland MA, Hickey JM, Kinghorn BP: Genotype imputation for the prediction of genomic breeding values in non-genotyped and low-density genotyped individuals. BMC Proceedings. 2011, 5 (Suppl 3): S6-10.1186/1753-6561-5-S3-S6.
Coster A, Calus MPL: Partial least square regression applied to the QTLMAS 2010 dataset. BMC Proceedings. 2011, 5 (Suppl 3): S7-10.1186/1753-6561-5-S3-S7.
Nadaf J, Pong-Wong R: Applying different genomic selection approaches on QTL-MAS 2010 data. BMC Proceedings. 2011, 5 (Suppl 3): S9-10.1186/1753-6561-5-S3-S9.
Schulz-Streeck T, Ogutu JO, Piepho H-P: Pre-selection of markers for genome-wide selection. BMC Proceedings. 2011, 5 (Suppl 3): S12-10.1186/1753-6561-5-S3-S12.
Shen X, Ronnegard L, Carlborg O: Hierarchical likelihood opens a new way of estimating genetic values using genome-wide dense marker maps. BMC Proceedings. 2011, 5 (Suppl 3): S14-10.1186/1753-6561-5-S3-S14.
Sun X, Dekkers JCM: Genomic Breeding Value Prediction and QTL Mapping of QTLMAS 2010 Data Using Bayesian Methods. BMC Proceedings. 2011, 5 (Suppl 3): S13-10.1186/1753-6561-5-S3-S13.
Zhang Z, Ding X, Liu J, de Koning D-J, Zhang Q: Genomic selection for QTL-MAS data using a trait specific relationship matrix. BMC Proceedings. 2011, 5 (Suppl 3): S15-10.1186/1753-6561-5-S3-S15.
Muir WM: Comparison of genomic and traditional BLUP-estimated breeding value accuracy and selection response under alternative trait and genomic parameters. J Anim Breed Genet. 2007, 124 (6): 342-355. 10.1111/j.1439-0388.2007.00700.x.
Mulder HA, Calus MPL, Veerkamp RF: Prediction of haplotypes for ungenotyped animals and its effect on marker-assisted breeding value estimation. Genet Sel Evol. 2010, 42 (1): 10.1186/1297-9686-42-10.
Mulder HA, Meuwissen THE, Calus MPL, Veerkamp RF: The effect of missing marker genotypes on the accuracy of gene-assisted breeding value estimation: a comparison of methods. Anim. 2010, 4: 9-19. 10.1017/S1751731109990838.
Meuwissen T, Goddard M: The use of marker haplotypes in animal breeding schemes. Genet Sel Evol. 1996, 28 (2): 161-176. 10.1186/1297-9686-28-2-161.
Schaeffer LR: Strategy for applying genome-wide selection in dairy cattle. J Anim Breed Genet. 2006, 123 (4): 218-223. 10.1111/j.1439-0388.2006.00595.x.
Bastiaansen J, Bink M, Coster A, Maliepaard C, Calus M: Comparison of analyses of the QTLMAS XIII common dataset. I: genomic selection. BMC Proceedings. 2010, 4 (Suppl 1): S1-10.1186/1753-6561-4-s1-s1.
Pszczola M, Mulder HA, Calus MPL: Effect of enlarging the reference population with (un)genotyped animals on the accuracy of genomic selection in dairy cattle. J Dairy Sci. 2011, 94 (1): 431-441. 10.3168/jds.2009-2840.
Daetwyler HD, Pong-Wong R, Villanueva B, Woolliams JA: The Impact of Genetic Architecture on Genome-Wide Evaluation Methods. Genetics. 185 (3): 1021-1031. 10.1534/genetics.110.116855.
Habier D, Tetens J, Seefried F-R, Lichtner P, Thaller G: The impact of genetic relationship information on genomic breeding values in German Holstein cattle. Genet Sel Evol. 2010, 42 (1): 5-17. 10.1186/1297-9686-42-5.
Moser G, Tier B, Crump R, Khatkar M, Raadsma H: A comparison of five methods to predict genomic breeding values of dairy bulls from genome-wide SNP markers. Genet Sel Evol. 2009, 41 (1): 56-10.1186/1297-9686-41-56.
Wolc A, Stricker C, Arango J, Settar P, Fulton J, O'Sullivan N, Preisinger R, Habier D, Fernando R, Garrick D, et al: Breeding value prediction for production traits in layer chickens using pedigree or genomic relationships in a reduced animal model. Genet Sel Evol. 43 (1): 5-10.1186/1297-9686-43-5.
Publication of this supplement was supported by Animal Breeding and Inseminating Center LTD in Bydgoszcz, Poland.
This article has been published as part of BMC Proceedings Volume 5 Supplement 3, 2011: Proceedings of the 14th QTL-MAS Workshop. The full contents of the supplement are available online at http://www.biomedcentral.com/1753-6561/5?issue=S3.
The authors declare no competing interests. MS and MP preformed comparative analyses.
MP and TS drafted the manuscript. MS and MP preformed comparative analyses. AW MS TS SM and MP critically revised the manuscript and contributed to discussion of the results.
About this article
Cite this article
Pszczola, M., Strabel, T., Wolc, A. et al. Comparison of analyses of the QTLMAS XIV common dataset. I: genomic selection. BMC Proc 5, S1 (2011). https://doi.org/10.1186/1753-6561-5-S3-S1
- Quantitative Trait
- Partial Little Square Regression
- Genomic Selection
- Mean Square Difference
- Binary Trait