 Proceedings
 Open access
 Published:
A new approach to enhance the performance of decision tree for classifying gene expression data
BMC Proceedings volumeÂ 7, ArticleÂ number:Â S3 (2013)
Abstract
Background
Gene expression data classification is a challenging task due to the large dimensionality and very small number of samples. Decision tree is one of the popular machine learning approaches to address such classification problems. However, the existing decision tree algorithms use a single gene feature at each node to split the data into its child nodes and hence might suffer from poor performance specially when classifying gene expression dataset.
Results
By using a new decision tree algorithm where, each node of the tree consists of more than one gene, we enhance the classification performance of traditional decision tree classifiers. Our method selects suitable genes that are combined using a linear function to form a derived composite feature. To determine the structure of the tree we use the area under the Receiver Operating Characteristics curve (AUC). Experimental analysis demonstrates higher classification accuracy using the new decision tree compared to the other existing decision trees in literature.
Conclusion
We experimentally compare the effect of our scheme against other well known decision tree techniques. Experiments show that our algorithm can substantially boost the classification performance of the decision tree.
Introduction
There are a lot of diseases available which needs to investigate more to understand them better. Due to lack of understanding of diseases e.g. breast cancer, often different outcome is shown for the same treatment applied to patients with similar clinical symptoms. Patient specific treatment could be one of the solutions to overcome this, however the varying outcome might be due to the limited knowledge about the relationship between treatment, disease development and clinical symptoms. The advent of gene expression data has opened up an opportunity to better understand diseases. However, to analyze the sheer amount of gene expression data is somehow complex and challenging due to the large dimension and small number of samples. The ultimate aim of analysis the data is to better diagnose and prognosticate diseases which in turn would provide an insight understanding the clinically relevant disease categories. Hence, an automated but simple computational technique is required to develop to classify diseases accurately using such high dimensional gene expression data. Among the existing machine learning techniques, decision tree is a well known and easy to understand classification technique [1, 2]. Its construction cost scales well for many features and instances and it is easily interpretable and require few parameter settings. Unfortunately, not many studies are available that used decision tree to classify gene expression data. This may be attributed due to the poor performance because of the limitation of dealing with high dimension but small number of instances. In bioinformatics, one of the general goals is to apply computers for analysis of gene expression data and classify them to appropriate diseases/disease status accurately. In this paper we describe our own modest efforts towards this goal through developing a new decision tree.
Decision tree is typically induced by selecting the gene feature for a node that has the least impurity when compared to the other gene features in the dataset. The gene expression dataset at the node is split into its child nodes using the selected gene feature such that the impurity is reduced as far as possible. Thus there are two phases and two issues that need to be considered while inducing a decision tree given the gene expression dataset with class levels. The phases are described in brief here:

Node selection: A decision tree is induced first by choosing a node as a root of the tree. For example, to classify the two types of lesion benign and malignant, we identified that the gene TFF3 can discriminate the patients with the least impurity. Hence, TFF3 gene is selected as root of the tree that divides the complete dataset into two or more subgroups with an aim to classify the dataset. The classification performance varies with varying choice of impurity measurement that would guide to induce the decision tree.

Splitting threshold: Once the node is selected, the dataset needs to be partitioned by choosing an optimal threshold value of the selected node. For example, we divide the complete dataset into two groups where in one group TFF3 >0.54 (i.e., the splitting threshold = 0.54) and in another group TFF3 â‰¤ 0.54. And we achieve the best classification performance for the above mentioned threshold among the performances by choosing other threshold values. Thus, the optimal splitting threshold for the node TFF3 will be 0.54.
The issues in inducing decision tree are described in brief here:

Stopping criteria In the process of inducing a decision tree the stopping criteria must be chosen to stop growing the tree at a suitable level, such that a better classification performance is achieved.

Labeling terminal nodes Terminal/leaf nodes are those nodes at which point growing of the tree is stopped, i.e., terminal nodes do not have any children. Usually class label (e.g., benign or malignant) is decided based on the labeling of the terminal nodes.
Several techniques have been applied over time to measure the impurity and the most popular ones are the entropylike uncertainty measures (i.e., gain ratio, information gain and gini index). Recently, Hossain et al. [3, 4] developed a decision tree called ROCtree, where Area Under Curve (AUC) is applied as an alternative to the entropylike uncertainty to attain an accurate classification of gene expression data. However, like other existing decision trees in ROCtree, each node is formed using a single gene feature. The feature that has the maximum AUC value with respect to the associated data is selected for a node. However, when more than one gene are combined using a linear function at each node of the tree, it can provide potentially even higher AUC value at each node compared to the single gene. Hence using multiple gene features for decision making at each node can improve performance substantially. In this paper we combined two genes together at each of the node of a decision tree to classify gene expression data. We call such trees as bivariate decision trees. We further motivate with the following example:
Example 1 Let us consider a gene expression dataset consisting of a large number of gene features A_{1}, A_{2}, A_{3}, ......, A_{ m } (m is some large number). Assume that the highest AUC is achieved for the feature A_{200} which is 0.6. This feature having the highest AUC among the all gene features, is selected as the node of the ROCtree. However, a linear combination of A_{450} and A_{29} (here, we consider a function to map the multiple gene features to a derived feature) provides an AUC of 0.75 that is higher than the maximum AUC of any single gene feature. Instead of building a tree using A_{200} as a node decision variable we propose to use the linear combination of A_{450} and A_{29} as the node of the tree.
As shown above, the limitation of ROCtree is, it only uses one gene at one node. In this paper, to alleviate this problem, we consider more than one gene expression at each node. In order to use multiple gene features at each node, we use least square estimation (LSE) to map the multiple features to one derived feature. Which is used to estimate the AUC value as in ROCtree. To split the dataset at the node we consider using a loss function known as HingeRankLoss [5]. Since, in this paper we restrict to use two gene features as a node to induce the decision tree, we call the new decision tree as bivariate ROCtree (BVROCtree).
Preliminaries
The receiver operating characteristic Curve
A receiver operating characteristics (ROC) curve, first used in signal detection theory, is used to evaluate the discriminative performance of a binary classifier. This is achieved by plotting the curve of the sensitivity vs. (1 âˆ’ specificity) for the binary classifier system by varying the discrimination threshold. The area under the ROC curve (AUC) can be computed using the trapezoidal integration. The maximum value of AUC can be 1 which indicates perfect classification whereas a value close to 0 indicates poor performance.
ROCtree
Previous work known as ROCtree [3, 4] has established the use of an ROC curve for node selection, to identify the discriminative features in the dataset and to induce a decision tree. First, the ROC curve is plotted for each of the pairs formed by each of the features and the class label. This means treating a single feature as a classifier and calculating the classification in terms of the sensitivity and specificity by varying the operating point. For each feature, the AUC is calculated and the feature with the highest AUC is selected for the node of the tree. The splitting threshold is chosen by taking each value of the selected feature from the dataset, and then attempting to classify based on a chosen value and calculating the misclassification rate for that value. The value with the minimum misclassification rate is finally chosen as the splitting threshold.
Bivariate ROCtree: BVROCtree
We now describe in more detail, the steps in our algorithm for building the BVROCtree.
Selection of more than one feature as node of the tree
In ROCtree a single feature is selected for a node that provides the maximum AUC among the all other features. According to the following theorem, if we map any single feature to a derived feature using any monotonic function f(.) the area under curve is not affected.
Theorem 1 The AUC(A) = AUC(f(A)), where, AUC(A) is area under curve using a single feature A and f(.) is any monotonic function.
The implication of the theorem 1 is in building a bivariate decision tree we do not need to consider mapping of any single feature. However, to use more than one feature at each node the set of features is mapped to a single feature using a linear function f(.) as in Eq. 1.
here, Ds = the dataset with several features selected from the training dataset D. \stackrel{\xe2\u2020\u2019}{b} represents the coefficient and {Y}^{\mathrm{\xe2\u20ac\xb2}} is the set of derived single feature values. The values of the coefficient \stackrel{\xe2\u2020\u2019}{b} are obtained by applying the least square estimation (LSE) formula.
Let us consider, D_{ s } = the dataset that contains all the values of the multiple selected features {A_{ Î± }, A_{ Î² }}. Thus, the values of \stackrel{\xe2\u2020\u2019}{b} is obtained using Eq. 2.
where, C={\left({D}_{\mathsf{\text{S}}}^{T}{D}_{\mathsf{\text{s}}}\right)}^{1} and Y=\left({y}_{1}\phantom{\rule{2.77695pt}{0ex}}{y}_{2}\phantom{\rule{2.77695pt}{0ex}}\xe2\u20ac\xa6{y}_{m}\right), m = total number of data instances.
Initially, the ROC curve is plotted for each of the pairs formed by each of the features along with the class label and the corresponding AUC is computed. The feature that has the highest AUC is identified. Let us assume this feature is A_{ s }. This selected feature is then paired with each feature in the remaining feature set and the corresponding linear coefficient is computed using LSE formula. For one such pair {A_{ s }, A_{ p }}, the coefficient is calculated using Eq. 2. We compute the values of {Y}^{\mathrm{\xe2\u20ac\xb2}} by using the coefficient values \stackrel{\xe2\u2020\u2019}{b} in Eq. 1. Then the AUC for the pair of {Y, {Y}^{\mathrm{\xe2\u20ac\xb2}}} is calculated. This AUC value indicates the level of influence of the corresponding pair of features in classifying the dataset. The feature set with the highest AUC value is selected as the node of the tree. Algorithm 1 presents pseudo code for selecting the most influential feature paired with another feature that has the highest AUC value, as a node in building the decision tree.
Algorithm 1: selectGenes: Selects the best combination of gene features based on AUC
Input: the training dataset: D, desired class labels Y, the best AUC value so far: bestAUC, the set of gene features that has generated bestAUC: GENE, the maximum number of gene features to be mapped onto a single value: limit
Comment(for BVROCtree limit = 2)
Output: the derived single feature values: {Y}^{\mathrm{\xe2\u20ac\xb2}}, the set of the features that generates the best AUC: selectedGenes, the best AUC: bestAUC
if GENE >= limit then
selectedGenes = GENE;
bestAUC = bestAUC;
{Y}^{\mathrm{\xe2\u20ac\xb2}} = the {Y}_{i}^{\mathrm{\xe2\u20ac\xb2}} value obtained for selectedGenes;
return;
end
if GENE=âˆ… then
for each gene feature A_{ i } in D do
calculate AU{C}_{{A}_{i}} for the pair of {A_{ i }, Y};
end
bestAUC = max(AU{C}_{{A}_{i}});
GENE = the ith gene feature A_{ i } that generates bestAUC;
else
for each gene feature A_{ i } paired with the gene feature or gene feature set GENE do
if A_{ i } is not in GENE then
D_{ s } = The dataset of all values for the gene features: A_{ i } âˆª gene;
\stackrel{\xe2\u0192\u2014}{b}\phantom{\rule{0.3em}{0ex}}=\phantom{\rule{0.3em}{0ex}}{\left({D}_{s}^{T}{D}_{s}\right)}^{1}{D}_{s}^{T}Y;
{Y}_{i}^{\mathrm{\xe2\u20ac\xb2}}={\stackrel{\xe2\u0192\u2014}{b}}^{T}\xc3\u2014\phantom{\rule{0.3em}{0ex}}{D}_{s};
calculate AU{C}_{{{Y}^{\mathrm{\xe2\u20ac\xb2}}}_{i}} for the pair of \left\{{{Y}^{\mathrm{\xe2\u20ac\xb2}}}_{i},\phantom{\rule{0.3em}{0ex}}Y\right\};
end
end
bestAUC = max(AU{C}_{{{Y}^{\mathrm{\xe2\u20ac\xb2}}}_{i}});
selectedGenes = the gene features A_{ i } âˆª GENE that generate bestAUC;
{Y}^{\mathrm{\xe2\u20ac\xb2}} = the {Y}_{i}^{\mathrm{\xe2\u20ac\xb2}} value obtained for gene features A_{ i } âˆª gene;
end
[{Y}^{\mathrm{\xe2\u20ac\xb2}}, selectedGenes, bestAUC] = SELECTGENES(D, Y, bestAUC, selectedGenes, limit);
Based on selection of a suitable splitting threshold (to be discussed shortly) for the selected set of genes, the dataset is then divided into two subsets. Each of the subsets is then used to further induce the tree in a similar way.
Example 2 Let us consider a dataset D of m examples, where each example comprises k gene features: A_{1}, A_{2}, A_{3}, ..., A_{ k }. Each of the k features has a differing discriminative power reflected by its respective AUC. Initially, to calculate the discriminative power that is expressed in terms of AUC, we compute the AUC for each gene feature paired with the desired class labels Y. The feature A_{ Î± } that produces the maximum AUC is selected. The selected gene feature A_{ Î± } is paired with each of the remaining features and for each pair the corresponding linear coefficients are calculated. Using the linear coefficients and Eq. 1 the set of class labels are predicted. For each set of predicted class labels the AUC value is computed. Suppose that the feature A_{ Î² }, where 1 â‰¤ Î² â‰¤ k and Î² â‰ Î± paired with the initially selected feature AÎ±. Consider \stackrel{\xe2\u2020\u2019}{b}(where \stackrel{\xe2\u0192\u2014}{b}=\xe2\u0178\xa8{b}_{\mathrm{\xce\pm}}\phantom{\rule{0.3em}{0ex}},\phantom{\rule{0.3em}{0ex}}{b}_{\mathrm{\xce\xb2}}\xe2\u0178\xa9) is the set of linear coefficients for this pair of features {A_{ Î± }, B_{ Î² }}. The predicted class labels are {Y}_{\mathrm{\xce\pm},\mathrm{\xce\xb2}}^{\mathrm{\xe2\u20ac\xb2}}, where
The AUC_{ Î±,Î² } is calculated for the pair of {Y, {Y}_{\mathrm{\xce\pm},\mathrm{\xce\xb2}}^{\mathrm{\xe2\u20ac\xb2}}}. If, AUC_{ Î±,Î² } is the maximum value among the AUCs for all other features (each of which is paired with the selected feature A_{ Î± }) and if, AUC_{ Î±,Î² } > AUC_{ Î± } then the set of features {A_{ Î± }, A_{ Î² } } is selected as the node. If, AUC_{ Î±,Î² } â‰¤ AUC_{ Î± } the feature A_{ Î± } is selected as the node. A suitable threshold is then obtained for this feature and the dataset D is divided into two subsets: D_{ left } and D_{ right }. Then, for each subset D_{ left } and D_{ right }, we recursively use the similar process by excluding the features used at the parent nodes and thus, induce the decision tree.
Splitting threshold
Splitting threshold is the value of the selected feature that discriminates the classes. This is an important step in inducing the tree to select the best threshold value such that misclassification of instances is minimum. To select the splitting threshold we use the HRL function [4, 5] in our BVROCtree. The HRL is a loss function which measures the loss or degree of error of a given classifier's output for a splitting threshold value. Consider a classifier whose output is real numbers. Assume that there are 8 data instances and the corresponding classified outputs by the classifier are: \xe2\u0178\xa8\mathsf{\text{1}},\phantom{\rule{2.77695pt}{0ex}}0.\mathsf{\text{4}},\phantom{\rule{2.77695pt}{0ex}}0.\mathsf{\text{7}},\phantom{\rule{0.3em}{0ex}}0.\mathsf{\text{9}},\phantom{\rule{2.77695pt}{0ex}}0.0\mathsf{\text{1}},\phantom{\rule{2.77695pt}{0ex}}0.\mathsf{\text{5}},\phantom{\rule{2.77695pt}{0ex}}0.\mathsf{\text{9}},\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{1}}\xe2\u0178\xa9, while the desired class labels for the corresponding instances are \xe2\u0178\xa81,\phantom{\rule{2.77695pt}{0ex}}1,\phantom{\rule{2.77695pt}{0ex}}+1,\phantom{\rule{2.77695pt}{0ex}}1,\phantom{\rule{2.77695pt}{0ex}}1,\phantom{\rule{2.77695pt}{0ex}}1,\phantom{\rule{2.77695pt}{0ex}}+1,\phantom{\rule{2.77695pt}{0ex}}+1\xe2\u0178\xa9. The outputs are ranked according to the values as in Table 1:
For a threshold value Î¸; the data instances are labeled either as 1 or +1 using classifier output as follows:
Considering Î¸ = 0.4, the labeling of the data instances are obtained as in Table 2:
In the HRL function, if the classifier's output is greater than Î¸ for an instance whose desired class label is 1 then it is counted as false positive (FP), otherwise it is a true negative (TN). Similarly, for a data instance with a desired class label +1, if the classifier's output is less than or equals to Î¸ then it is a false negative (FN), otherwise it is a true positive (TP). The rank distance penalty for each FN or TP instance corresponds to its distance in terms of ranks from the threshold. The HRL is the sum of all rank distance penalties. In the above example the total penalty for the false positives is 2 and the total penalty for the false negatives is 1 + 2 = 3, and the overall HRL is 2 + 3 = 5 [5].
In selecting the splitting threshold, we attempt to classify the dataset considering each value in the {Y}^{\mathrm{\xe2\u20ac\xb2}} ({Y}^{\mathrm{\xe2\u20ac\xb2}} is obtained for the feature computed through linear combination of more than one gene). For each chosen value Î¸ the corresponding HRL is computed. The Î¸ that generates the minimum HRL is selected as the splitting threshold.
Example 3 Let us consider the dataset D of m instances, where each instance has k features: A_{1}, A_{2}, A_{3}, ..., A_{ k }. Suppose, the linear combination of the pair of features {A_{ Î± }, A_{ Î² }}, where 1 â‰¤ Î± â‰¤ k, 1 â‰¤ Î² â‰¤ k and Î± â‰ Î², has the highest AUC and is selected to be the node in the tree. The set of features {A_{ Î± }, A_{ Î² }} is projected to a set of single values {Y}_{\mathrm{\xce\pm},\mathrm{\xce\xb2}}^{\mathrm{\xe2\u20ac\xb2}}where \left\{{{y}^{\mathrm{\xe2\u20ac\xb2}}}_{1},{{y}^{\mathrm{\xe2\u20ac\xb2}}}_{2},{{y}^{\mathrm{\xe2\u20ac\xb2}}}_{3},\phantom{\rule{2.77695pt}{0ex}}.\phantom{\rule{2.77695pt}{0ex}}.\phantom{\rule{2.77695pt}{0ex}}.\phantom{\rule{2.77695pt}{0ex}},{{y}^{\mathrm{\xe2\u20ac\xb2}}}_{m}\right\}\xe2\u02c6\u02c6{Y}_{\mathrm{\xce\pm},\mathrm{\xce\xb2}}^{\mathrm{\xe2\u20ac\xb2}}Each value of {Y}_{\mathrm{\xce\pm},\mathrm{\xce\xb2}}^{\mathrm{\xe2\u20ac\xb2}}is sorted and ranked. Let us rank the values without loss of generality as in Table 3:
Then for each value {y}_{1}^{\mathrm{\xe2\u20ac\xb2}},{y}_{2}^{\mathrm{\xe2\u20ac\xb2}},{y}_{3}^{\mathrm{\xe2\u20ac\xb2}},\phantom{\rule{2.77695pt}{0ex}}.\phantom{\rule{2.77695pt}{0ex}}.\phantom{\rule{2.77695pt}{0ex}}.\phantom{\rule{2.77695pt}{0ex}},{y}_{m\phantom{\rule{0.3em}{0ex}}}^{\mathrm{\xe2\u20ac\xb2}}of {Y}_{\mathrm{\xce\pm},\mathrm{\xce\xb2}}^{\mathrm{\xe2\u20ac\xb2}}, we form the rule:
where 1 â‰¤ j â‰¤ m. We then attempt to classify the dataset with this rule, and note the HRL.
Then the value {y}_{j}^{\mathrm{\xce\pm}}with the minimum HRL is selected as the splitting threshold for the pair of features {A_{ Î± }, A_{ Î² }}.
Pseudo code for calculating the splitting threshold is presented in Algorithm 2.
Stopping criterion
To stop growing the tree, the AUC of the selected combination of genes is tested. If the AUC value is equal to 1, yields that the combination of genes can classify the training dataset accurately with 100% sensitivity and 100% specificity. Therefore, there is no need to grow the tree further at this node. However, to avoid over fitting, we choose an AUC value â‰¥ 0.95 in order to stop growing the tree for a node. This facilitates us not to grow the tree for a smallest subset of the training dataset.
Labeling the leaf nodes
Each leaf node is labeled with a class label which is obtained by the majority of the class instances in that node. Algorithm 3 presents the pseudo code for inducing the BVROCtree using the functions presented in Algorithm 1 and Algorithm 2.
Related work
Several methods for constructing multivariate decision trees exist. In this section we describe some of the existing multivariate decision trees.
Linear discriminant analysis(LDA) has been used to combine multiple features at each node of the decision tree known as linear discriminant tree (LDT) developed by [6]. In this process, the impurity measurement is same as the C4.5 except that the splitting of combination of feature is done using LDA. It is claimed that the LDA based multivariate decision
Algorithm 2: CalculateSplitThreshold
Input: Y : The actual class label, {Y}^{\mathrm{\xe2\u20ac\xb2}}: The derived value from more than one gene feature
Output: Î¸: Splitting threshold for the node
HRL = âˆ…; Î¸ = âˆ…;
Sort and rank {Y}^{\mathrm{\xe2\u20ac\xb2}};
for each rank r do
splitThreshold = the value of {Y}^{\mathrm{\xe2\u20ac\xb2}} that corresponds to r;
D_{ neg } = âˆ…; D_{ pos } = âˆ…;
for each {Y}^{\mathrm{\xe2\u20ac\xb2}}do
if {Y}_{i}^{\mathrm{\xe2\u20ac\xb2}}â‰¤ splitThreshold then
D_{ neg } = D_{ neg } âˆª {{Y}_{i}^{\mathrm{\xe2\u20ac\xb2}}, Y_{ i }}
end
D_{ pos } = D_{ pos } âˆª {{Y}_{i}^{\mathrm{\xe2\u20ac\xb2}}, Y_{ i }}
end
TotalHRL = âˆ…;
for each Y_{ i } in D_{ neg } do
if Y_{ i } = +1 then
TotalHRL = TotalHRL + D_{ neg } âˆ’ i + 1;
Comment: D_{ neg } represents the total number of instances in D_{ neg } and the value of i ranges from 1 to D_{ neg };
end
end
for each Y_{ i } in D_{ pos } do
if Y_{ i } = 1 then
TotalHRL = TotalHRL + i;
Comment: the value of i ranges from 1 to Dpos;
end
if TotalHRL < HRL then
HRL = TotalHRL;
Î¸ = splitThreshold;
end
end
end
return Î¸;
tree can learn faster than other multivariate trees, however, the classification performance is no better than the other multivariate trees.
Breiman et al. [1] first introduced Classification And Regression Tree abbreviated CART, where multiple features are combined at a node of the tree. The algorithm looks for a splitting point followed by a linear test that achieves the least impurity. The limitation of CART is that, it can get stuck in a local minimum since the algorithm stops searching for the further combination of features when the impurity gets an increase in next to the current execution of above process. However, OC1 [7] a variant of CART solves this problem where the parameter update method follows the CART method, but includes random perturbations
Algorithm 3: BVROCtree
Input: The matrix of training examples: \mathcal{D}\phantom{\rule{0.1em}{0ex}}; the vector of class labels: Y
Output: T: A BVROCtree decision tree
if \mathcal{D}=\mathrm{\xe2\u02c6\dots} then
return a single node with âˆ…;
end
if Y consists of records all with the same value for the class label then
return a single leaf node with that value;
end
[{Y}^{\mathrm{\xe2\u20ac\xb2}}, GeneSet, AUC] = SELECTGENES(D, Y, 0, âˆ…, 2);
Î¸ = CALCULATESPLITTHRESHOLD(Y, {Y}^{\mathrm{\xe2\u20ac\xb2}});
Assign {\mathcal{D}}_{left} and {\mathcal{D}}_{right} as the subsets of \mathcal{D}\phantom{\rule{0.1em}{0ex}} consisting of records respectively with the value greater than or equal to and less than Î¸;
Assign Y_{ left } and Y_{ right } as the subsets of Y that correspond to the examples in {\mathcal{D}}_{left} and {\mathcal{D}}_{right} respectively;
Recursively apply BVROCtree to subsets {{\mathcal{D}}_{left}, Y_{ left }} and {{\mathcal{D}}_{right}, Y_{ right }} until they are empty or the stopping criteria are met; return a tree \mathcal{T}\phantom{\rule{0.1em}{0ex}} with root or node labeled \mathcal{A}\phantom{\rule{0.1em}{0ex}} and arcs labeled a_{1} and a_{2}, going respectively to the trees BVROCtree({\mathcal{D}}_{left}, Y_{ left }) and BVROCtree({\mathcal{D}}_{right}, Y_{ right });
of the parameters when a local minimum is reached and restarts from random location.
Logistic Model Tree (LMT) is another multivariate decision tree developed by [8], where features are combined using linear logistic regression and the selection of features and splitting are done following the same process as in C4.5.
Our BVROCtree described in this paper is different to all other existing trees in that we use a novel method based on AUC and the linear mapping function (using least square estimation) to select the combination of features to form a node. We also use a splitting criteria based on HRL as was used in ROCtree [4](the predecessor of our BVROCtree).
Experimental setup and datasets
For the experimental analysis, we compare against a number of well known simple decision tree induction techniques: ROCTree a predecessor of the proposed method, C4.5 [9], Ferri et al.'s [10] AUCsplit technique for decision trees, ADTree [11], Random Forest [12], REPTree and Random Tree. We also compare against the nondecision tree classifiers: NaÃ¯ve Bayes and kNN.
Datasets and validation scheme
Each of the techniques is applied on seven gene expression datasets. The properties of the datasets are illustrated in Table 1. To evaluate the performance of BVROCTree, a 10fold cross validation (CV) scheme is used 5 times for all datasets.
Results and discussion
The classification accuracies for all techniques on the considered gene expression datasets are presented in Table 5. The classification performances in AUC are presented in Table 6. In each table, the best performances among that of the reported classifiers are marked in bold.
Classification performance of BVROCtree
The classification performance of BVROCTree on the gene expression datasets clearly outperforms that of all the other reported decision trees (see Table 2). ROCTree, the predecessor of BVROCTree, have been reported in a previous study to perform consistently better classification in terms of accuracy and AUC measurement compared with other variants of decision tree classifiers including C4.5. Interestingly, the classification performance of BVROCTree is even better than its predecessor ROCTree. More specifically, for the datasets GE4 and GE7 this performance improvement of the BVROCtree is respectively at least 37% and 48% better than the ROCtree. Furthermore, the performance improvement of the BVROCtree over the other best performing decision trees is at least 17%, 3%, 10%, 4%, 10% and 37% for the datasets GE1, GE3, GE4, GE5, GE6, and GE7 respectively. This is evident that one of the reasons for this better performance is due to the application of more than one feature at each node of the tree along with the better computation of discriminative power of features used when building the tree and better splitting criteria that balances the loss and gain.
Comparison of AUC values
We also computed the overall AUC value of all classifiers considered in this paper (see Table 3), resulting from the 5 Ã— 10fold cross validation over the geneexpression datasets. The AUC values of BVROCtree is as good as of ROCtree for the datasets GE3, GE5 and GE6. For the other datasets the AUC values of BVROCtree are much higher than that of ROCtree. As the classification accuracy, the AUC values of ADTree for datasets GE2 is the best among all classifiers. However, for other datasets BVROCtree and its predecessor ROCtree outperform ADTree. Specifically, for six of the seven gene expression datasets, we see the BVROC âˆ’ tree has better AUC than other classifiers.
Comparison of tree sizes
The size of each tree built using the BVROCtree method always smaller compared to the other decision trees for all the datasets considered in this paper. We see in Table 4 that, the range of the size of BVROCtree is in between 2 to 3. While this range for C4.5 is in between 3 to 39. Although the range of the size of REPTree is from 1 to 51, the performance of REPTRee is much lower than the performance of BVROCtree (see Table 7). Since, in BVROCtree, a maximum two features are used to form a node, the range of features used in inducing BVROCtree is from 4 to 6 features. The combination of multiple features at each node using linear mapping can achieve a better discriminant strength compared to the single feature, and hence using a smaller tree size as induced in BVROCtree, a better classification performance is achieved when compared to the other decision trees.
Conclusion
We proposed a new decision tree BVROCtree which considers more than one feature at each node. The selection of features at each node makes use of linear mapping of multiple features to obtain a derived feature whose AUC values can be easily computed. Our experimental results show that the BVROCtree outperforms several state of the art competing classifiers both in accuracy and AUC values. Our experimental results show that our method is very effective for gene expression data with high number of dimensions. We believe that our proposed algorithm is a very practical and useful solution in classifying gene expression data. Since the classification accuracy has not been achieved as 100%, there exist scopes to enhance the BVROCtree. In BVROCtree we restricted the algorithm to combine a maximum of two features at each node of the tree. The classification performance could be improved by combining more than two features at each node, however in this case an intelligent method must be introduced such that the computational complexity remains reasonable. We plan to replace the linear mapping with any existing nonlinear mapping (e.g., through application of polynomial kernel or gaussian kernel) of multiple features as a node of the decision tree with an aim to further enhance the performance of the BVROCtree.
References
Breiman L, Friedman JH, Olshen RA, Stone CJ: Classification and Regression Trees. 1984, Chapman & Hall
Quinlan JR: Induction of decision trees. Machine Learning. 1986, 1: 81106.
Hossain MM, Hassan MR, Bailey J: Roctree: A novel decision tree induction algorithm based on receiver operating characteristics to classify gene expression data. Proceedings of the 2008 SIAM International Conference on Data Mining (SDM 2008). 2008, 455465.
Hossain MM: Using roc curves to improve classification with applications to gene expression data. 2009, Ph.D. dissertation, Department of Computer Science and Software Engineering, The University of Melbourne
Steck H: Hinge rank loss and the area under the roc curve. Machine Learning: ECML 2007, 18th European Conference on Machine Learning, Warsaw, Poland, September 1721, 2007, Proceedings. 2007, 347358.
John GH: Robust linear discriminant trees. Learning from Data: Artificial Intelligence and Statistics. 1996, 375385.
Murthy S, Kasif S, Salzberg S, Beigel R: Oci: Randomized induction of oblique decision trees. Proceedings of the Eleventh National Conference on Artificial Intelligence. 1993, 322327.
Landwehr N, Hall M, Frank E: Logistic model trees. Machine Learning. 2005, 59 (12): 161205. 10.1007/s1099400504663.
Quinlan JR: C4.5: Programs for Machine Learning. 1993, San Mateo, CA, USA: Morgan Kaufmann Publishers
Ferri C, Flach P, Ã¡ndezOrallo Hern: Learning decision trees using the area under the roc curve. Proceedings of the 19th International Conference on Machine Learning (ICML2002). 2002, Morgan Kaufmann, 139146.
Freund Y, Mason L: The alternating decision tree learning algorithm. Proceeding of the Sixteenth International Conference on Machine Learning. 1999, Bled, Slovenia, 124133.
Breiman L: Random forests. Machine Learning. 2001, 45 (1): 532. 10.1023/A:1010933404324.
CritchleyThorne RJ, Yan N, Nacu S, Weber J, Holmes SP, Lee PP: Down regulation of the interferon signaling pathway in t lymphocytes from patients with metastatic melanoma. PLoS Medicine. 2007
Zizhen Y, Jochen J, Walter R, Cecile M, Mary E, Uta F, Dianna M, Stephen S, Eileen M: A marfan syndrome gene expression phenotype in cultured skin fibroblasts. BMC Genomics. 2007, 8 (1): 31910.1186/147121648319.
Gordon GJ, Jensen RV, Hsiao LL, Gullans SR, Blumenstock JE, Ramaswamy S, Richards WG, Sugarbaker DJ, Bueno R: Translation of microarray data into clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer and mesothelioma. Cancer Research. 2002, 62: 49634967.
Singh D, Febbo PG, Ross K, Jackson DG, Manola J, Ladd C, Tamayo P, Renshaw AA, D'Amico AV, Richie JP, Lander ES, Loda M, Kantoff PW, Golub TR, Sellers WR: Gene expression correlates of clinical prostate cancer behavior. Cancer Cell. 2002, 1: 203209. 10.1016/S15356108(02)000302.
Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES: Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science. 1999, 286 (5439): 531537. 10.1126/science.286.5439.531.
Notterman DA, Alon U, Sierk AJ, Levine AJ: Transcriptional gene expression profiles of colorectal adenoma, adenocarcinoma, and normal tissue examined by oligonucleotide arrays. Cancer Research. 2001, 61 (7): 31243130.
Acknowledgements
Authors acknowledge the support of Deanship of Scientific Research, King Fahd University of Petroleum and Minerals in conducting the research.
Funding The work described in this paper is supported by the SABICFAST track research grant (Project Number: FT101001) awarded by King Fahd University of Petroleum and Minerals, Saudi Arabia.
Declarations
Publication of this work was supported by Ramamohanarao Kotagiri's funding at the University of Melbourne.
This article has been published as part of BMC Proceedings Volume 7 Supplement 7, 2013: Proceedings of the Great Lakes Bioinformatics Conference 2013. The full contents of the supplement are available online at http://www.biomedcentral.com/bmcproc/supplements/7/S7.
Author information
Authors and Affiliations
Corresponding author
Additional information
Competing interests
MRH has received grant from Deanship of Scientific Research, King Fahd University of Petroleum and Minerals in conducting the research. RK received support from the University of Melbourne for the cost of the publication. All authors declare that there is no financial relationships other than the above mentioned ones with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the work described in this paper.
Authors' contributions
The idea of the BVROC was generated by MRH. MRH implemented the method and prepared the manuscript. RK helped in preparing the manuscript.
Rights and permissions
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Hassan, M.R., Kotagiri, R. A new approach to enhance the performance of decision tree for classifying gene expression data. BMC Proc 7 (Suppl 7), S3 (2013). https://doi.org/10.1186/175365617S7S3
Published:
DOI: https://doi.org/10.1186/175365617S7S3