Evaluation

Here we attempt to evaluate individual methods by combining several characteristics together. This gives a more complex and comprehensive look at the performance of methods.

In particular, we look at:

  1. fraction of correct assignments
    correct assignments are weighted according to the complexity of the structure: thus 1-domain proteins are given weight of 1, 2-domain proteins are given weight of 2, 3-domain proteins are given weight of 3 and 4-,5-, and 6-domain protein are given weight of 4.
  2. fragmentation of discontinuous domains
  3. precision of domain overlap
    domain overlap is evaluated at the threshold of 80%: the chain is considered the be assigned correctly if the number of domains is assigned correctly and 80% of the residues in each domain are assigned correctly.
Method correctly assigned chains based on the number domains (in %) correctly fragmented domains (in %) Precision of domain overlap (at 80% threshold) (in %)
PDP 68.07 81.4 96.6
NCBI 43.31 86.6 86.4
DomainParser 33.17 99.2 98.3
PUU 38.32 62 94

Values of each of the characteristics contributing to the composite evaluation.

The three features above are weighted and combine to produce the final score, which is a composite evaluation score.


Composite evaluation of methods. Three characteristics are weighted as follows: fraction of correctly assigned chains: 60%, fragmentation of discontinuous domains: 20%, precision of domain overlap:20%. The overall performance is represented in percent.

We varied the weighting schema and found that regardless of the placed weights, PDP method appears to be superior and PUU method is inferior. However, DomainParser and NCBI methods trade places depending on whether fragmentation and precision of domain overlap (strong points of DomainParser) or correct assignment of domains (strong point of NCBI method) are weighted more heavily.



Weighting schema PDP NCBI DomainParser PUU
80-10-10 72.28 51.94 46.31 46.24
70-10-20 75.3 56.25 52.82 51.8
60-20-20 76.4 60.6 59.4 54.2
50-20-30 79.31 64.9 65.93 59.75
40-30-30 80.64 69.22 72.57 62.12

Composite evaluation of methods under different weighting schemas. The weighting parameters are as follows: first parameter: : fraction of correctly assigned chains, second parameter: fragmentation of discontinuous domains, third parameter: precision of domain overlap.

This work is sponsored by the National Institutes of Heath (NIH) Grant Number GM63208 (NIH/NIGMS)