Evaluating the performance of a method using only the number of domains it assigns might not be sensitive enough to determine the accuracy of an automated method. At times, methods disagree with experts on exactly where domains starts and/or end, i.e. there is disagreement on the placement of domain boundaries, while there is agreement on the number of assigned domains.

To address the issue of overlap between the domains assigned by experts and those assigned by automatic methods we first define a boundary consistency criterion. A boundary consistency of 75% requires that 75% of the domain length, as defined by two domain assignment methods, is the same. Empirically we chose boundary_consistency_90 as a definition of identical assignment by two methods: domain assignment is considered identical if domains overlap for at least 90% of the chain length.

Fraction of accurately assigned domains at 90% of domain boundary accuracy

Tabular data for the figure

	Fraction of accurate assignments	Fraction of chains that failed domain overlap criterion	Fraction of chains that failed domain number criterion
DPD	0.83	0.03	0.14
DomainParser	0.79	0.01	0.20
PUU	0.69	0.08	0.23
NCBI	0.68	0.14	0.18

Accuracy of domain assignments – a strict assessment. 90% of the domain boundary consistency is required (see discussion above). Boundaries_consistency_90 for automatic methods evaluated using Green is accurate assignments, yellow is inaccurate domain overlapping, and red is incorrect domain numbering. The evaluation is performed using Balanced_Domain_Benchmark_3.

Fraction of mis-assigned domain boundaries at varying levels of boundary consistency

Tabular data for the figure

fraction of domain overlap	PDP	DomainParser	PUU	NCBI
0.35	0.003774	0.004167	0.004329	0.042802
0.45	0.003774	0.004167	0.008658	0.054475
0.5	0.003774	0.004167	0.012987	0.081712
0.55	0.007547	0.004167	0.012987	0.085603
0.6	0.007547	0.004167	0.017316	0.089494
0.65	0.007547	0.008333	0.021645	0.101167
0.7	0.011321	0.0125	0.030303	0.116732
0.75	0.018868	0.0125	0.030303	0.128405
0.8	0.033962	0.016667	0.060606	0.136187
0.85	0.041509	0.020833	0.073593	0.151751
0.9	0.067925	0.033333	0.125541	0.225681
0.95	0.120755	0.058333	0.194805	0.287938

Performance of each method is evaluated at different levels of domain boundary accuracy. The evaluation is performed using Balanced_Domain_Benchmark_3.