Automatic Assessment of CASP5 (by Michael Levitt 18 January 2003) ================================================================= (1) All the assessment is based on the official GDT_TS scores calculated and released at the Asilomar meeting. (2) The targets in each of the three categories are those selected by the three human assessors. (3) Ranks are based on a total Z-Score. This total is the sum of Z-scores above 1.0 for all models of a particular group. Groups ID's are from the official score file. The names were added manually and those servers that were in CAFASP are indicated in bold. (4) For Comparative Modeling and Fold Recognition the first model is the only one used. For New Folds, all targets are assessed. When more than a single target is assessed, the weights of the models of a particular group and particular target are weighted to have a total weight of 1.0 (i.e. the same as that when just one target is assessed). (5) The Z-Score of a particular model for a particular target is calculated as: Z-Score = ( GDT_TS - mean ) / SD. (6) The mean and SD of the model are calculated without assuming a score distribution. * The mean value is the value of GDT_TS with 52.5% of the data having a lower value. For a normal distribution, this is equivalent to eliminating the data worse than 2 standard deviations (in lowest 2.5%) and then taking the mean of the remaining data. * The SD value is calculated as the GDT_TS value that has 15.105% of the scores above it. For a normal distribution, this would just be the standard deviation. Some General Observations ========================= Comparative Modeling -------------------- * Groups from Poland dominate this section with human entries in rank 1 (Bujnicki-Janus), 2 (Ginalski) and 3 (GeneSilico) and an automatic server in rank 4 (BIOINFO.PL, Leszek Rychlewski). Honig in rank 4 is the best non-server related entry in the top nine. * The best real server (non-meta) is ORNL-PROSPECT (rank 6); no other non-meta server makes the top-40 list. Fold Recognition ---------------- * Groups from Poland also dominate this section with human entries in rank 1 (Ginalski) and an automatic server (BIOINFO.PL) in rank 4. The Skolnick and Baker entries are at ranks 2 and 3 and the Baker server is at rank 5 (BAKER-ROBETTA). Shortle did well in rank 5 and Brooks in rank 7. * The best real server (non-meta) is ARBY-SCAI in rank 16. New Folds --------- * Surprise entries on the list are I-sites/Bystrof (rank 3) and Chimera (rank 8). * The meta-servers, BAKER-ROBETTA and PMODEL3 are tied in ranks 10 and 11. * The best non-meta server is SAMUDRALA-NF (which is essentially the same as PROTINFO-AB). All Targets ----------- * The best server, BIOINFO.PL ranks 5-th and is almost as good as three out of the four best human group (Bujnicki-Janusz, Baker and Skolnick-Kolinski in ranks 2, 3 and 4). The top ranking human (Ginalski) used the BIOINFO.PL server for difficult targets. * The best real server (non-meta) is ORNL-PROSPECT in rank 12. People who were asked to speak in the Comparative Modeling, Fold Recognition and New Fold parts of the Asilomar meeting are colored by the section in which they spoke. There is not much correlation with the section in which people spoke and the rank on the targets in that section but by and large most of the groups who did well were asked to speak. CM Comparative Modeling (targets manually assessed by Anna Tramontano) CM =================================================================== CM 130 132 133 136_1 136_2 140 141 142 143_1 143_2 149_1 CM 150 151 152 153 155 159_1 159_2 160 165 167 168_1 CM 168_2 169 172_1 176 177_1 178 179_1 179_2 182 183 184_1 CM 184_2 185_1 186_2 185_3 188 189 191_2 193_2 195 CM Use the first model for all the targets. CM CM Total Z-Scores Above 1.0 for CASP5 All Targets Listed Above: CM ----------------------------------------------------------------------------- CM Rank Group Z-Score Ngood Npred NgNW NpNW Group-name CM ----------------------------------------------------------------------------- CM 1 G020 47.17 24.50 38.50 25 39 Bujnicki-Janusz CM 2 G453 46.55 27.00 42.00 27 42 Ginalski CM 3 G517 42.82 25.50 39.50 26 40 GeneSilico CM 4 G110 36.03 24.00 35.00 24 35 Honig CM 5 G006 31.23 19.00 40.00 19 40 BIOINFO.PL CM 6 G012 27.22 19.50 35.00 20 36 ORNL-PROSPECT CM 7 G427 27.21 18.50 38.00 19 39 Fischer CM 8 G265 25.75 15.50 32.00 16 33 Sasson-Iris CM 9 G223 25.60 16.00 30.00 16 30 3D-SHOTGUN-3DS3 CM 10 G100 24.95 14.00 29.00 14 29 Sbi CM 11 G028 23.88 17.50 34.50 18 35 Celltech CM 12 G153 23.29 17.00 35.00 17 35 Chimera CM 13 G041 23.13 12.00 28.00 12 28 Cbrc CM 14 G329 23.04 13.00 32.00 13 32 Dunbrack CM 15 G224 22.16 15.00 27.00 15 27 3D-SHOTGUN-3DS5 CM 16 G450 21.98 13.00 34.50 13 35 Tome CM 17 G448 21.55 14.00 17.00 14 17 Murzin CM 18 G425 21.34 12.00 17.00 12 17 Venclovas CM 19 G096 20.85 14.50 31.50 16 35 Bates-Paul CM 20 G010 20.81 11.00 26.00 11 26 Skolnick-Kolinski CM 21 G029 18.46 13.00 33.00 13 34 BAKER-ROBETTA CM 22 G222 17.30 12.00 26.00 12 26 3D-SHOTGUN-INBGU CM 23 G112 16.90 10.00 34.00 11 35 Friesner CM 24 G475 16.48 7.00 22.00 7 22 Bionomix CM 25 G067 16.35 12.00 30.50 13 32 Jones CM 26 G040 15.84 11.00 32.00 11 32 PMODEL CM 27 G002 15.83 8.00 24.00 9 26 Baker CM 28 G464 15.72 7.00 22.00 7 22 Atome CM 29 G417 14.86 9.00 22.00 9 22 Cbsu CM 30 G471 14.73 6.00 14.00 6 14 Martin-Andrew CM 31 G368 13.80 11.00 32.50 11 33 Jose CM 32 G180 13.23 9.00 25.00 9 25 evolutionaries CM 33 G093 13.18 5.00 17.50 5 18 Schulten-Wolynes CM 34 G246 12.85 7.00 15.00 7 15 MZ-Brussels CM 35 G301 12.85 9.00 20.50 9 21 Ucla-doe CM 36 G084 12.84 9.00 30.00 9 30 Sbc CM 37 G170 12.70 9.00 23.00 9 23 Chimerax CM 38 G202 11.50 8.00 24.00 10 27 Yasara-Pushchino CM 39 G447 11.20 7.00 28.00 7 28 Cam-Biochem CM 40 G288 10.87 6.00 27.00 6 27 Lomize-Andrei CM ----------------------------------------------------------------------------- CM NgNW is the number of good predictions without weighting for multiple models. CM NpNW is the number of total predictions without weighting for multiple models. CM All-uppercase names are of CAFASP registered servers. FR Fold Recognition (targets manually assessed by Nick Grishin) FR ============================================================ FR 134 138 156 157 174 193_1 135 147 148 162_1 162_2 187_2 191_1 FR Use the first model for all the targets. FR FR Total Z-Scores Above 1.0 for CASP5 All Targets Listed Above: FR ----------------------------------------------------------------------------- FR Rank Group Z-Score Ngood Npred NgNW NpNW Group-name FR ----------------------------------------------------------------------------- FR 1 G453 24.26 9.00 12.00 9 12 Ginalski FR 2 G010 21.64 7.00 12.00 7 12 Skolnick-Kolinski FR 3 G002 19.55 8.00 12.50 9 14 Baker FR 4 G006 16.88 6.00 10.00 6 10 BIOINFO.PL FR 5 G349 15.25 7.00 7.00 7 7 Shortle FR 6 G029 14.56 6.50 11.50 7 13 BAKER-ROBETTA FR 7 G373 13.49 4.00 11.00 4 11 Brooks FR 8 G437 11.34 3.00 6.00 3 6 Ho-Kai-Ming FR 9 G068 10.45 3.00 5.50 3 6 Jones-NewFold FR 10 G001 9.61 5.00 8.00 5 8 Sam-T02-human FR 11 G067 9.19 4.00 9.00 4 9 Jones FR 12 G427 9.04 5.00 10.00 5 10 Fischer FR 13 G045 9.00 4.00 9.00 4 9 PMODEL3 FR 14 G028 8.43 5.00 6.00 6 7 Celltech FR 15 G224 8.09 5.00 10.00 5 10 3D-SHOTGUN-3DS5 FR 16 G183 7.86 4.50 7.50 5 8 ARBY-SCAI FR 17 G223 7.29 4.00 10.00 4 10 3D-SHOTGUN-3DS3 FR 18 G096 7.29 3.00 10.50 3 11 Bates-Paul FR 19 G110 7.11 3.00 8.00 3 9 Honig FR 20 G020 6.79 5.00 7.00 5 7 Bujnicki-Janusz FR 21 G046 6.73 4.00 9.00 4 9 PCOMB FR 22 G222 6.72 4.00 10.00 4 10 3D-SHOTGUN-INBGU FR 23 G214 6.50 3.00 7.00 3 7 Advanced-Onizuka FR 24 G012 6.33 5.00 11.00 5 11 ORNL-PROSPECT FR 25 G517 6.25 4.00 11.00 4 11 GeneSilico FR 26 G368 5.86 3.00 7.00 3 7 Jose FR 27 G464 5.60 2.00 6.00 2 6 Atome FR 28 G476 5.50 3.00 7.00 3 7 123d-server FR 29 G423 5.28 2.00 8.00 2 8 Taylor FR 30 G450 4.82 2.50 8.00 3 10 Tome FR 31 G078 4.47 2.00 7.00 2 7 Rost FR 32 G041 4.39 2.00 9.00 2 9 Cbrc FR 33 G537 4.34 1.00 5.00 1 5 Nec-asogawa FR 34 G435 4.30 3.00 7.00 3 7 Fujita FR 35 G417 3.72 3.00 6.00 3 6 Cbsu FR 36 G288 3.68 3.00 9.00 3 9 Lomize-Andrei FR 37 G242 3.59 3.00 8.00 3 8 Genesilico.pl-servers-on1 FR 38 G265 3.54 2.00 9.00 2 9 Sasson-Iris FR 39 G447 3.54 2.00 7.00 2 7 Cam-Biochem FR 40 G039 3.54 1.00 6.00 1 6 PCONS3 FR ----------------------------------------------------------------------------- FR NgNW is the number of good predictions without weighting for multiple models. FR NpNW is the number of total predictions without weighting for multiple models. FR All-uppercase names are of CAFASP registered servers. NF New Folds (targets manually assessed by Rob Russell) NF ==================================================== NF 129 149_2 161 162_3 181 146_1 146_2 146_3 172_2 173 186_3 187_1 170 NF Use all the models weighted to have a total weight of 1.0 NF NF Total Z-Scores Above 1.0 for CASP5 All Targets Listed Above: NF ----------------------------------------------------------------------------- NF Rank Group Z-Score Ngood Npred NgNW NpNW Group-name NF ----------------------------------------------------------------------------- NF 1 G002 25.72 9.33 12.50 47 63 Baker NF 2 G349 17.57 8.25 10.00 14 17 Shortle NF 3 G132 13.46 5.00 10.00 5 10 I-sites/Bystroff NF 4 G010 11.78 5.69 11.51 29 59 Skolnick-Kolinski NF 5 G001 11.31 5.53 11.33 20 38 Sam-T02-human NF 6 G016 10.50 6.70 9.30 26 36 Levitt NF 7 G068 10.39 5.40 7.00 27 35 Jones-NewFold NF 8 G153 9.07 4.00 6.00 4 8 Chimera NF 9 G450 7.64 4.03 9.60 13 29 Tome NF 10 G029 7.59 4.50 11.80 28 58 BAKER-ROBETTA NF 11 G045 7.59 4.75 10.15 15 41 PMODEL3 NF 12 G051 7.29 3.80 9.27 19 49 SAMUDRALA-NF NF 13 G517 7.08 2.75 8.50 6 14 GeneSilico NF 14 G140 7.04 3.60 8.30 18 44 PROTINFO-AB NF 15 G040 6.94 4.93 11.67 22 49 PMODEL NF 16 G453 6.81 3.50 10.50 4 11 Ginalski NF 17 G373 6.57 3.95 11.30 16 46 Brooks NF 18 G020 6.40 3.50 5.00 4 6 Bujnicki-Janusz NF 19 G112 6.05 4.77 9.67 13 30 Friesner NF 20 G437 5.94 3.92 9.17 9 18 Ho-Kai-Ming NF 21 G475 5.84 3.00 4.00 3 4 Bionomix NF 22 G170 5.74 4.00 9.00 4 9 Chimerax NF 23 G006 5.39 3.00 11.00 3 12 BIOINFO.PL NF 24 G531 5.17 2.80 5.60 14 24 Kias NF 25 G067 5.02 3.00 7.50 3 8 Jones NF 26 G099 4.83 2.00 3.67 6 10 Camacho-Carlos NF 27 G224 4.45 3.00 8.00 3 8 3D-SHOTGUN-3DS5 NF 28 G314 4.31 2.40 4.00 12 20 Scheraga-Harold NF 29 G427 4.18 2.75 8.00 4 15 Fischer NF 30 G105 4.10 3.00 9.00 3 9 Sternberg NF 31 G084 3.88 2.83 11.50 5 18 Sbc NF 32 G429 3.68 2.20 3.50 11 19 Keasar NF 33 G096 3.50 3.00 9.00 3 9 Bates-Paul NF 34 G423 3.37 2.00 4.67 2 6 Taylor NF 35 G214 3.34 2.50 8.50 5 15 Advanced-Onizuka NF 36 G288 3.31 2.00 4.00 2 4 Lomize-Andrei NF 37 G516 3.22 2.50 4.50 4 6 Burnham NF 38 G464 3.18 2.20 8.03 9 36 Atome NF 39 G265 3.10 2.00 8.00 2 8 Sasson-Iris NF 40 G039 3.07 2.28 8.07 7 35 PCONS3 NF ----------------------------------------------------------------------------- NF NgNW is the number of good predictions without weighting for multiple models. NF NpNW is the number of total predictions without weighting for multiple models. NF All-uppercase names are of CAFASP registered servers. AL All Models (targets manually assessed by one of the three assessors) AL ==================================================================== AL 129 149_2 161 162_3 181 146_1 146_2 146_3 172_2 173 186_3 AL 187_1 170 130 132 133 136_1 136_2 140 141 142 143_1 AL 143_2 149_1 150 151 152 153 155 159_1 159_2 160 165 AL 167 168_1 168_2 169 172_1 176 177_1 178 179_1 179_2 182 AL 183 184_1 184_2 185_1 186_2 185_3 188 189 191_2 193_2 195 AL 134 138 156 157 174 193_1 135 147 148 162_1 162_2 AL 187_2 191_1 AL Use the first model for all the targets except those in the NF category. AL AL Total Z-Scores Above 1.0 for CASP5 All Targets Listed Above: AL ----------------------------------------------------------------------------- AL Rank Group Z-Score Ngood Npred NgNW NpNW Group-name AL ----------------------------------------------------------------------------- AL 1 G453 75.94 38.50 64.50 39 65 Ginalski AL 2 G020 59.00 33.00 50.50 35 53 Bujnicki-Janusz AL 3 G002 57.76 25.33 48.88 80 127 Baker AL 4 G010 57.59 25.79 49.89 58 116 Skolnick-Kolinski AL 5 G517 55.07 32.25 59.00 36 65 GeneSilico AL 6 G006 52.74 28.00 61.00 28 62 BIOINFO.PL AL 7 G110 42.82 27.00 45.00 27 47 Honig AL 8 G427 42.38 27.17 56.33 31 72 Fischer AL 9 G450 37.35 21.03 53.65 33 84 Tome AL 10 G029 36.14 23.10 56.20 50 116 BAKER-ROBETTA AL 11 G153 35.65 23.00 49.00 23 52 Chimera AL 12 G012 35.64 25.30 54.80 37 107 ORNL-PROSPECT AL 13 G224 34.62 23.00 45.00 23 45 3D-SHOTGUN-3DS5 AL 14 G028 33.72 23.50 47.30 41 88 Celltech AL 15 G265 32.39 19.50 49.00 20 50 Sasson-Iris AL 16 G223 32.39 20.00 46.00 20 46 3D-SHOTGUN-3DS3 AL 17 G349 31.38 15.25 17.00 22 25 Shortle AL 18 G067 30.56 19.00 47.00 20 49 Jones AL 19 G096 30.49 19.50 51.00 21 55 Bates-Paul AL 20 G041 30.31 15.60 42.70 22 62 Cbrc AL 21 G001 29.62 15.48 33.58 36 75 Sam-T02-human AL 22 G100 27.82 16.40 34.90 20 44 Sbi AL 23 G222 26.85 18.03 42.93 24 63 3D-SHOTGUN-INBGU AL 24 G112 26.80 17.63 54.37 31 85 Friesner AL 25 G329 25.12 15.00 36.00 15 36 Dunbrack AL 26 G464 25.09 11.00 37.43 25 79 Atome AL 27 G040 24.85 17.53 54.27 41 110 PMODEL AL 28 G448 24.80 16.00 23.00 16 23 Murzin AL 29 G475 23.78 11.00 29.00 11 29 Bionomix AL 30 G045 23.37 14.75 48.55 29 97 PMODEL3 AL 31 G068 22.51 9.50 12.50 41 53 Jones-NewFold AL 32 G425 21.34 12.00 17.00 12 17 Venclovas AL 33 G373 20.97 8.15 39.50 24 93 Brooks AL 34 G170 20.83 15.00 43.00 15 43 Chimerax AL 35 G368 20.79 15.00 49.50 15 50 Jose AL 36 G016 18.66 10.10 36.10 35 75 Levitt AL 37 G084 18.65 12.83 49.00 15 58 Sbc AL 38 G417 18.46 11.80 29.40 15 37 Cbsu AL 39 G437 18.38 7.92 17.17 13 27 Ho-Kai-Ming AL 40 G288 17.86 11.00 40.00 11 40 Lomize-Andrei AL ----------------------------------------------------------------------------- AL NgNW is the number of good predictions without weighting for multiple models. AL NpNW is the number of total predictions without weighting for multiple models. AL All-uppercase names are of CAFASP registered servers.