Error Estimates for Solid-State Density-Functional Theory Predictions: An Overview by Means of the Ground-State Elemental Crystals

K. Lejaeghere, V. Van Speybroeck, G. Van Oost & S. Cottenier Critical Reviews in Solid State and Materials Sciences 2014, 39, 1-24
Contributed by David Bowler
Reposted from Atomistic Computer Simulations with permission

The question of how to characterise the accuracy of a computer code is a difficult one, and I have touched on these issues before (here, here and here for instance). However, given the large number of codes available, it should be possible to compare them to each other and to experiment (or higher level calculations) to test them. This is a well-established process in the quantum chemistry community, where there are various test sets for different properties, including enthalpies of formation(G97/2)[1], weak bonding (S22)[2] (and a vast database of different properties[3]).

A recent paper[4] and associated web site[5] offers a first approach for solid state codes, with the comparison based on the differences between all-electron and pseudopotential calculations. The presumption here is that all-electron calculations are the touchstone (though the website notes that the all-electron results have been refined to use extremely accurate tolerances and small muffin-tin radii, so there is clearly room for improvement in any method).

The idea of the comparison is to calculate binding energy curves for most elements in the periodic table, and from these curves to derive a single number which characterises the deviation from all-electron results. The deviation per element can be viewed, as can the deviation from experiment for the all-electron calculations. The deviation, delta, is defined by an integral over all calculated values of volume, and thus includes implicitly both the lattice constant and the bulk modulus, though not the cohesive energy. Most results shown on the website are for plane-wave codes, which generally perform rather well (the old norm-conserving FHI pseudopotentials are less accurate, though, and should be treated with care).

The approach is a good one, though a little heavy on the tester, and scripts to perform the necessary calculations are made freely available. However, the choice of the elemental form makes the tests rather restricted: there is no way to examine different types of bonding or different oxidation state, for instance. It is quite easy to imagine developing a set of these test suites for different purposes in solid state codes, just as there are different test sets in quantum chemistry.

The accuracy tests seem to be most illuminating for the pseudopotentials, rather than the codes themselves, and I think that it would be of immense value to the community if the pseudopotential generation details were made available. This should not just include the core radii and the reference configurations, but also a clear description of the pseudopotential algorithm (or an appropriate reference along with details).

There is something of a danger of using codes and supplied pseudopotential libraries as black boxes: there is a need to test parameters, though it’s rare (and I am happy to acknowledge that I don’t do it as much as I should). This paper and associated developments should go some way to standardising plane wave codes, and giving quantitative information on their reliability.

Note As I was writing this, a new paper in Science has just been published which takes a different view, and compares results from different functionals; it’s worth a read[6]

[1] J. Chem. Phys. 106, 1063 (1997) DOI:10.1063/1.473182
[2] Phys. Chem. Chem. Phys. 8, 1985 (2006) DOI: 10.1039/B600027D
[3] http://t1.chem.umn.edu/db/ and see arXiv http://arxiv.org/abs/1212.0944
[4] Crit. Rev. Sol. Stat. Mat. Sci. 39, 1–24. DOI:10.1080/10408436.2013.772503
[5] http://molmod.ugent.be/deltacodesdft
[6] Science 345, 197 (2014) DOI:10.1126/science.1253486

Confidence limits, error bars and method comparison in molecular modeling. Part 1: The calculation of confidence intervals

Anthony Nicholls Journal of Computer-Aided Molecular Design 2014 (Open Access)
Contributed by +Jan Jensen

Almost every computed number we report has an uncertainty and ...

... without an assessment of this uncertainty, or a description of how to estimate it, what we have really delivered is a report, not a prediction; “we did X, followed by Y, and got Z”. 

Of course we all know this and we faithfully report RMSD values, Pearson's correlation coefficient ($r$) and other measures of uncertainty.  However, when was the last time you saw an uncertainty attached to these quantities? In others words, how likely is it that a future study would compute the same RMSD value for a different set of experimental values using my method? Or, my $r$ value looks great but do I have enough data points?

This wonderful and very readable paper tells you how to compute the uncertainty in your uncertainties and what they mean. There will be a follow-up paper that will describe how meaningfully compare quantities for which such uncertainties have been computed. I can't wait.


This work is licensed under a Creative Commons Attribution 4.0 International License.

Numerical Errors and Chaotic Behavior in Docking Simulations

Miklos Feher and Christopher I. Williams, Journal of Chemical Information and Modeling, DOI: 10.1021/ci200598m (paywall)

The authors have made a very interesting study of the effect of numerical problems in molecular docking. Specifically, they have studied the two docking software GOLD and GLIDE, both with three different accuracy settings.

As the authors point out in the conclusion:  
"This study clearly demonstrates that seemingly insignificant differences in ligand input, such as small coordinate perturbations or permuting the atom order in an input file, can have a dramatic effect on the final top-scoring docked pose."

The authors have investigated how robust the docking algorithms are with respect to poses and scores for two different input variations. First, very small variations to the input structure (max 0.1 degree of torsional angle changes, total max 0.1 Å RMSD). Second, permutations of atom order in the ligand input files. Both are variations that the normal user would expect to have no effect on the final output of a docking run.

The results are highly interesting and show that the virtual screening settings in the software seem to correlate with low robustness. Hence, using standard settings (or higher) improves robustness for these two methods.
While one could expect that GLIDE which is a deterministic approach (empirical scoring function) perhaps would be less prone to robustness problems than GOLD (genetic algorithm), this does not seem to be the case. The only major difference between the two softwares is that GOLD seem to generate more normal based distributions of scores and RMSDs than GLIDE.

Both small changes in the torsions of the input structures and the permutation of atom order in the input files lead to many cases where the docking protocols simply fail to be robust and can result in vastly different top scoring poses, especially for the low accuracy settings. Thus, applications of docking software with virtual screening settings are more prone to problems with reproducibility, but even with the highest accuracy robustness cannot be guaranteed.

For the future, it would be highly interesting to see if these problems are the same for other docking software, and how they can be alleviated.

[edit: changed the comment on CPU time relative to robustness, since there is no statistical significant difference of the robustness of the two more accurate settings used in the two software]