Automated tracking of scientific computations - Andrew Davison
Reproducibility of experiments is one of the foundation stones of science. A related concept is provenance, being able to track a given scientific result, such as a figure in an article, back through all the analysis steps (verifying the correctness of each) to the original raw data, and the experimental protocol used to obtain it. In computational, simulation- or numerical analysis-based science, reproduction of previous experiments, and establishing the provenance of results, ought to be easy, given that computers are deterministic, not suffering from the problems of inter-subject and trial-to-trial variability that make reproduction of biological experiments, for example, more challenging. (see abstract for more details: http://www.stodden.net/AMP2011/)
To ensure reproducibility of a computational experiment we need to record: (i) the code that was run, (ii) any parameter files and command line options, (iii) the platform on which the code was run, (iv) the outputs. To keep track of a research project with many hundreds or thousands of simulations and/or analyses, it is also useful to record (i) the reason for which the simulation/analysis was run and (ii) a summary of the outcome of the simulation/analysis. Recording the code might mean storing a copy of the executable, or the source code (including that of any libraries used), the compiler used (including version) and the compilation procedure (e.g. the Makefile, etc.) see abstract for more details...
In this talk I will present the solution we are developing to the challenges outlined above. Sumatra consists of a core library, implemented in Python, on which is built a command line interface for launching simulations/analyses with automated recording of provenance information and a web interface for managing a computational project: browsing, viewing, and annotating simulations/analyses.
Sumatra (i) interacts with version control systems, such as Subversion, Git, Mercurial, or Bazaar, (ii) supports launching serial or distributed (via MPI) computations, (iii) links to data generated by the computation, (iv) aims to support all and any command-line drivable simulation or analysis program, (v) supports both local and networked storage of information, (vi) aims to be extensible, so that components can easily be added for new version control systems, etc., (vii) aims to be very easy to use, otherwise it will only be used by the very conscientious.