Systems Seminar – by John Thomson

All wecome.

The Results Delusion

Abstract:

It is often said that any subject which requires the word ‘science’ to be placed somewhere in its name, is unlikely to be very scientific. This is unfortunately far too true for systems research in general. Every systems conference, papers are presented which show significant speedups over previous approaches to problem X, but these improvements are rarely replicated in output from industry. Why? The unpalatable answer is that a significant amount of systems research is the result of self-delusion, bad science and, I suspect occasionally, fraud.

Standards of scientific rigour in CS often fall well below what would be taken for granted in other sciences – particularly with regard to measurement of results, statistical analysis and replicability of results. I would like to do something about this, and will be presenting the idea for a new CS journal, which focuses on this exact problem. Oh, and peer review is gone too! Pitfalls abound. Would love to hear your comments, objections and advice.