Category Archives: Research

Reinventing Discovery

I highly recommend the book (published in 2011, but I have only just read it – it’s hard to be on the cutting edge) Reinventing Discovery by Michael Nielsen. He “wrote this book with the goal of lighting an almighty fire under the scientific community”. His overview of Open Science, of which Open Access to publications is just one component, is very compelling and optimistic, without losing sight of difficulties.

Submission to the Electoral Commission Review of MMP

I missed the first deadline for proposals for submissions to the review, but now that the Proposals Paper has been released, it has focused attention on a smaller number of issues. With Michael Fowlie (current COMPSCI 380 project student) I have made a submission based on simulations of what we hope are “realistic” elections. We find that the party vote threshold should be lower than the 4% recommended by the commission. I have been told by the EC that our submission will be an appendix to their report due out on 31 October. It will be interesting to see a) their recommendations b) whether they lead to any actual change.

Addendum: our submission appears as Appendix D in the commission’s final report to Parliament. They went with the 4% recommendation in the end.

Open access update

There is a lot of new material out there, and some older stuff I hadn’t yet seen. These may be useful.

Division of labour in prepublication peer review

It seems to me to be a good idea to separate out the traditional refereeing (pre-publication review) functions. In mathematics at least, a paper should be “true, new and interesting”. It is often easier to check the first rather than the second, especially for less experienced researchers.  It makes sense for more experienced researchers to be asked for a quick opinion on how interesting and new a paper is, while more junior ones check correctness. This has some other advantages: if it becomes widespread, authors will have an incentive to write in a way that is understandable to new PhDs or even PhD students, which will probably improve exposition quality overall. It would reduce the load on senior researchers (I received an email yesterday from a colleague who said he had just received his 40th refereeing request for the year!) Doing a good job as a junior researcher could lead to a good CV item, so there would be an incentive to participate. Some sort of rating of reviewers will probably need to be undertaken: just as with papers that pass “peer review”, postpublication feedback from the whole community will be involved.

Peer review

I intend to present ideas (mostly not my own) about how to improve the current peer review system. This is a background post.

What is the purpose of peer review of scholarly publications?

  • Certification of correctness of the work
  • Filtering out work of low interest to the research community, to allocate attention more efficiently
  • Improving the quality of the work

Michael Eisen (among others) has argued that the current system is broken. Michael Nielsen debunks three  myths about scientific  peer review. Daniel Lemire has several interesting posts, including: the perils of filter-then-publish, peer review is an honor-based system.

Certification is still important, and very discipline-specific. In (parts of?) physics it seems to be a fairly low standard: not obviously wrong. The journal PLoSOne seems to check more rigorously for correctness, but is very relaxed on significance (see here). Mathematics journals I have experience with seem to be more finicky, and traditional journals with a high reputation are much tougher in assessing significance, often rejecting without looking at the technical details.

It seems clear to me that improvements in the current system are sorely needed. Excessive attention to whether work is “interesting” risks reducing science to a popularity contest, and there are too many boring but correct papers to read. Who has time to help others improve their work, if refereeing is anonymous and there is so much pressure to publish yourself?

 

Better citation indices

Daniel Lemire and colleagues are aiming to find a better algorithm to measure importance of research articles by incorporating the context in which the citation is made (for example, distinguishing between “courtesy citations” inserted to placate referees and real pointers to important work). They need some data and it looks like a low burden for each researcher to provide it. Check out this site  for more.

I think we have passed the point of no return with bibliometrics in evaluating researchers and articles. They will be used, so it is to our benefit to ensure that less bad ones are used.

Beyond the boycott

The boycott against Elsevier has been interesting so far. I have had some discussions with colleagues, most of whom have not signed. I am still struggling to detect any principled reason: worry about destroying the “brand” of the journal (from members of an editorial board of an Elsevier journal) is the only one I can sympathize with, although the fact (opinion, at least) that Elsevier has subtracted value from these brands after buying them from other publishers should be noted. Very few people have stated their reasons for not signing – Ariel Procaccia is a welcome exception. I suspect many researchers don’t give the issue more than a passing thought, others are too timid, and others like a free ride. Perhaps a few really think the current system is good, but in that case I would have to question their fitness for research work. The main feedback I have had informally is that it is all too hard, since they own the “best” journals in my field, I am an editor of one of their journals and I like it, etc. One interesting opinion is that although it is a problem, it will soon be fixed by advances in technology. I wish I could be confident of that.

In any case Elsevier has made some grudging concessions, so I guess that this will mean fewer people feel pressure to sign up. However, researchers shouldn’t have to waste their time to get such small concessions. The current system is clearly unsustainable and it seems almost impossible to imagine the commercial incentives of a company such as Elsevier ever allowing them to do what is right for society, or science/scholarship as a whole. Thus alternatives must be explored, and now is an important time for discussion. One forum for such discussions is Math2.0, which I read often and recommend highly.

Separating out the many related issues takes time and I will write several posts. For now, I have some concrete recommendations for mathematical researchers, none of them original to me. Many take very little effort.

  • Sign the boycott petition, or at least don’t work for journals that are exploiting free labour to make large profits.
  • Practice self-archiving rigorously. Use the arXiv and update your paper to the final accepted version. List all your papers on your own webpage. Encourage all colleagues to do the same – if you want to read their paper and can’t find it on their website, ask them to put it there.
  • Familiarize yourself with the policies of the publisher on author self-archiving. Stand up for your rights as an author (see this important article by Kristine Fowler in Notices of the American Mathematical Society).
  • When citing a journal publication, also give the arXiv version of the paper (if they are essentially the same).
  • Encourage those involved in hiring and evaluation at your institution to ignore journal impact factors and use article-level metrics and other more nuanced measures.
  • Encourage granting agencies you deal with to require open access to all publications associated with grants they award.
  • If you are on an editorial board of a journal run by Elsevier or the like, talk to your co-editors about moving to another publisher with better practices, or at least registering your displeasure to the current publisher and tell them they need to change if they want to keep you. Discuss with other editors of other journals, and share approaches that work.
  • Find out what your professional society (AMS, IMU, …)  is doing about these issues, and whether its publishing arm is helping the cause of open access, or harming it. Get involved on committees in the organization where possible.
  • Talk to your institution’s librarians. Find out what they can offer in terms of institutional repositories, hosting journals, etc.

 

 

 

2011 referendum simulator: experience so far

Several months ago I realized that the 2011 referendum in NZ on the voting system for parliamentary elections was coming soon. Geoff Pritchard and I developed a simulator with the aim of enabling voters to understand the consequences of a change to another system. In order to do this in a way that is useful to the non-expert, some simplifying assumptions must be made. We had substantial media coverage and some criticism.

Initial surprises:

  • How few people bothered to read the detailed FAQ before criticizing.
  • How many people thought that the simulator was trying to “back-cast” historical elections, and were certain that our results were unrealistic, without giving any evidence.
  • How much the criticisms, even unfounded ones, helped to clarify my understanding of what we had actually done, and suggested further research.
  • How short the attention span of internet visitors is.

In particular I want to respond to comments by David Farrar on his well-known site Kiwiblog. The relevant extract:

Now in 2002 National actually won 21 electorate seats out of 69 or 70. So this model is saying if there were 50 extra electorate seats, National would win 11 fewer seats!!

Why? Because they have come up with a formula based on the last 50 years or so of FPP elections, which they applied to the party vote figures for 2002. They ignored the actual electorate vote. It is a classic academic approach.

The more pragmatic approach, which is what others have done, is to say well if National won 21 electorate seats in 2002 out of 70, then if there 120 seats, their estimated number of seats would be 21*120/70, which is 36 seats.

In fact we did not look at any of the historical FPP elections The “formula” is based completely on the MMP party vote from the 2008 election (so yes, we did ignore the electorate vote, for what we think are good reasons).

However this got me thinking about how we might try to validate our assumptions. One way which I don’t (yet) claim is rigorous, but makes at least as much sense as the above, is to apply the simulator (the FPP part) to the historical FPP elections, and scale the 120 seats down to whatever it was historically (80 for many years, then increasing over time). The results surprised me greatly, as they are much better than expected, and this cries out for explanation (“further research”, always good for academics). Here they are. Note that these simulator results explicitly do not use any historical data, seat boundaries and parties have changed, etc.

1969: Real result was Nat 45 Lab 39; simulator scaled was Nat 46.9, Lab 37.1
1972: Real was Nat 55 Lab 32; simulator scaled was Nat 54.4, Lab 31.6
1975: Real was Nat 55 Lab 32; simulator scaled was Nat 55.1, Lab 31.9
1978: Real was Nat 51  Lab 40 SoCred 1; simulator scaled was Nat  47.5, Lab 44.5.
1981: Real was Nat 47 Lab 43 SoCred 2; simulator scaled was Nat 48.3, Lab 43.7
1984: Real was Nat 37 Lab 56 SoCred 2; simulator scaled was Nat 37 Lab 58.
1987: Real was Lab 57, Nat 40; simulator scaled was Lab 50.2, Nat 46.8
1990: Real was  Nat 67, Lab 29, NewLab 1; simulator scaled was Nat 71, Lab 26
1993: Real was Nat 50, Lab 45,  NZF 2, Alliance 2; simulator scaled was Nat 53.6, Lab 45.4