What metrics should we implement for ranking papers?

(1) Download popularity

(2) Response levels (i.e. numbers of comments and reviews, weighted by length / level of substance)

(3) Average rating (1-10) by reviewers/commenters (weighted by meta-ratings)

(4) Average rating (1-10) by all readers (unweighted). [Disputed below]

[Edit this page to add further suggestions, or discuss details below.]

Q: Should we allow anonymous ratings of papers (i.e. without having to leave a comment/review to explain the rating), as per proposal #4?

cons - Doesn't allow for meta-ratings. Less transparency = more likely to be abused. No advantage over scale #3?

Q: Should meta-ratings ("How helpful was this review?") be a binary +/- option, rather than a similar 1-10 scale?

pro - simpler.

con - less fine-grained (but may not matter so much here).

Q: Should meta-ratings be anonymous?

pro - simpler.

Q: Is it possible to include a 'citation count' to track whether the paper is discussed/cited elsewhere?

A possible argument: To the extent that would be feasible on this level, it may be redundant; depending on exactly what sort of discussion and citation is in view, there are often instruments already existing that might serve (Technorati, Google Scholar, etc.). It might, however, be useful to include a set of such links on the site (e.g., on the front page or on some sort of sidebar).

 Create your own free wiki!HelpPublic, not yet premium Privacy Policy RSS