4.5 Guidance for those assessing abstracts

4.5.1      Guidelines for reviewers

Each abstract is randomly assigned to four reviewers.  All members of the SSM Committee and appropriate members of the ASM Committee are expected to review abstracts; other academics and researchers at the host institution are also invited to review abstracts.  The reviewers are given the following guidelines, which are also made available on the abstract submission site to those wishing to submit an abstract:

General

Abstracts will be assessed by at least four reviewers, independently, and the final score derived.[1]  Reviewers will remain masked to the authors and their institution throughout.  Abstracts submitted from the same institution as a reviewer will not be assessed by that reviewer and subject to this constraint, abstracts will otherwise be allocated randomly to reviewers.  The scoring system ranges between 0 and 10, with zero given to abstracts not compliant with the Guidelines for Abstraction Submission and will be rejected at this stage; this includes abstracts not written in English, not written in a structured abstract style as appropriate. The online abstract submission form has been set so that a structured abstract is now readily adhered to and it will not allow the author to exceed the word count of 400 (excluding title).

The top scoring abstracts submitted for oral presentations will be selected for the plenary and parallel sessions.  This will be based on the number of available slots at each conference.  For example, if there are five plenary sessions and 90 slots in the parallel sessions, amongst the 95 top scoring abstracts, 5 will be selected for plenary presentations and 90 will be selected for the parallel sessions.  For plenary sessions only, the Society committee will consider the extent that abstracts overlap to avoid two presentations from the same study or on very similar themes; from amongst the top scoring abstracts, 5 will be selected to ensure a varied programme at the plenary sessions.  The top 10 scoring abstracts not selected for plenary sessions will be highlighted as high scoring in the abstract booklet.

Reviewer scoring guidelines

JUDGING CRITERIA FOR SSM ABSTRACT REVIEWERS

Please score the abstract according to the following guidelines. Scores are to be awarded in two parts: 0-10 points for the abstract ‘structure & detail’ plus 0-5 points for ‘interest, appeal & impact’; reviewer variability in each score will be assessed. Only the 0-10 score will be used to rank the abstracts, though 0-5 score may be used inform decisions on tied scores or plenary selection amongst the top scoring abstracts.

Tables, charts or references should not be included in the abstract. The PICO format may be a helpful reference: http://www.usc.edu/hsc/ebnet/ebframe/PICO.htm.

Reviewers have the option to score ‘zero’ any abstract deemed to be seriously flawed; all zero scores will be reviewed by the SSM committee.

Title / Structured Headings / Overall Presentation: 0 – 1 point

Is the title specific, adequate and concise? Does it accurately describe the population studied, the study design or method of data collection or analysis, the research objective or question?

Introduction/ Objectives / Hypotheses or Research Question(s): 0 – 1 points

Is the context made clear? Is the scientific rationale clearly stated? Are the aims, objectives, hypotheses or research question(s) clearly stated?

Methods: 0 – 3 points

For all types of study, are the Methods clearly described? Are the data sources clearly specified? Are the methods, analytical techniques and software tools specified? Are the methods appropriate to the question being investigated?

For qualitative studies: Are qualitative methods appropriate to answering the research questions/addressing research objectives? Are the recruitment method(s), sample population(s), methods of data collection, and methods of data analyses described and appropriate?

For quantitative, observational experimental or modelling studies: Are the sample frame(s), sampling method(s), sample population(s), intervention and control conditions, methods of data collection, main outcome measures, assumptions and statistical methods all clear and appropriate?

For mixed methods: Is there appropriate use of quantitative/qualitative methods, each clearly described, in an order that makes sense, and each appropriately integrated at the right stage of the analysis/interpretation?

Systematic reviews: Should state objectives; data sources; study eligibility criteria, search strategy (e.g. data/text mining), participants, and interventions; study appraisal and synthesis methods (e.g. meta-analysis, meta-regression, narrative synthesis, meta-ethnography). Has the risk of bias and quality of the included studies been considered?

Results: 0 – 3 points

Are results available and described appropriately to be confident that sufficient material will be presented at the conference? Abstracts should not say only that ‘results will be presented’.

For quantitative, observational experimental or mixed methods studies: Do data presented give a clear indication of precision, favouring confidence intervals over p-values? Do modelling studies present sensitivity analyses?

For qualitative and mixed methods studies: Are the sample characteristics described? Are themes and/or categories presented systematically and meaningfully? Is the context in which data were produced recognised in the language used, for example, are data recognised as reported?

 

For mixed methods: Describe the data resulting from each method as well as integrated analyses.

Systematic reviews: Should report search results at each stage plus main outcomes.

Conclusions: 0 – 2 points

Are the conclusions clear and concise? Do they reflect the aims and objectives? Are they supported by the results presented? Are key study limitations acknowledged? Where appropriate, are the implications made clear for policy, practice and further research?

Interest, appeal and impact: 0 – 5 points

Is it interesting? Would it appeal to a broad ASM audience? Does it have the potential to create impact (i.e. change clinical or public health practice or policy, improve health, reduce inequalities in health, change the course of science)? Is it novel/exciting/much better methodologically than other studies in the area?

4.5.2      Moderation of marks

The scores are then collated and adjusted for inter-reviewer variation, using a method developed by Prof Mark Gilthorpe (Professor of Medical Statistics, Leeds University) (see Appendix D).

The scoresheet (with adjusted and unadjusted scores and ranks) is circulated, along with copies of the 10 highest scoring abstracts (adjusted scores) to the SSM Committee members prior to the April/May meeting, at which the committee selects:

  • three high scoring abstracts for oral presentation at a plenary session (because a range of topics, studies, and institutions is desirable, these are not necessarily the three highest scoring abstracts);
  • 90 (96 in 2012) abstracts for oral presentation during parallel sessions;
  • 60+ abstracts for poster presentations; and
  • the threshold below which abstracts should not be offered a presentation, should vacancies occur.

At the final plenary session of the ASM, the pre-PhD ECR and the post-PhD ECR who submitted the abstract with the highest score are each presented with vouchers (amount increased to £100 each in 2012), paid for by the Society.

 

 4.5.3      Preparing the programme

Deadlines: There is a timeline. For example, the provisional timeline for ASM 2013 was:

STAGE Action Restrictions on any change of dates Suggested Deadline for 2013 Responsible parties
1 Prepare abstract form. SSM to send updated guidelines, categories, award information, conference logos   Monday  7 January HG3 / ASM LOC
2 Finalise and set up abstract form   Monday 21 January HG3 / ASM LOC
3 Conference page: send website structure, colour scheme, SSM & Conference logos to the WebMaster   Monday 28 January ASM LOC
4 Get form live and link on conference web page   by Tuesday 5 February HG3 / ASM LOC
5 Send out email to all members of the Society advising them the abstract form is live     SSM National Committee
6 Send HG3 a list of peer reviewers’ contact details so that the reviewers website can be set up   Friday 1 March ASM LOC
7 Send updated peer reviewer guidelines     ASM LOC
8 Abstract Submission Deadline (expect 200+ abstracts) 4 weeks after the abstract form is live Tuesday 5 March  
9 Send abstract allocator (currently Honorary Treasurer) spreadsheet with submitting author’s ID No, Title, presenting author, co-authors and affiliations   Wednesday 6 March HG3
10 Receive randomised allocation of abstracts to reviewers from Honorary Treasurer   Thursday 7 March SSM National Committee
11 Allocate individual abstracts to reviewers and create  pdf’s   Friday 8 – Monday 11 March HG3
12 Send pdf and spreadsheet to webmaster so he can allocate abstracts on reviewer’s site   Friday 15 March HG3
13 Email reviewers with instructions as to how to log onto the reviewers website   Monday 18 March HG3
14 Reviewers to complete their scores on line   18 March-9 April Reviewers
15 Scores are collated and sent out to Honorary Treasurer   Friday 12 April HG3
16 Updated Registration form to HG3, bookings open in May   Friday 12 April (or earlier) ASM LOC
17 Honorary Treasurer to review the collated score sheets in preparation for the meeting   12 April – 22 April Honorary Treasurer
18 Meeting of the Society’s Committee to make abstract selection   Monday 22 April SSM National committee
19 Local committee Chair plus 1 or 2 others meet to put abstracts into session batches/themes   Friday 26 April ASM LOC
20 FREE PLACE APPLICATIONS DEADLINE – priority to those who have abstracts accepted; and those who have submitted   Friday 3 May  
21 Registration form goes live   Friday 10 May HG3
22 FREE PLACE applicants notified prior to letters being sent out (if giving registration deadline)   Before 13 May SSM National committee
23 Initial author letters are sent out advising submitting authors whether they have been selected or rejected,  asking them to register and pay by 31 May (if giving deadline)   week commencing 13 May HG3
24 Send updated guidelines for posters & speakers to HG3 (these will be sent out with confirmation letters)   week commencing 20 May ASM LOC
25 Send out author notification letters confirming oral presentation slots/poster numbers   week commencing 20 May HG3
26 Programme Files to be prepared for Printers   week commencing 27 May HG3
27 Prepare provisional oral programme, publish on website so authors can check when and where they need to attend   ideally by 24 May ASM LOC
28 DEADLINE for authors to register and pay – you need to chase up those who have not (if necessary extend the deadline) [remember to follow-up later on, amending the programme if some drop out]   Friday 31 May HG3 / ASM LOC
29 JECH Copy Deadline (give yourself an extra week)   11 July (for 17 July) – tbc ASM LOC
30 Chase Authors who have not registered   August HG3 / ASM LOC

 

Informing presenters: This is currently done by the professional conference organisers, HG3.

Preparing the programme: This is done by the ASM committee.

Preparing the abstracts for publication in a JECH supplement: This is done by HG3, under the guidance of the ASM committee.  See the timeline.  It is important that negotiations with JECH commence early in the calendar year, to agree a cost and the timings to ensure delivery of printed supplements in time for distribution to conference participants.

Penalties for withdrawal: None, apart from losing some or all of their registration fee if they have already registered (except where specific extenuating circumstances lead the ASM chair to decide this should be reimbursed.

Post ASM Evaluation – each year an online evaluation survey is conducted for those who attend the ASM. SSM has appointed a survey officer to do this. Findings are discussed at the January committee meeting.

[1] Either as arithmetic means, or obtained from a multiple-membership model; the latter adjusts for variation between examiners, though its application will depend upon the availability of a volunteer to perform this task.