Experiments Within IRBs

At the July 28 JERHRE conference many excellent research questions were raised about the IRB process and how it might be improved. In the September 2006 issue of JERHRE, Robert J. Levine raises two more such questions: (1) Can continuing reviews of ongoing research be done as competently by expedited review as by full review? And, (2) is there a better alternative to reviewing all adverse events at a convened IRB meeting?

Is such research feasible? The investigator who hopes to visit IRBs and gather data from files, interviews, or observation of meetings is quickly overwhelmed by problems of getting approval and informed consent from many layers of stakeholders, including investigators who want to protect their intellectual property.

A better approach may be for IRBs that wish to assert leadership and improve their performance to collaboratively conduct their own within-IRB experiments. For any given experiment, each participating IRB would employ the same experimental procedures. The results from each IRB would be pooled in some methodologically sound way. Both quantitative and qualitative data would be obtained. The work involved for each IRB might be no more complex than normal quality assurance activities. However, the pooled results would provide a robust evaluation of the innovation, and successful innovations would be published for other institutions to emulate. The following are a few preliminary thoughts:

Research designs . The following are some designs that may be appropriate:

Parallel comparison of innovation with old practice . For regulatory and control purposes, IRBs may not want to eliminate the old practice while experimenting with a new one. For example, Bob Levine has proposed that the findings of an expedited reviewer be compared with the findings of the full committee review for a period of perhaps 6 months to a year. Likewise Bob recommends that one experienced IRB member review all non-serious and anticipated adverse events and compare these findings with the full committee review to determine whether his recommended procedure allows any noteworthy events to slip through.

Pre- post-comparison designs : As part of many experiments, including parallel comparisons, the IRB may take baseline measures of researcher satisfaction with IRB performance, and IRB member satisfaction with the workload and the way IRB meetings are conducted. These comparisons would be made by measuring satisfaction before the innovation is begun, and, perhaps in a time series, on a monthly basis for 6 months or a year.

Within IRB-member, experimental and control design: There are many innovative ways to conduct IRB reviews (e.g., having the relevant regulations handed out or projected on a screen as each protocol is discussed; or having expert consultation on protocols for obscure research topics). Within-member comparisons might involve having protocols randomly assigned to the new or old method of review; IRB members would compare their reactions to reviewing protocols with the innovation to their reactions to similar other protocols reviewed in the old way. In addition to IRB member ratings, other measures might be employed such as amount of time taken to arrive at a satisfactory resolution, and researcher satisfaction with the decision (e.g., how competent and well reasoned they think it was, fairness, how helpful the IRB was to the investigator, and how much time was taken -- from submission of protocol to letter to PI -- to arrive at the decision).

Static-group comparison : In this design, an IRB which has experienced an innovation is compared with one which has not, on various relevant measures.

A host of other experimental and quasi-experimental designs may prove useful depending on the particular innovation to be studied (Campbell & Cook, 1979).

There are a number of factors that can threaten the validity of field research such as this, including “history” or extraneous events that happen to coincide with the innovation, such as a major increase or decrease in workload, or resignation of key IRB member, “mortality” such as some of the less enthusiastic IRBs dropping out of the study, “reactive or Hawthorn” effects such as feeling very enthusiastic about being an innovative IRB. For these reasons, it would seem important that each participating IRB carefully note any extraneous factors likely to affect the results, that any study be conducted over a period of perhaps a year with time-series data, and that consideration be given to stratifying participating IRBs along relevant factors such as size, workload, or nature of protocols reviewed.

Some Suggested Research Topics to Define and Study:

• How risks are assessed
• How word-smithing concerns are handled
• Whether outside consultants are used; how they are found; how they perform
• Approaches to working with investigators
• How expedited reviews are handled
• How exemptions are handled
• Ability to identify which risks are serious and which not, and to handle accordingly
• How difficult-to-review methods are handled (e.g., ethnography)
• How difficult departments are handled; how student protocols are handled
• How IRB members are recruited and educated
• Membership turnover and how it is managed
• How faculty and students are educated about research ethics
• Website features as aids to the IRB and investigators

Some Suggested Dependent Measures:

• Protocol turn-around time
• Expertise of board in relation to types of protocols reviewed
• Effectiveness and collegiality of problem solving by IRBs with investigators
• IRB member satisfaction
• PI satisfaction (e.g., use of the IRB Researcher Attitude Test)
• Allocation of time to functions at IRB meetings
• Results of quality checks of protocols

Specifically, Dr. Levine asks: would it be as safe and effective to have: a) prompt reporting to the DSMB of all adverse events that are both serious and unanticipated; b) prompt reporting to the local IRB of adverse events that are both serious and unanticipated only if they occur locally (in multi-site clinical trials) and c) routine reporting (usually annually) to the local IRB and DSMB of events that are not both serious and unanticipated. [See Levine, R.J.(2006.Empirical research to evaluate ethics committees' burdensome and perhaps unproductive policies and practices: A proposal. Journal of Empirical Research on Human Research Ethics, 1 (3), 1-4.]