SACHRP Recommendation on GAO-23-104721, INSTITUTIONAL REVIEW BOARDS: Actions Needed to Improve Federal Oversight and Examine Effectiveness

Approved October 19, 2023

Executive Summary

The Office for Human Research Protections (OHRP) requested that the Secretary’s Advisory Committee on Human Research Protection (SACHRP) provide a recommendation for identifying, measuring, and improving IRB effectiveness in protecting research participants in response to the General Accounting Office’s report entitled, INSTITUTIONAL REVIEW BOARDS: Actions Needed to Improve Federal Oversight and Examine Effectiveness (GAO Report 23-104721), released on February 16, 2023. The GAO report is an important step in the long history of federal government oversight of human subject research, which began in the 1950s.

In this document, SACHRP provides recommendations about how to prioritize further work in this important area but recognizes we may lack the comprehensive depth and breadth of expertise required to provide definitive answers to improving the Institutional Review Board (IRB) system. In the short term, SACHRP believes the best way to ensure and measure the effectiveness of IRBs is accomplished by:

1. Using existing and well-established governmental and non-governmental standards, including routine and for-cause inspections of IRBs and educational materials by OHRP, FDA, and federal agencies governed by the Common Rule.  There are written standards in place for conducting inspections.  Non-governmental standards include certification, accreditation and peer review or other processes for evaluating IRB effectiveness, as well as training programs for IRB members and staff. However, we recognize these are proxies for quality and effectiveness. Increased funding for inspections, peer review, and training programs can positively contribute to ensuring IRB quality and effectiveness. 
2. Convening stakeholder groups from the community, including evaluation experts.
3. Developing and sharing common definitions and novel tools to measure quality and effectiveness in the IRB ecosystem.

In addition, SACHRP believes that grant funding should be made broadly available for assessing the best methodologies and processes to measure IRB effectiveness.  Such funding should also encompass broader assessment of the effectiveness of Human Research Protection Programs (HRPP) in protecting the rights, interests, safety, and welfare of research participants.

Charge to SACHRP

Released on February 16, 2023, the GAO report entitled, INSTITUTIONAL REVIEW BOARDS: Actions Needed to Improve Federal Oversight and Examine Effectiveness, includes the following recommendation, which is listed as #4:

The Secretary of Health and Human Services should ensure that OHRP and FDA convene stakeholders to examine approaches for measuring IRB effectiveness in protecting human subjects and implement the approaches as appropriate. These could include effectiveness measures; peer audits of IRB meetings and decisions; mock protocols; surveys of IRB members, investigators, and human research participants; or other approaches.

To fulfill this recommendation, OHRP asked SACHRP to answer questions based on the following three charges:

1. What constitutes effectiveness in protecting research participants? This could be defined in terms of avoiding harms, ensuring subjects exercise informed consent, protecting subjects’ rights and welfare, treating subjects equitably or fairly, or achieving greater consistency in applying the regulation, or something else. Depending on what is being protected, the IRB’s actions could differ, and measures of effectiveness would vary accordingly. What definition of IRB effectiveness is the most important to focus on and measure?
2. SACHRP is one HHS “stakeholder”. What other stakeholder groups should HHS convene as part of examining approaches for measuring IRB effectiveness? What factors make an entity an appropriate stakeholder?
3. GAO provides several potential effectiveness measures. How do these approaches differ, and what are their benefits and limitations? What approaches should HHS and stakeholders prioritize? Are there other approaches should HHS and these stakeholders consider for measuring IRB effectiveness in protecting human subjects?

Historical Context

This current GAO recommendation is the most recent in a long history of United States government actions and inquiries focused on the protection of human research subjects.  In 1953, the Clinical Research Center at the National Institutes of Health issued the first US guidelines to protect medical research subjects.  In 1966, this intramural policy was broadened to be an intermural policy applying to institutions receiving funding from the agency.  This policy was formally converted to DHEW-wide regulatory status in 1974.  Congress also passed the National Research Act in 1974, which established the National Commission for the Protection of Human Research Subjects.  Based on the work of the National Commission, the human protection regulations were revised and adopted by additional agencies, such as the FDA.  In 1991, the human subject regulations were revised slightly and adopted by fifteen agencies across the federal government as the Common Rule.  In 1997, Congress had a series of hearings chaired by Christopher Shays on oversight of NIH and FDA, bioethics, and the adequacy of informed consent.  As a result of those hearings, the Office for the Protection of Research Subjects (OPRR) and FDA increased their scrutiny of IRBs and suspended or took other related actions against several of them.  In 1998, the HHS Office of the Inspector General issued four reports on the IRB system, including a central report entitled “Institutional Review Boards: A Time for Reform.” In 2009, the GAO issued a report entitled “Undercover Tests Show the Institutional Review Board System Is Vulnerable to Unethical Manipulation”.  

In addition, the federal government has supported a series of advisory committees on human research protections after the 1974 National Commission, including the Ethics Advisory Board (EAB) 1978-80, the President’s Commission for the Study of Ethical Problems in Medicine and Biomedical and Behavioral Research 1980-83, the President’s Advisory committee on Human Radiation Experiments (ACHRE) 1994-95,  the National Bioethics Advisory Commission (NBAC) 1996-2001, the National Human Research Protections Advisory Committee (NHRPAC) 2000-03, and this current committee, the Secretary’s Advisory Committee on Human Research Protection (SACHRP) 2003-present.  There were also non-governmental responses to the increased agency scrutiny in the last thirty years, such as the creation of both the Association for the Accreditation of Human Research Protection Programs (AAHRPP) accreditation, Public Responsibility in Medicine and Research (PRIM&R) Certified IRB Professional (CIP) certification, and the Collaborative Institutional Training Initiative (CITI) for training investigators and other research staff. More recent non-governmental efforts include the Consortium to Advance Effective Research Ethics Oversight (AEREO).

It is within this long tradition that SACHRP considers the OHRP charge to provide commentary on the GAO’s fourth recommendation.  Whereas past government actions and inquiries have focused on establishing the administrative structure and substantive framework for the ethical oversight of research, with an emphasis on regulatory compliance, the current GAO recommendation #4 stresses assessment of the efficacy of IRBs and the IRB system.  To that end, SACHRP recommends both short term and aspirational goals.  In the short term, SACHRP believes that the current regulatory oversight system should be enhanced by increased funding for inspection purposes to OHRP, FDA, and all of the agencies that have adopted the Common Rule.  Furthermore, non-governmental mechanisms that have been developed, including certification, accreditation and other peer review or equivalent evaluation processes for evaluating IRB effectiveness, as well as training programs for IRB members and staff, should be reviewed and considered for improving IRB effectiveness.  In the longer term, there should be focused government inquiries and grants devoted to determining how best to measure the effectiveness of IRBs, of Human Research Protection Programs (HRPP), and public knowledge, trust, and attitudes toward these systems, including the best approach to making these assessments.

Scope of Recommendation

The conduct of research involving human subjects operates in a complex environment involving many individual and organizational roles, all of which have a shared responsibility to protect the rights and welfare of human subjects in research.  Consequently, there is universal agreement that we as a community seek to promote effectiveness in how IRBs operate and function. The GAO Report focuses solely on IRBs and raises questions on what is needed to improve federal oversight and examine the effectiveness of IRBs.  Consistent with charges, SACHRP will restrict our discussion to IRBs and will not address, except in passing, the roles of the many other parties involved in the protection of the rights, interests, safety, and welfare of human research subjects.  These other stakeholder groups include but are not limited to the human research participants themselves, patient advocacy groups, research investigators, research staff, institutional officials, government, industry and sponsors of research, contract research organizations (CROs), bioethicists, certification/accrediting entities, and professional interest groups such as PRIM&R.  We will highlight some of these organizations and entities with additional specificity when responding to Charge 2 later in this document.

In responding to the three charges, SACHRP has combined Charge 1 on effectiveness definition/standard with Charge 3 on potential approaches to measures.  

Defining Effectiveness in Protecting Research Participants

Charge:  What constitutes effectiveness in protecting research participants? This could be defined in terms of avoiding harms, ensuring subjects exercise informed consent, protecting subjects’ rights and welfare, treating subjects equitably or fairly, or achieving greater consistency in applying the regulation, or something else. Depending on what is being protected, the IRB’s actions could differ and measures of effectiveness would vary accordingly. What definition of IRB effectiveness is the most important to focus on and measure?

The charge to SACHRP is to define effectiveness in protecting research participants. SACHRP suggests that an effective IRB could be defined as follows:

“A high-quality IRB could be described as one in which well-trained, well-supported IRB members with appropriate expertise carefully deliberate on the ethics and regulatory compliance of proposed and ongoing research in ways that promote participant protection and understanding, while also supporting and partnering with researchers to facilitate ethical, compliant research without imposing overly burdensome or inefficient requirements and reviews." (See the article “We Measure What We Can Measure”: Struggles in defining and evaluating institutional review board quality, Lynch et al., Social Science and Medicine, Nov 27, 2021.)

SACHRP would add that an additional component of an effective IRB may include appropriate representation of the communities that it serves, and outreach as necessary to review protocols.

SACHRP suggests that the best way to move forward at this point is to focus on standards to protect human research participants. SACHRP has identified a list of standards that could be used as a framework for assessing IRB effectiveness. SACHRP has analyzed each one of the standards based on three factors: (1) feasibility; (2) measurability; and (3) whether the standard measures effectiveness on a procedural or substantive basis.

Charges 1 and 3 Combined: Possible Standards and Potential Effectiveness Measures

SACHRP has identified possible standards for assessing IRB effectiveness. It is important to recognize, however, that evaluating and prioritizing these standards according to the three identified criteria is challenging because each standard requires a different complex, nuanced assessment and balancing of procedural and substantive factors.¹ Thus, our discussion of each standard is provisional only.

SACHRP has grouped the proposed standards into two broad categories: 1. SACHRP Provisional Analyses ofStandards that are Directly Tied to IRB Effectiveness; 2. SACHRP Provisional Analyses of theAdditional Standards that are Partially Impacted by IRB Operations and Review.

SACHRP Provisional Analyses of Standards that are Directly Tied to IRB Effectiveness

• Compliance with Regulatory Requirements, which include the Belmont Principles
  1. The Avoidance of Preventable Harms - Based on the Principle of Beneficence
  2. Appropriate Engagement of Subjects – Based on the Principle of Respect for Persons
  3. Treating Subjects Equitably – Based on the Principle of Justice
• IRB Member and Staff Education
• Certification, Accreditation and Other Peer Review or Equivalent Evaluation Processes
• IRB Reasonableness

SACHRP Provisional Analyses of the Standards that are Partially Impacted by IRB Operations and Review

• Human Research Participant Experiences with Research and Research Oversight
• Compliance of Other HRPP Constituencies with IRB requirements

SACHRP has analyzed each of these proposed standards of measuring IRB effectiveness according to the following considerations:

1. Feasibility for government to enact
2. Measurability, and
3. Procedural versus substantive nature of the proposed standard.  This is an assessment of whether the standard is more administrative/procedural or more substantive ethics.  For instance, turnaround time is an administrative measure, whereas ensuring participant autonomy can be viewed as substantive. 

SACHRP Provisional Analyses of Standards that are Directly Tied to IRB Effectiveness with Regulatory Requirements

Based on the ethical principles identified in the Belmont Report, the Common Rule seeks to protect research subjects’ and groups’ rights and welfare, while also promoting research. The failure of an IRB to comply with the regulations can be assessed through audits, and this is the standard currently used by OHRP and FDA to assess IRBs.   

There are processes and documents in place for conducting these audits.  This process has government authority, with provisions in place for corrective actions and penalties as appropriate.  The findings of inspections are publicly available, so that the process is transparent and generally well understood. 

Feasibility for Government to enact

This standard is feasible for government to enact as it is in place now at FDA, OHRP, and other Common Rule agencies.  That said, it is clear that these programs could benefit from increased funding so that more inspections are performed.

In addition, education and training resources which have been gradually cut over the years from OHRP and FDA budgets could be restored and expanded. SACHRP views these educational and training opportunities as critical components to ensure regulatory compliance and improved research community understanding of what is expected of them.

Measurability

There is a process for conducting and documenting inspections already in place. FDA has a publicly available in-depth inspection manual. ² OHRP has similar processes, outlined in the OHRP guidance document, “Compliance Oversight Procedures for Evaluating Institutions (2009).” 

Assessment of whether more administrative/procedural or more substantive ethics. 

This standard is a combination of both procedural and substantive, although as noted above, in application the emphasis tends to be on the more easily measurable administrative aspects, such as convened meeting quorum and elements of consent.  However, on occasion both FDA and OHRP have looked closely at the interpretation of substantive ethical/regulatory issues, such as occurred in oversight of the SUPPORT study.³

Example of Effectiveness Measures

When FDA inspectors make observations of issues during FDA inspections, these are documented on an FDA Form 483.  The observations are then reviewed centrally by FDA and can be classified as No Action Indicated (NAI), Voluntary Action Indicated (VAI), and Official Action Indicated (OAI).  When an OAI classification is made, this is normally communicated to the inspected party via a Warning Letter.  However, it may also be communicated in more serious communications such as a Notice of Initiation of Disqualification Proceedings and Opportunity to Explain (NIDPOE).  Investigators, IRBs, and other entities may be disqualified by FDA and can no longer participate in FDA regulated research. 

OHRP has similar processes, outlined in the OHRP guidance document, “Compliance Oversight Procedures for Evaluating Institutions (2009).”  OHRP can take actions up to and including suspension of an institution’s FederalWide Assurance (FWA), and debarment of institutions and investigators. 

Compliance with the Belmont Principles⁴

In addition to enforcing the regulations, inspections should track whether the IRB is enforcing the Belmont Report principles, which are referenced in the Common Rule. The Belmont Report highlights three ethical principles: (1) respect for persons; (2) beneficence; and (3) justice.  In the Common Rule, the principles are connected to requirements concerning informed consent, benefits v risks, and the equitable selection of research subjects. We suggest that the government (FDA, OHRP and other Common Rule agencies) assess an IRB’s adherence to the Belmont Principles and further develop tools for this purpose.  There are some evolving approaches to better understand this standard, but they are not widely used or recognized.  This will be an important priority for future work.

Feasibility for government to enact

This standard is feasible to enact because there is significant overlap between the ethical principles and the regulations.

Measurability

There are measures that have been developed for this purpose.  For example, researchers have provided suggestions for using equity impact assessment tools to measure beneficence and justice. ⁵ These tools can not only be used by IRBs to assess protocols, but also by the government to assess the effectiveness of IRBs.  In addition, SACHRP has previously issued a recommendation on applying the principle of justice.⁶

Assessment of whether more administrative/procedural or more substantive ethics. 

This standard of effectiveness is more substantive than procedural. 

Example of Effectiveness Measures

FDA, OHRP, and other agencies’ findings regarding the appropriate application of the Belmont Principles as codified in Section 111 of the Common Rule can be documented in an FDA Form 483 and in OHRP determination letters.

•  IRB Member and Staff Education

IRB members and staff should have appropriate education on and knowledge of the regulatory and ethical requirements for the conduct of ethical research.  This knowledge can be assessed in a variety of ways.

As the charge to IRBs is to review each protocol individually, and there is great complexity and variability in the research reviewed by IRBs, this approach focuses on ensuring the proper structure is in place as opposed to measuring the outcome.

However, IRB member and staff education is a surrogate endpoint, not a true measure of outcomes and effectiveness.  Furthermore, it is a very narrow measure given the many issues that could be included in a standard for IRB effectiveness.  Furthermore, well-trained IRB members or staff, despite their training, could be overburdened or face other issues which negatively impact IRB effectiveness. 

Feasibility for Government to enact

Some Federal Agencies currently require IRB member and staff training and even testing. Agencies could recommend or require that IRB members or staff obtain certification or document formalized training.

Measurability

Educational activities can be measured, just like standard education programs. 

Assessment of whether more administrative/procedural or more substantive ethics. 

This standard is more administrative, as it doesn’t measure effect of the training on the ethics/regulatory compliance of the research.

Examples of Effectiveness Measures

1. Passing a test such as the CIP, CIM, or CCRP
2. Number of Staff with CIP, CIM, or CCRP Certification or an equivalent certification
3. Documentation of completed electronic training modules

• Certification, Accreditation and Other Peer Review or Equivalent Evaluation Processes

The GAO report asks whether peer review should be one of the standards used to assess the effectiveness of IRB effectiveness.  There are several existing non-governmental programs utilizing peer review and other methods to assess the effectiveness and compliance of IRBs. 

However, certification, accreditation and other peer review or equivalent evaluation processes for evaluating IRB effectiveness are voluntary and not required by regulation, and only a small percentage of IRBs undergo such processes.  We believe certification, accreditation and other peer review or equivalent evaluation processes for evaluating IRB effectiveness would arguably improve the effectiveness of IRBs across the United States.

Becoming certified or accredited is a resource-intensive endeavor, both administratively and financially.  In order for such measures to become effective across all IRBs in the US, it would need to be government mandated.  This in turn would likely cause many smaller, less well funded IRBs to cease operations.  As an example, many academic institutions that do not have a medical school still have an IRB for the conduct of social behavioral research, which is often not externally funded.  It would be difficult for many such IRBs to procure sufficient funding to obtain any type of accreditation.

Another option would require voluntary peer review assessments using Quality Management/Improvement Systems. For example, ISO 9001 sets out the criteria for a quality management system. It is based on a number of quality management principles including a strong customer focus, the motivation of top management, the process approach, and continual improvement.

ISO certification standards are internationally accepted across a wide variety of industries.  There are many available organizations to provide the assessment and final certificate.

However, ISO 9001 is a general focus on quality that is not specific to substantive research ethics issues.

Feasibility for Government to enact

To be an IRB in the US, the government could encourage or require institutions to adopt and implement accreditation and/or quality improvement/management systems. As described above, these types of non-governmental systems are in place, but this would transfer the financial and administrative burden in large part from government to IRBs/institutions.

Measurability

AAHRPP, other accrediting bodies, and ISO Certificate and other quality improvement/management systems have developed effective tools with detailed measurements. 

Assessment of whether more administrative/procedural or more substantive ethics. 

The existing certification, accreditation and other peer review or equivalent evaluation processes for evaluating IRB effectiveness use measures that include a mixture of substantive and procedural aspects.

Example of Effectiveness Measures

Achieving accreditation or certification is a measure of effectiveness as it is evidence that the standards utilized by the given system have been met.  However, as with training, it is a surrogate endpoint, not a true measure of outcomes and effectiveness. 

•  IRB Reasonableness

IRBs were created to review complex research proposals and apply ethical principles and regulatory criteria to those proposals.  By focusing on whether the IRB is reasonable, the focus becomes one of assessing the broad protection of having IRB review as opposed to individual outcomes. Factors related to the assessment of reasonableness could include an IRB’s regular engagement and consultation with impacted communities and individuals with appropriate knowledge and expertise. Consultation could include a review of the protocol, and other research materials.  An assessment of reasonableness may include consistency with prior decisions or rationale for inconsistent decisions, providing justification for decisions, transparency of decision-making and education, and the avoidance of clearly unreasonable or wrong decisions.⁷  This standard reflects the rationale for establishing and requiring the IRB system in the first place, a group of qualified individuals considering the ethics of each protocol individually. However, this standard would be very difficult to quantify or measure consistently.  Also, this measure would reflect the cultural biases of the IRB members, which might not represent the research participants’ culture.  IRB reasonableness is also illustrated by its due diligence to ensure that researchers have solicited and incorporated feedback from impacted communities.

Feasibility for Government to enact

Overall, this requires development of tools to measure reasonableness

Measurability

Some parts of IRB reasonableness will be difficult to measure, while others may be less so, such as community engagement.

Assessment of whether more administrative/procedural or more substantive ethics

Substantive

Example of Effectiveness Measures

Measurements of IRB reasonableness could include an assessment that an IRB regularly makes decisions that are compatible with regulatory requirements, general popular acceptability, and research ethics literature.  While there can often be disagreement on an IRB decision, if it is disagreement that includes acknowledgement that the critic can see the other point of view, then the IRB was likely being reasonable.

SACHRP Provisional Analyses of the Additional Standards that are Partially Impacted by IRB Operations and Review

• Human Research Participant Satisfaction with Research and Research Oversight

Research participants’ opinions about IRB effectiveness and personal experience navigating through the process as an actual participant could be obtained through interviews or surveys.  Participant protection is key goal of the IRB system, so a standard for measuring participant satisfaction would focus on a key issue. Most participants do not have a deep understanding of the role of the IRB or the IRB criteria for approval, but they could discuss their experience with the consent process and their experience participating in the research.  This is not a direct measure of IRB effectiveness, because it most likely would reflect their overall experience in the research.

Feasibility for Government to enact

Government would need to encourage such a system to obtain feedback from the research participants.  However, that feedback is unlikely to be focused solely on IRBs.

Measurability

Two ways to measure the research participants’ views on the effectiveness of IRBs would be through surveys and interviews.  Research participants often don’t know much about IRBs, and rarely interact with them. However, it would be feasible to measure research participants’ experiences in research, which is one component of IRB effectiveness.   We recommend evaluation experts be sought outside of SACRHP to develop productive tools for integrating this into oversight activities. 

Assessment of whether more administrative/procedural or more substantive ethics. 

Potentially more substantive than procedural. 

Example of Effectiveness Measures

Positive results from research participant surveys and interviews.

• Compliance of Other HRPP Constituencies with IRB requirements

Other parties that participate in the conduct of research, particularly investigators, but also research staff, sponsors and CROs, must comply with the requirements of the IRB.  For instance, the IRB approved consent form must be used in the consent process, unanticipated problems involving risks to subject or others must be promptly report to the IRB, and investigators are required to provide regular progress reports for certain research protocols.  Satisfaction of these IRB requirements could provide an assessment of the effectiveness of the IRB.

However, this standard involves the broader Human Research Protection Program (HRPP), whereas this recommendation is focused on IRBs.  Also, it would be difficult to determine if appropriate behavior was attributable to the IRB.

Feasibility for Government to enact

This standard would be very difficult for government to enact beyond the usual inspections and monitoring that currently take place.

Measurability

This would be difficult to measure because it would be difficult to determine if the appropriate behavior of the third parties was attributable to the IRB.

Assessment of whether more administrative/procedural or more substantive ethics. 

Both substantive and procedural.

Example of Effectiveness Measures

Measures could include review of FDA Form 483s and other inspection and audit documentation that the parties were not found to be out of compliance with IRB requirements, such as using an IRB approved consent form.

Charge 2:  Stakeholders

Charge:  SACHRP is one HHS “stakeholder”. What other stakeholder groups should HHS convene as part of examining approaches for measuring IRB effectiveness? What factors make an entity an appropriate stakeholder?

SACHRP suggests that one essential factor in making an individual or organization a stakeholder for the purposes of this charge is involvement in the funding, conduct, education, or oversight of research involving human subjects.  In addition, those who participate in research or benefit from its findings are also important stakeholders.  SACHRP has identified the following list of potential stakeholders:

• Individuals who volunteer to participate in research
• Communities where research activities take place
• Communities that are defined as vulnerable in the regulations
• Marginalized communities, including those not included in research activities
• Patient advocacy groups
• Organizations that advocate on behalf of current and future research subjects
• Institutions that conduct research, from Academic Medical Centers to single practitioner sites
• IRBs, including all five types identified by GAO:  University, Hospital or Health Organization, Private, Government, and Independent
• Organizations that host all five types of IRBs identified by GAO:  University, Hospital or Health Organization, Private, Government, and Independent Institutions
• IRB-affiliated members
• IRB unaffiliated members
• IRB staff
• Investigators and sub-investigators
• Research staff
• Federal Agencies - all Common Rule agencies and FDA
• SACHRP
• Industry sponsors of clinical research and commercial entities providing financial and other support of human research
• Contract Research Organizations (CROs)
• Public Responsibility in Medicine and Research (PRIM&R)
• Association for the Accreditation of Human Research Protection Programs (AAHRPP)
• The Consortium to Advance Effective Research Ethics Oversight (AEREO)
• Consortium for Applied Research Ethics – Quality (care-q.org)
• Society of Clinical Research Associates (SOCRA)
• Society of Research Administrators International (SRA)
• National Council of University Research Administrators (NCURA)
• Association of Clinical Research Professionals (ACRP)
• American Association of Medical Colleges (AAMC)
• Specialized professional experts (e.g., Bioethicists, law firms, or IRB/HRPP consultants)

We believe this is a comprehensive list of stakeholders to provide input on the question of IRB effectiveness. 

Conclusion

As noted in the executive summary, SACHRP believes the best way to ensure and measure the effectiveness of IRBs is accomplished by:

1. Using existing and well-established governmental and non-governmental standards, including routine and for-cause inspections of IRBs and educational materials by OHRP, FDA, and federal agencies governed by the Common Rule.  There are written standards in place for conducting inspections.  Non-governmental standards include certification, accreditation and other peer review or equivalent evaluation processes for evaluating IRB effectiveness, as well as training programs for IRB members and staff. However, we recognize these are proxies for quality and effectiveness. Increased funding for inspections, peer review, and training programs can positively contribute to ensuring IRB quality and effectiveness. 
2. Convening stakeholder groups from the community, including evaluation experts.
3. Developing and sharing common definitions and novel tools to measure quality and effectiveness in the IRB ecosystem.

In addition, SACHRP believes that grant funding should be made broadly available for assessing the best methodologies and processes to measure IRB effectiveness.  Such funding should also encompass broader assessment of the effectiveness of Human Research Protection Programs (HRPP) in protecting the rights, interests, safety, and welfare of research participants.

---

Endnotes

¹  The activities performed by IRBs, and the standards for review used by IRBs, are complex.  There is extraordinary variability in the design and focus of research submitted to IRBs, from medical intervention to basic physiology to social behavioral research.  The permutations of study designs and factual patterns are endless.  IRBs are tasked with individually looking at each research protocol and judging their merits in terms of the ethical principles of respect for persons, beneficence, and justice, as described in the Belmont Report and codified in the Common Rule.  These ethical principles are broad in scope and carry equal moral weight.  The principles at times provide conflicting directions in the IRB’s analysis of a given protocol, making difficult any assessment of whether those principles have been applied optimally in IRB consideration of any specific protocol.  In fact, the very structure of IRB review tolerates that in considering the same protocol, different IRBs may be justified in reaching differing conclusions due to putting greater or lesser emphasis on one of the principles.  Ethical tension is expected and inevitable, and IRB review requires the members to make good faith contextual judgment calls; thus, reasonable IRB members at times will disagree on a given protocol. 

Moreover, adoption of specific measures of effectiveness can drive a system to seek primarily to ensure that those standards are met, while de-prioritizing other operational considerations.  In education policy, for example, imposition of specific performance measures on a school system can result in “teaching to the standard,” even if this results in suppressing performance on other measures.  This does not mean that we should not seek to measure IRB performance, but rather that in choosing our standards, we must ensure that they have the closest “fit” to what we as a national community expect IRBs to achieve in their daily work.

²  FDA has an fifty-nine page inspection tool for its IRB inspectors, “Food and Drug Administration Compliance Program Manual No 7348.809, Subject: Institutional Review Boards,” online at https://www.fda.gov/regulatory-information/search-fda-guidance-documents/chapter-48-7348809-bioresearch-monitoring.

Dr. Min-Fu Tsan has proposed four examples of regulatory criteria that could be used to assess IRBs, including:  Appropriately determining the status of protocols as exempt, expedited or needing full IRB review; Whether IRBs consider and document all Common Rule criteria and conflicts of interest prior to approving or disapproving protocols; whether IRBs appropriately consider and approve waivers of consent or waivers of documentation of consent; and whether IRBs conduct timely continuing review when required.  Tsan, Min-Fu, Assessing the quality and performance of institutional review boards, 2022, Journal of Empirical Research on Human Research Ethics, pg. 50.

While the regulatory criteria for approval are intended to embody the Belmont principles, government inspections at the present time often do not address the sections of the regulations based on the Belmont principles.  For instance, FDA findings are unlikely to include findings such as an inappropriate risk benefit ratio under 21 CFR 56.111(a)(2).  Instead, the findings tend to focus on more easily quantifiable regulatory elements such as necessary elements of minutes, convened meeting quorum, and missing elements of consent.  However, that is not to say that OHRP and FDA do not at times address difficult ethical issues in research.  OHRP spent a considerable amount of resources looking into allegations regarding the SUPPORT study.  https://www.hhs.gov/ohrp/regulations-and-policy/requests-for-comments/draft-guidance-disclosing-risk-in-standards-of-care/index.html.  Similarly, FDA has considered difficult issues of pediatric trial design, such as the FDA review of a pediatric muscular dystrophy study.  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6002149/. 

A final issue with the government process of conducting inspections to assess compliance with regulatory requirements is that it does not capture issues that are self-reported by institutions, sponsors, and investigators.  For instance, if there is a Corrective Action Plan that resolves an issue without a resulting government inspection, that will not be captured in the publicly available documentation.

³  https://www.hhs.gov/ohrp/compliance-and-reporting/determination-letters/2013/march-7-2013-u-of-alabama-birmingham/index.html

⁴  Currently an institution must designate the ethical standards it relies on as part of the submission of a FederalWide Assurance.  At present most IRBs in the United States choose the Belmont principles, but some choose the Declaration of Helsinki, and a few choose other standards.  Additional assessment of each Belmont principle follows:

The Avoidance of Preventable Harms - Based on the Principle of Beneficence

The goal of IRBs is to prevent the types of harms that occurred in research prior to the IRB system’s implementation.  These harms are broadly defined, such as individuals being involved in research without their consent, exploitation of vulnerable subjects as a matter of convenience, and research with unacceptable risk/benefit ratios causing unwarranted harm to subjects.  Anecdotally, it appears that these problems have been resolved or at least lessened, but it is uncertain whether this can be attributed wholly or in part to the creation of the IRB system.   It may also be attributable to better education of investigators, more transparency in medical care and research as a whole, or other reasons.  A central difficulty with using avoidance of harms as the standard is that it is difficult to measure the amount the harms are reduced without a randomized trial or other tool by which to measure the effect of the IRB’s oversight as opposed to other factors both within the HRPP and external to the HRPP.  To the extent surrogate markers of avoiding harm are used, they should be centered on reducing risk to human research participants, broadly defined, as opposed to just harms alone.  Some research will inevitably include risks, and the IRB is tasked with determining that the risk benefit ratio is acceptable.  The IRB is not charged with eliminating risk in research, and thus there may be harms that occur that do not reflect a failure of IRB review.  Because this approach is participant focused, it gets to the core of the purpose of the creation of the IRB system.

Ensuring Subjects Exercise Informed Consent – Based on the Principle of Respect for Persons

The exercise of informed consent could be a useful if partial standard for assessing the effectiveness of IRBs.  The principle of autonomy calls for subjects or their legally authorized representatives to be provided with relevant information and make independent, uncoerced decisions on whether they choose to participate in research.  The exercise of informed consent is a central tenant in the ethics of the conduct of research. If this standard were adopted, it would be important to assess whether the consent forms consistently include the elements of consent and that the consent process was conducted with sufficient opportunity for consideration, free from undue influence or coercion.

Treating Subjects Equitably – Based on the Principle of Justice

The Belmont Report includes justice as one of the three principles, along with Respect for Persons and Beneficence.  As such, the equitable and fair treatment of research subjects is one of the goals of both the Belmont Report and the Common Rule.  There are two general rules based on justice that are important to the ethical conduct of research.

• Vulnerable subjects should not be exploited in the conduct of research based on their ready availability and inability to protect their interests.
• Distinct populations, such as children, women, and minorities should be equally included in research which provides a potential benefit to those populations.

As such, justice calls both for the protection, often through exclusion, of vulnerable subjects, and the inclusion of appropriate populations to ensure appropriate research is conducted to provide knowledge about conditions that affect them.  There must be a focus both on group harms and benefits as well as individual harms and benefits.  Justice has historically been the most debated and difficult to interpret of the three Belmont principles.  In order to assess the effectiveness of IRBs in applying the principle of justice, there would have to be a good elucidation of the appropriate interpretation of justice at the protocol level. 

⁵  Ruqaiijah Yearby, Exploitation in Medical Research: The Enduring Legacy of the Tuskegee Syphilis Study, 67 Case W. Res. Univ. L. Rev. 1171 -1226 (2017); Ruqaiijah Yearby, Missing the “Target”: Preventing the Unjust Inclusion of Vulnerable Children for Medical Research Studies, 42 Am. J. of L. & Med 797-833 (2016); Ruqaiijah Yearby, Involuntary Consent: Conditioning Access to Health Care on Participation in Clinical Trials, 44 J. of L. Med. & Ethics 445-461 (2016).

⁶  https://www.hhs.gov/ohrp/sachrp-committee/recommendations/attachment-a-consideration-of-the-principle-of-justice-45-cfr-46.html

⁷  See ““We measure what we can measure”: Struggles in defining and evaluating institutional review board quality,” Fernandez Lynch et. al., 292 Social Science & Medicine 114614 (2022); “Responding to the Call to Meaningfully Assess Institutional Review Board Effectiveness,” JAMA, published online doi:10.1001/jama.2023.8457 (June 2023).