The Academic Relations Committee has updated the AAR’s Guide for Reviewing Programs in Religion and Theology. This document provides those involved with program review considerations for conducting a program review and choosing external reviewers. We have also included guidance for those who undertake the task of being an external reviewer.
This document is written primarily to provide faculty and department or program leadership with suggested steps and strategies for conducting a self-study and external review. These guidelines do not aim to prescribe elements of an effective program; rather, they provide practical suggestions for conducting the most constructive internal and external program evaluations. These guidelines suggest strategies around similarities across institutions and thus, of necessity, overlook specific institutional contexts. Users of this document should always follow institutional guidelines first.
This document is modeled on a document [needs new link] approved by the Academic Relations Task Force in 1999. That earlier guide, which was modeled on guides produced for other fields, was meant to be a work in progress, and this document aims to be the same. We ask for your comments and suggestions, especially your assessment of its helpfulness for your own review process. In the future, we hope to work on different iterations of the Guide more closely tailored to specific institutional segments, e.g., private liberal arts college, public university, church-related college, etc.
Department- or Program-Level Review
All institutions engage in some form of program review, either routinely on campus or as part of accreditation processes (or both). Many campuses follow a predictable calendar of review (e.g. every 5 years, every 7 years, once each decade, or in conjunction with accreditation cycles); others demand reviews when necessary (e.g. in the face of tightening resources, as the result of a request for a change in status, or amid administrative concerns about a program’s productivity). Program reviews are always designed around specific departmental/program and institutional realities and needs, so there is no one-size-fits-all approach. All reviews should keep institutional mission and strategic plans in mind and all should bear in mind the point of view of the administration, which must consider competition for resources on campus and competition for students more broadly.
Although they are labor intensive and sometimes contentious, program reviews can provide opportunities to raise questions and discuss issues with the shared goal of program improvement. They can be a tool for reconfiguring a program, for lobbying for new resources, or for arguing to preserve existing resources. A program review can allow any department or program to advocate for itself and improve on deficiencies or build on strengths commensurate with the institutional mission.
That said, most review documents should be designed to speak to various stakeholders, including administration, faculty in other departments or programs, faculty within the department or program, and any reviewers external to campus (a committee or an individual). These multiple audiences can be particularly tricky for departments of religious studies, whose work is often misunderstood by their colleagues. Some would argue that the most important audience for the entire process is those administrators who have power over a program, from controlling resources to making decisions about program viability. Others would privilege the department or program faculty as the key audience for any review. Still others write a self-study primarily for external reviewers. In every case, the stakes of program review can be high. Awareness of and focus on the most critical stakes and stakeholders are essential to productive program review.
Steps in the review process
Most reviews follow this sort of timetable and generally take a full academic year to complete:
- Preparation for the review (collection of data)
- Conducting the self-study and writing a narrative
- Internal review (by faculty and/or administration)
- External review (by a visiting committee of faculty from other institutions)
- Response to external review accompanied by plan going forward
- Administrative response
Preparation for the review:
The department and programs should plan one or more meetings to discuss the review process and the stakes. Affiliated faculty and students should also be invited to participate, if appropriate. The department or program should take advantage of the wisdom provided by departments and programs on their own campus who have recently gone through the process.
The best program reviews allow a department or program to highlight their contributions to the institutional mission, the success of their own (and, when appropriate, the general education) curriculum, the scholarship conducted by their faculty and students, and their engagement in the life of the institution and/or its community. Gathering evidence of those contributions is a key part of the early stages of any review process.
Reviews are usually driven by administrators who set parameters (e.g. specific foci for the study, who will serve as external reviewers [both on- and off-campus], and opportunities for departmental or program response). If there is input from the department or program as to the design of the review, the department or program should think in terms of the results they want and the kinds of questions and approaches that would best lead to those results.
In an ideal world, department or program faculty should talk together regularly about program goals and strategies to achieve those goals. Ongoing conversation makes for an easier self-study and external review process. In the absence of ongoing conversations, for whatever reason, a program review can provide an opportunity to discuss and assess issues together. A program review can be a formative experience for all involved, resulting in changes as far ranging as updates to the department or program website to a fully revised curriculum.
Also in an ideal world, departments and programs will track their productivity at least annually. In general, departments and programs should regularly track, for instance, faculty and student awards, publications of all types, internal and external funding awards, alumni achievements, faculty service on campus and beyond, and student testimonies of teaching and mentoring. Those documents will be collated and used to assemble the self-study.
Departments and programs should be sensitive to the kinds of information administrators might expect. In many cases, numbers count (e.g. SCH or major numbers, enrollments, job placement of students, and external funding). Departments and programs may be asked to analyze enrollment trends over time; to examine the balance of introductory courses vs. advanced courses; to track patterns of over-enrollment and under-enrollment; and to assess whether current course offerings meet department or program learning goals by examining longitudinal assessment data. Some numbers will need significant context, which can mean doing institutional research beyond the department or program.
Conducting the self-study and writing a narrative
The self-study document sets the tone for the entire review. This document is written for multiple audiences and should serve as a candid assessment of department or program strengths and weaknesses. The focus of the self-study should almost always include an in-depth critical examination of the curriculum and student learning, as those metrics most directly affect workload and resource decisions. The department or program should thus be committed to examining the current curriculum, the balance of courses taught at various levels, and the department or program and course-level goals for student learning. Consideration of the curriculum should include a review and analysis of new directions or innovations within the discipline more broadly.
Preparation for writing the self-study involves amassing data from various sources, as described above. Most institutions have some institutional research outlet that will provide data over time. Some departments or programs will want to develop an alumni survey to find out about career placement, for a retrospective evaluation of learning from former students, and to find out about outstanding alumni. Some data is more difficult to gather, such as the role of the department or program in extracurricular activities or community engagement. Annual collation of this information, as noted above, helps enormously in this effort.
Evidence (much of which will be presented in raw form in appendices to the self study) of department or program productivity (or lack thereof) must be embedded in a coherent argument about the state of the program. Some institutions will provide a template for this document. Most self studies describe the history of the department or program as well as its current goals and dynamics. Honest disclosure of pressures and responses help to contextualize decisions and approaches. The self study should aim to make a persuasive case for current or future strategies.
It is essential for all self-study documents to be focused and concise. The narrative itself should be no longer than 15-20 pages (excluding appendices). In those 15-20 pages, the self study should
- Consider information and data pertinent to the department or program and its operation, including but limited to enrollment data, results from an alumni survey, or institutional data like senior surveys or graduate school admission results.
- Use those data to assess the department’s or program’s effectiveness in furthering its goals and objectives and the mission of the college, to identify strengths and weaknesses, and to highlight key issues it wishes to raise during the review. This document is a chance to identify and define problems or challenges so they aren’t defined or identified for you. It is also a chance to propose realistic solutions.
- Present evidence of faculty achievement and engagement.
- Present a preliminary plan for the subsequent five to ten years.
The following elements are often included in descriptive and evaluative narrative:
- Brief history of the program
- Defining characteristics of the program (subject areas; missions; in context of campus; beyond campus)
- Overview of curriculum, including fieldwork and internship and other “high impact” opportunities.
- Discussions of data about enrollments, degrees awarded, and such
- Assessment system and results over time, including exit experiences and capstone projects
- Advising
- Available resources
- Faculty (number, expertise, contributions, workload)
- Relations with other departments or programs
- Staff
- Funding and fundraising
- Resources (space, equipment, library resources)
- Graduate program, if any, description and outline of requirements and support
Appendices often contain the following elements:
- Overview of campus (size, organizational structure, denominational affiliation, if any)
- Catalog program description
- Program mission statement
- Advising handbooks, if any
- Degree requirements for any degrees
- Descriptions of core courses, including syllabi when possible (generally over the last 3 years)
- A summary of course enrollment patterns as well as numbers of majors and minors over the past decade
- A summary of learning and career outcomes over the last five to ten years
- List of faculty, with ranks and titles and year joining faculty
- CVs of all faculty members who regularly teach in the department or program
- Official description of governance structure
- Data over 5- or 10-year period (e.g. majors, degrees awarded, student/faculty ratio, particularly in light of college or institutional data)
- Program budget (not always required by administrators)
- Other documents that provide context for program goals and achievements
Internal review
This is an on-campus review that precedes the external review. Most institutions constitute an internal review committee comprised of faculty with some familiarity (or none, depending) with the program being reviewed. This committee can provide invaluable insight into resources and priorities on a particular campus. Bear in mind that internal reviewers might need more orientation to a particular department or program than one might assume.
External review
The external review is often a pivotal part of the program review process. The nomination process for constituting a committee is specific to institutions, with some involving the department or program more closely in the process and with others working with more “top down” approach. When the department or program has a say, it is best to suggest reviewers in similar departments or programs who are active in the guild and who, ideally, have some administrative experience. Departments and programs are often permitted to suggest reviewers from which administrators will choose. The Executive Office of the AAR can help match reviewers with programs (see Request an External Reviewer on the AAR Academic Relations Committee website), and inquiries of peer departments or programs on comparable campuses can yield good results.
Most external review committees consist of two or three faculty from other institutions who possess pertinent experience and who, in the view of the department or program under review, can offer objective, useful advice from a disciplinary perspective. External reviewers usually visit campus to meet with administrators, undergraduate majors, graduate students (if any), faculty, staff, and any other groups that the department, program, or upper administration chooses. This visit should take place over a day or two, with ample time for the review committee to work together while on campus to draft their report.
A typical schedule for an external review might look something like this:
The External Review Committee usually meets with the department or program members over dinner on the evening before the first full day. Although this is usually a more informal social gathering, this dinner should help orient the External Committee and review the agenda for the following day(s).
Some campuses will arrange for the External Review Committee to meet with upper-level administrators (president or chancellor, dean(s), and chairs) as the initial on-campus meeting.
The External Review Committee needs considerable time (one hour per faculty member if possible) to meet individually with members of the department or program and with faculty and staff in other departments or programs, as appropriate. Members of the External Review Committee should also meet with students (usually in a group).
Time should be made available for the committee to meet by itself to discuss the review as it progresses. Several hours on the last half day of the visit should be set aside for the committee to discuss and draft its report.
The External Committee sometimes meets with the entire faculty in the department or program under review to discuss its findings and outline the general conclusions it will present in its written report.
The committee often meets with upper-level administrators for an exit interview.
There are a few best practices in preparing and treating external reviewers. Reviewers should have at least two weeks to read the self study and any other relevant documents (e.g. institutional strategic plan, department or program strategic plan). They should always be given time to work together while on campus to process issues. This is particularly important prior to any exit interview with administrators. It is imperative that faculty in the department being reviewed be honest about any problems and proposed solutions. It is also important to focus on the most critical issues and to remember to highlight strengths without being too modest. If, for instance, the department or program contributes significantly to the general education program, give outside reviewers a chance to meet with people who can speak to that.
Above all, remember that outside reviewers are often the source of diplomatic and effective solutions that will get the attention of administration. Be open to constructive suggestions, both during the visit and in the final report.
The external review committee will be expected to provide a final report on a timeline and often following a template provided by upper administration. This report, which is often around 7-10 pages, provides an outside perspective on the self study documents as well as the campus experience. The department or program should use this document to help advocate for any needed change.
Response to external review accompanied by plan going forward
Much of the work of the program review is done after the external reviewers have sent their reports, but that doesn’t mean that departments or programs need to wait for the official report to start responding to it. Think in terms of making changes regarding issues raised by the review and of producing a tactical document that shows the department or program sees itself clearly and has a plan for fixing any issues, either internally or by asking for resources to do so. If the department or program disagrees with anything in the external review report, it should be able to explain its reasoning. The response document should include strategies and timetables for improvements and should make an argument to administration about why the program is worth funding.
Administrative response
Institutions differ widely on how much or how formal any administrative response might be. This document cannot begin to anticipate what administrative officials might do with any self-study or what any department or program’s response should be.