HomeFaculty & StaffAcademic RegulationsChapter 4: Institutional Effectiveness and Institutional Research

Institutional Effectiveness and Institutional Research

 

  1.  Purpose.  This chapter provides guidelines and procedures for Institutional Effectiveness (IE) and Institutional Research (IR) evaluation and planning processes for Marine Corps University (MCU). 

  2. Background.  The purpose of the IE and IR processes at MCU is to support the mission, vision, purposes, and goals of MCU to enhance the quality of education.  This requires a systematic examination of all goals and objectives, assessment of outcomes, dissemination of information, and use of results by decision makers.  The information obtained through the IE and IR processes is valuable for MCU accountability to higher headquarters, the Board of Visitors (BOV), accreditation organizations such as the SACSCOC and the PAJE, and other external agencies.  Additionally, the IE and IR processes play an important role in the conduct of budget reviews, strategic planning, and University-level reporting, such as the Executive Steering Committee (ESC), President’s Planning Council (PPC), Curriculum Review Boards (CRB), and other MCU decision-making bodies. The administrative unit charged with the IE and IR functions for MCU is the Office of Institutional Research, Assessment, and Planning (IRAP).

  3. IE and IR Philosophy at Marine Corps University.  IE and IR are integral elements in ensuring high-quality education is provided throughout the University.  The Director, IRAP will coordinate the University's efforts in this regard.  While the majority of the IE and IR efforts will be centralized at the University level, data collection and analysis directed at the specifics of the curriculum will be provided to the individual schools.  The implementation of IE and IR procedures and activities will also include administrative and educational support units under each vice president, the Gray Research Center and History Division, the Lejeune Leadership Institute, the Center for Advanced Operational Culture and Learning (CAOCL), and the National Museum of the Marine Corps (NMMC). In the distributed mode, the Director, IRAP will maintain University oversight to include access to all data, whether generated by IRAP or collected by the schools and the administrative and educational support units (AES units).  The data collection, analysis of data, and reporting on the details of effectiveness of schools and AES units will be conducted by each school/unit with the assistance of IRAP, as needed.  The common framework for documenting the collection and analysis of data, as well as the use of results, is the MCU Four Column Matrix (Appendix D).  Schools and AES units will submit an annual assessment report (Appendix E and Appendix F) at the end of the academic year as outlined in paragraph six to IRAP for consolidation and forwarding to the President, MCU.  At the University level, data collection and analysis will focus on University goals and objectives, overall University effectiveness, and accomplishment of student learning and administrative and educational support outcomes. 

  4. Core Indicators of Institutional Effectiveness.  To assess the effectiveness of the University in accomplishing its educational goals and outcomes, a set of indicators of effectiveness is required to provide unity of effort.  As shown below, the basic framework for the MCU core indicators consists of four broad areas, specific indicators in each area, and the proponent(s) responsible for assessment.  The indicators will be routinely measured to help determine the health of the University using those instruments listed in paragraph 5 (below).  When possible, multiple means of assessment will be utilized for each indicator to allow for a convergence of evidence and ensure complementary data sets are established for verification and reliability.  The four areas are as follows:

    a.Academic Programs

    i.Student enrollment and graduate totals (MCU Registrar).

    ii.Student achievement of MCU President-approved Student Learning Outcomes (Individual Schools).

    iii.Student satisfaction with academic courses and programs (Individual Schools, IRAP).

    iv.Faculty satisfaction with academic courses and programs (Individual Schools, IRAP).

    b.Services, Support, and Resources

    i.University is properly staffed to accomplish its mission (MCU Civilian Manpower).

    ii.University is properly resourced to accomplish its mission (MCU Finance, Logistics/Supply).

    iii.Student, faculty, and staff satisfaction with support and services (Individual Schools, AES units, IRAP).

    iv.Administrative and educational support unit accomplishment of AES unit review board approved outcomes (Individual AES Units).

    c.Perception and Customer Satisfaction

    i.Identificationofcustomerneedsandexpectation(Individual Schools, AES units, IRAP).

    ii.Customer satisfaction with graduate’s skills/performance (Individual Schools, IRAP).

    iii.Perception and understanding of MCU (Individual Schools, AES units, IRAP).

    d.Organizational Quality

    i.Faculty and staff professional development and enrichment programs (Individual Schools, MCU Academic Support).

    ii.Organizational climate (IRAP).

  5. IE and IR Instruments.  MCU uses a variety of internal and external evaluation instruments and procedures to conduct the IE and IR process.

    a.Internal evaluation instruments used to measure effectiveness and assess educational programs at MCU include the following:

    i.Student Critiques.  Students will complete critiques to evaluate the content of instruction, to determine how well instruction is presented, and to measure the quality of reading and reference materials assigned.  Additionally, students will complete an end-of-academic year assessment of overall satisfaction of educational programs. Student focus groups are also used to augment the ongoing quantitative data collection of student feedback. 

    ii.University Student, Faculty, and Staff Surveys.  The students, faculty, and staff will be administered an annual survey that addresses University-wide issues.  Topics will include support services, organizational quality, professional development, and general education topics.

    iii.Course Content Review Board (CCRB).  As part of outcomes assessment at MCU, the schools, colleges, and academies will convene an internal CCRB to serve as the forum for recording information and making recommendations to improve the effectiveness and efficiency of each school’s curriculum.  The CCRB is a formal meeting with representation from the student body, faculty, subject matter experts, and school administrators who are knowledgeable of the instructional program and its implementation.

    iv.Academic and Administrative and Educational Support Annual Assessments.  Schools and AES unit directors will submit an annual assessment to the Director, IRAP no later than 15 July of each academic year.  Due to differences in academic scheduling, CDET will submit a partial report by 15 July, with the final report submitted no later than 15 September. The report must include a completed MCU Four Column Matrix.  This report will be used to assess the effectiveness of the academic and administrative and educational support programs.

    b.External evaluation instruments and procedures used to measure effectiveness and assess educational programs and graduate job performance data are as follows:

    i.Graduate (Alumni) Surveys.  Questionnaires will be administered annually to recent graduates to determine the relevance of the curriculum and preparation of the graduate for subsequent assignments.

    ii.Reporting Senior (Supervisor) Surveys.  Questionnaires will be administered annually to supervisors of recent graduates to determine if the curriculum equipped the graduate(s) with requisite knowledge and skills to successfully perform job duties in assignments within the Operating Forces or in the joint arena.  These surveys will be distributed approximately eighteen months after a class has graduated in order to allow time for supervisors to assess the value of their MCU education.

    iii.External Scan of Senior Leaders. Visits and telephone conversations with senior officials of the Marine Corps or DoD provide input addressing program outcomes and objectives, course content, methodologies, overall effectiveness, and relevancy to graduates’ current assignments.

    iv.Data Compiled Through the Use of Personnel Databases.  Variables from these sources include fields such as promotions, school selections, job assignments, job performance, etc.

    v.Feedback from the Operating Forces and the Joint Arena.  Feedback from Commanders in the Operating Forces or in the Joint Arena may be solicited through telephone conversations or field study visits.

  6. Procedures.  The integration of data from a wide variety of sources will be used to assess the overall health of the University.  When possible, data and information will be collected from multiple direct and indirect sources to allow for a more complete analysis.

    a.Course Content Review Board.  As previously described, the CCRB is the basic internal review system utilized by each educational program for schoolhouse-level analysis of the effectiveness of its curricula.  This structured process is used to make curriculum modifications based on assessment of student accomplishment of CRB-approved learning outcomes, student feedback, faculty recommendations, or guidance received from higher headquarters.  A CCRB is conducted for each major block of instruction or sub-course within a curriculum.  The educational program director determines the exact composition of the CCRB.  The majority of the data considered in a CCRB comes from learning outcome assessment data, student critiques, and faculty input.  Additional sources of information are inputs from the operating forces, graduate surveys, and reporting senior surveys.  A record of proceedings of CCRBs, including the respective director’s decisions related to course improvements, is maintained by each school. The main product produced by CCRB deliberations is a Record of Proceedings that includes the MCU Four Column Matrix (Appendix D).  Each unit is able to adjust and improve programming on a continuous basis in response to the assessment and feedback received. Any changes and the results of those changes are tracked and documented through the MCU Four Column Matrix process.

    b.Annual Assessment.  This process provides an assessment of institutional performance as it relates to each school and AES unit.  Schools and AES units must plan and conduct IE assessments in order to provide a complete examination of University functions.

    c.Creating the IE assessment plan.  When developing IE assessment plans, schools and AES units establish outcomes to support MCU’s mission and purpose (first column of the MCU Four Column Matrix).

    i.Academic programs will populate column one of the MCU Four Column Matrix with CRB-approved Student Learning Outcomes for each major block of instruction of the curriculum.  AES Units will populate column one on the MCU Four Column Matrix with AES Review Board-approved outcomes.

    ii.Each school and AES unit must determine what types of measures of effectiveness and success criteria will be used to assess accomplishment of Student Learning Outcomes for academic units or accomplishment of unit goals for AES units (column two of the MCU Four Column Matrix).

    iii. Academic programs will assess student accomplishment of CRB-approved learning outcomes by focusing on objective data gleaned from examinations, student research projects, practical application exercises, rubrics, etc.  MCU surveys may also generate some subjective data related to the overall effectiveness of educational programs and customer satisfaction, as well as specific information on facilities, support, and services. However, objective data is more compelling proof of accomplishment of outcomes and goals.  Administrative and educational support units will assess the achievement of AESURB-approved outcomes based on measures of effectiveness and indirect measures captured through survey data.

    iv.An IE plan will be developed at the start of the academic year.  Schools will utilize the CRB-approved Student Learning Outcomes (column one of the MCU Four Column Matrix) and Assessment Measures (column two of the MCU Four Column Matrix) for the IE plan.  AES Units will utilize the AES Review Board-approved outcomes (column one of the Four Column Matrix) and Assessment Measures (column two of the MCU Four Column Matrix) for the IE plan (see Appendix E and Appendix F).  A summary of the results of student accomplishment of CRB-approved learning outcomes or AES units’ accomplishment of stated outcomes (column three of the MCU Four Column Matrix) and use of results of data collection and analysis to incorporate process improvement (column four of the MCU Four Column Matrix) must be completed and submitted in the Annual Assessment Report by 15 July of each academic year.

    d.MCU Annual Assessment Report.  The Annual IE Report consists of a completed Four Column Matrix and Director’s Report from each of the schools and AES Units.  The Director, IRAP will collect and consolidate the IE Reports to develop a comprehensive assessment document for the University, known as the Annual Assessment Report.  The Annual Assessment Report is the primary vehicle used to record policy changes, curriculum modifications, and other decisions that impact a program. They must be reviewed in subsequent assessments to track results of assessment, any changes instituted, and the subsequent results of the change.  Additionally, the Director, IRAP will collect data from other sources relating to the effectiveness of the University.  Trends across the University, as well as documentation of change and the results of any changes, will be of special note.  Resource shortfalls and any other issues impacting educational programs will also be highlighted.

    e.MCU Four Column Matrix.  A major component of the Annual Assessment Report is the MCU Four Column Matrix.  Schools complete and submit the MCU Four Column Matrix (Appendix D) for each major sub-course of a program of instruction.  AES Units complete and submit the MCU Four Column Matrix (Appendix G).   Appendix H provides a template for the types of questions and information that the Four Column Matrix is designed to convey and is applicable to both academic and AES units.  The MCU Four Column Matrix is completed and submitted to IRAP as part of the Annual Assessment Report by 15 July of each academic year.

    f.IRAP Assessment.  The Director, IRAP will report annual assessment results to the President, MCU, via the ESC no later than 15 August of each year.  Periodically, special studies, program evaluations, and/or other data collections may also be conducted and reported by IRAP.

    g.Curriculum Review Board.  As a member of the CRB, the Director, IRAP will utilize the proceedings and documentation of the CRB as one of the multiple measures of Institutional Effectiveness.  Policies and procedures for the CRB are covered in Chapter Three.

    h.Administrative and Educational Support Review Board.  Biennially, unless there is a change to an outcome, each Administrative and Educational Support Unit will conduct a formal review and present its Outcomes to the AES Review Board for approval.  The AES Review Board will ensure the AES units establish specific outcomes that focus on the overarching goals and objectives of the University’s Strategic Plan.  Additionally, the AES Review Board will identify linkages, gaps, and impacts of the AES Units throughout the University.  The AES Review Board is comprised of fifteen standing members.  Membership includes the Chief of Staff, vice presidents, deputy directors, Director of History Division/GRC, Director of the Lejeune Leadership Institute, Director of Institutional Research, Assessment, and Planning, Director of CAOCL, Director of National Museum of the Marine Corps, and the financial director. 

    i.Strategic Plan.  The MCU Strategic Plan is the primary source document that defines the general direction of all University programmatic and developmental initiatives.  The plan highlights the goals, objectives, and action items the University will pursue over the next five years.  Successful execution of the plan is based on advancement within the major functional areas, and serves as an indicator of IE. The President’s Planning Council (PPC) reviews the University’s progress and amends the Strategic Plan, as appropriate.

    j.External Requests.  Throughout the academic year, schools will receive requests from external sources wishing to conduct surveys to assess specific areas of interest.  All such requests, regardless of originator, will be vetted through the Director, IRAP to ensure validity and applicability to the students at MCU and value to MCU and the Marine Corps.

  7. Standards.

    a.Activities used to provide assessment data include results of assessing the achievement of Student Learning Outcomes and Administrative Outcomes, surveys, and focus groups of students, faculty, staff, graduates, supervising seniors in the field, and members of the larger Professional Military Education (PME) community.

    b.At the educational program level, direct measures of Student Learning Outcomes represent a student’s learning at particular points in his or her learning experience.  These measures provide evidence of student learning as assessed by faculty.  Generally speaking, all students are expected to achieve a minimum grade of B-/80% for an educational program or course.  Refer to Chapter Fifteen for a detailed description of grading standards.

    c. The indirect measures of students, faculty, staff, and external stakeholders’ perceptions are gathered through the collection of survey data. The goal is to achieve ≥ 80% of responses on surveys in either the “strongly agree” or “agree” categories indicating favorable levels of satisfaction.

    d.The areas assessed include academic programs and educational service organizations as well as perceptions of faculty and staff services. In addition, students, faculty, and staff are invited to provide input regarding their experiences as a part of the MCU community through comprehensive annual surveys.

  8. Responsibilities.

    a.VPEIOP.  The Vice President for Education Integration, Operations, and Plans provides oversight of University IE and IR programs.   

    b.Director, IRAP.  The Director of Institutional Research, Assessment, and Planning reports to VPEIOP and is responsible for the following:

    i.Data collection and analysis on the effectiveness of the University in fulfilling or achieving its stated mission or purpose.

    ii.Ensuring that individual schools and colleges are properly performing assessment functions in order to measure student achievement of CRB-approved learning outcomes.

    iii.Ensuring that Administrative and Educational Support (AES) Units are properly performing assessment functions in order to best support academic programs and the achievement of Student Learning Outcomes.

    iv.Providing technical advice and procedural guidance for the development, assessment, and administrative management of the University-level institutional research program.

    v.Preparing the annual assessment report that analyzes data collected during MCU annual surveys, reporting senior surveys, curricula assessment, and all school and AES Unit IE assessments and external sources.

    vi.Advising the President, MCU on institutional research issues.

    vii.Serving as a member and advisor to the PPC to incorporate institutional research and assessment findings in University decision-making.

    c.Educational Program Directors.  All educational program directors will perform the following functions:

    i.Establish an institutional effectiveness plan or program, and designate an IE and IR Coordinator as the POC for assessment processes and reporting.

    ii.Submit to the Director, IRAP an Annual Assessment Report (Appendix E & Appendix F) no later than 15 June of each academic year.

    iii.Use questionnaires to survey, assess, and document internal and external evaluation.

    iv.Regularly conduct CCRBs and document the record of proceedings, including changes regarding course improvements, for subsequent incorporation in the annual Director’s Report.

    v.Utilize results of the CCRB to improve curricula delivery and improve the IE and IR process.

    vi.Participate in a biennial Curriculum Review Board (CRB) for the college/school in conjunction with the office of VPAA to ensure academic rigor and relevancy.

    vii.Collect data related to the Quality Enhancement Plan (QEP) as appropriate and present the data to the Director, IRAP for analysis of student improvement in support of the University’s QEP.

    d.Administrative and Educational Support Units.  All MCU administrative and educational support units will perform the following functions:

    i.Account for Institutional Effectiveness (IE) through coordination with the Director, IRAP.

    ii.Collect data related to the effectiveness of the section in accomplishing its stated goals and outcomes.

    iii.Regularly conduct reviews and chronicle evidence of program improvements for inclusion in the annual Director’s Report.

    iv.Submit an annual assessment report to the Director, IRAP (Appendix F, G) to include a completed Four Column Matrix, no later than 15 July of the year.

    v.Participate in a biennial AES Unit Review Board (AESURB) in conjunction with the applicable vice presidents and program directors to ensure continuous improvement.

    e.University Faculty.  Appropriate roles and functions for faculty in the IE and IR process include the following:

    i.Select the appropriate assessment metric to evaluate the accomplishment of CRB-approved Student Learning Outcomes.

    ii.Develop, administer, grade, report, and maintain program examinations used to measure student achievement of CRB-approved learning outcomes.

    iii.Use assessment results to improve academic programs.

    Participate in the CCRB process to improve curricula content and delivery techniques based on assessment of student accomplishment of CRB-approved learning outcomes.