David E. McCullin, DM, is an independent researcher in the areas of national security, strategic studies, and judgment and decision making. He holds a doctoral degree in management of complex adaptive systems, master’s degrees in national security and strategic studies and business policy, and a bachelor’s degree in engineering. Dr. McCullin served for 24 years in the U.S. Army, with 13 years in Army Special Operations with the Civil Affairs and Psychological Operations Command. He served in Operations Desert Shield and Desert Storm (1990–91) in Saudi Arabia and Kuwait, Operation Joint Forge (1998) in Bosnia, and Operation Enduring Freedom (2001–14) in Afghanistan. He also served nine years with the U.S. Department of Homeland Security as an intelligence analyst and an infrastructure security analyst. Dr. McCullin is a subject-matter expert in the military decision-making and the federal plan development processes.

The views expressed in this article are solely those of the author. They do not necessarily reflect the opinions of Marine Corps University, the U.S. Marine Corps, the Department of the Navy, or the U.S. government.

Exploring Evidence-Based Management
in Military Planning Processes
as a Critically Appraised Topic

David E. McCullin, DM
https://doi.org/10.36304/ExpwMCUP.2021.04

PRINTER FRIENDLY PDF

Abstract: This study uses a critically appraised topic (CAT) to explore the potential of integrating evidence-based management (EBMgt) and military judgment and decision-making (MJDM). The study uses the five focus areas of the Commandant’s Planning Guidance: 38th Commandant of the Marine Corps to search scholarly databases such as the ABI/INFORM Collection from ProQuest and Business Source Premier from EBSCO. The search process was conducted using a research question variable synthesis (RQVS), introduced specifically for this study by the author. The RQVS applied to each of the Commandant’s five priority focus areas in separate inquiries, producing five separate data sets. The RQVS is a search enhancement methodology developed by the author. This methodology improves the linkage between the search process and the research question (RQ) and enhances rigor and transparency of the overall study. The key findings are that there is sufficient scholarship to address problem areas in each of the Commandant’s five priority focus areas. This study demonstrates that an integration of EBMgt and MJDM is both feasible and pragmatic.

Keywords: evidence-based framework, evidence-based management, EBMgt, military planning, judgment and decision-making, MJDM, U.S. Marine Corps, Commandant’s Planning Guidance

In a presidential memorandum on restoring faith in government dated 27 January 2021, U.S. president Joseph R. Biden Jr. stated, “It is the policy of my Administration to make evidence-based decisions guided by the best available science and data.”1 With this declaration, the Office of Science and Technology Policy was subsequently charged with the responsibility of ensuring “the highest level of integrity in all aspects of executive branch involvement with scientific and technological processes.”2 The strategy posited in this memorandum is that U.S. federal agencies integrate an evidence-based framework into decision-making for planning and policy. The memorandum reestablishes the Foundations for Evidence-Based Policymaking Act of 2018.3      

This article explores the feasibility of integrating evidence-based management (EBMgt) and military judgment and decision-making (MJDM) using a critically appraised topic (CAT). In its Guideline for Critically Appraised Topics in Management and Organizations, the Center for Evidence-Based Management (CEBMa) defines a CAT as an 11-step process that “provides a quick and succinct assessment of what is known (and not known) in the scientific literature about an intervention or practical issue by using a systematic methodology to search and critically appraise primary studies.”4 Excerpts from this publication will be used in this article to illustrate the context of the 11 steps within a CAT.

Definitions
The following definitions are provided here to offer context and clarity to the terms used in this article. They represent a compilation of evidence and experiences.

  1. Critically appraised topic (CAT): provides a quick and succinct assessment of what is known (and not known) in the scientific literature about an intervention or practical issue by using a systematic methodology to search and critically appraise primary studies.5
  2. Inclusion criteria: determines whether a study will be included in the CAT. It should be guided by the CAT question and the population, intervention, comparison, outcomes, and context or PICOC.6
  3. Research question (RQ): a question developed from a pending decision to focus a research effort that is designed to create evidence.7
  4. Evidence-based management (EBMgt): a decision-making framework that draws evidence from experience, stakeholder input, organizational data, and scholarship.8
  5. Evidence-based practice framework: includes asking, acquiring, appraising, aggregating, applying, and assessing.9
  6. Research framework logic: portfolio of a logic framework that defines specific parameters to guide the development of an RQ and validate data set content. A common thread within the portfolio is the use of variables that define who the study impacts, the instrument(s) used, and what is expected or what the study will produce.10
  7. PICOC: a logic framework using population, intervention, comparison, outcome, and context as its defining variables.
  8. Research question variable synthesis (RQVS): a new introduction to database query. The RQVS uses variables from the RQ to create search strings. This process increases transparency by employing a specified process, and its use of the RQ variables intricately links the RQ to the final data set.
  9. Stakeholders: individuals or organizations directly impacted by a judgment or decision.11
  10. Military judgment and decision-making (MJDM): a spectrum of decision-making processes related to the arts and sciences of national defense. Within this spectrum, quantitative and qualitative processes are employed to make decisions based on multiple courses of action.
  11. Theoretical framework: links theory and practice. The author of a study selects one or more theories of social science research to help explain how a study is linked to a practical approach identified in a RQ.12
  12. Conceptual framework: an explanation of how the constructs of a study are strategically used in addressing the RQ. A theoretical framework is often accompanied by a sketch showing how the constructs of a study are related.13

Series Overview
This article is the first in a four-part series that will explore the integration of EBMgt and MJDM. In this article, the feasibility of integration is explored. The other articles will detail how integration could be implemented by discussing the EBMgt framework, the development of a data set, and the evaluation of data. The article on EBMgt framework will describe systematic review methodologies as a tool for leveraging scholarship into the integration of EBMgt and MJDM. The article on developing a data set will explain how to set the parameters that scope a data set and introduce a search methodology that uses the RQ to identify the scholarly articles that become a data set. The article on evaluating data will introduce a model for critically appraising scholarship, organizational data, subject matter expertise, and stakeholder input.      

The order in which these four articles should be read depends on the background of the reader. For example, researchers who are familiar with evidence-based framework would not need to familiarize themselves with the systematic review format and its associated methodologies, such as building a database and evaluating evidence, before reading this article. Planning practitioners who are familiar with the concept of EBMgt but unfamiliar with the systematic review tools and methodologies may find it more enlightening to read this article last. These four articles complement and supplement one another by overlapping key constructs. Regardless of the order in which these articles are read, together they are designed to provide a comprehensive understanding of the effort and resources that an integration of EBMgt and MJDM would entail.

Evidence-Based Management and Military Judgment and Decision-Making
The operational arts and designs that currently frame and execute MJDM in the United States include the Army problem-solving process, as described in U.S. Army Leadership, Field Manual (FM) 22-100; naval operational planning, as described in Navy Planning, Navy Warfare Publication (NWP) 5-01; Air Force operations planning and execution, as described in Air Force Operations Planning and Execution, Department of the Air Force Instruction (DAFI) 10-401; Marine Corps planning guidance, as directed in the Commandant’s Planning Guidance: 38th Commandant of the Marine Corps; and the Joint operations planning process, as described in Joint Planning, Joint Publication (JP) 5-0.14 These processes balance both cognitive and system constraints to plan and make decisions. Balancing cognitive and system constraints for MJDM employs the collecting, assessing, analyzing, and synthesizing of command experience data, organizational data, and stakeholder input data.      

The EBMgt framework integrates the arts and sciences of practitioner expertise, organizational data, stakeholder buy-in, and a methodology for incorporating scholarly research into a framework for management decision-making.15 According to CEBMa, a critical evaluation of the best available research evidence, as well as the perspectives of those people who might be affected by the decision, epitomizes the core concept of EBMgt (figure 1).16      

Figure 1. Evidence-based practice

Source: Eric Barends, Denise M. Rousseau, and Rob B. Briner, Evidence-Based Management: The Basic Principles (Amsterdam, Netherlands: Center for Evidence-Based Management, 2014), 7, adapted by MCUP.            

EBMgt parallels MJDM by integrating subject matter expertise, organizational data, stakeholder perspectives, and scholarship in decision-making. The only component of decision-making that differentiates EBMgt from MJDM is scholarship. The integration of scholarship is constrained by three factors: a doctrinal precedence for an integration; the belief that the scholarship is inaccessible; and the potential relevance of scholarship to MJDM.      

In terms of doctrine, there is no presidential directive mandating the use of scholarship to support the existing focus areas of EBMgt that are used in military planning today.17 Therefore, commanders are free to explore integrating the full spectrum of EBMgt in MJDM. In terms of accessibility, databases such as EBSCO, ProQuest, and Google Scholar, which contain the requisite scholarship that currently supports worldwide EBMgt, are available through standard internet subscriptions. Further, open-access journals and monographs, such as those published by Marine Corps University Press and other similar publishers, are made available to readers at no cost. Therefore, commanders and staff have access to scholarship.18      

In terms of relevance, this study explores the integration of scholarship in MJDM using the Marine Corps’ 2019 Commandant’s Planning Guidance. The exploration is done through a systematic review methodology in which scholarly literature is collected, assessed, analyzed, and synthesized to inform a management decision. The potential of merging EBMgt and MJDM is the proposition explored in this study. The CAT is the specific systematic review tool to conduct the study to explore this integration.

The Critically Appraised Topic
As defined above, a CAT “provides a quick and succinct assessment of what is known (and not known) in scientific literature on a given topic, using a systematic methodology to search and critically appraise primary studies.”19 The CAT used in this study is organized into five sections. The introduction describes the organization of the study. The background section provides the purpose and scope. The theoretical framework section offers the perspective from which this study was conducted. The conceptual framework section explains the linkages between the concepts. The methodology section explains how the CAT was conducted and identifies the rigor and transparency of the study. Each of the 11 steps of the CAT will be described using citations from CEBMa’s Guideline for Critically Appraised Topics in Management and Organizations.      

Theoretical Framework
A theoretical framework is not a mandatory component of the CAT. However, as postulated by Norman G. Lederman and Judith S. Lederman, theoretical frameworks are “critically important” to quantitative, qualitative, and mixed methods research to “justify the importance and significance of the work.”20 In many journals, “the lack of a theoretical framework is the most frequently cited reason for [an] editorial decision not to publish a manuscript.”21 A theoretical framework provides a lens through which a reader can gain an understanding of how the author sees the practical application of theory.      

The theoretical lens through which this appraisal was viewed is the isomorphic properties theory. Isomorphic properties are defined as characteristics of units in two or more separate sets that are common in each set. In addition, both populations face similar sets of environmental conditions within their respective populations.22 In this view, EBMgt and MJDM share isomorphic properties in that an informed decision in either process is supported by problem identification, the collection and evaluation of information, and the creation of knowledge to serve as the basis for recommendations to solve the identified problem.      

Regardless of Service component or process, all MJDMs are used to inform decisions related to the disposition of resources. Therefore, all Service component MJDM methodologies are isomorphic. By this same logic, this study recognizes the properties of EBMgt and MJDM as isomorphic. Evidence-based management, policy, and practice are all executed in the evidence-based framework of asking, acquiring, appraising, aggregating, applying, and assessing. MJDM operates in the framework of identifying the problem, gathering information, developing courses of action, evaluating courses of action, and comparing courses of action.      

Figure 2 depicts the judgment and decision-making continuum where EBMgt and MJDM are shown as individual sets with isomorphic properties. Both sets are depicted as bows that are trained on a target depicted as a globe. The bows represent the judgment and decision properties. The arrows represent resources to be allocated. The globe represents complex adaptive systems where the resources will be allocated to accomplish a goal. The decision-making properties empower the allocation of resources. The resources are directly targeted at a specific goal in complex adaptive environment.      

Figure 2. Properties of EBMgt and MJDM

Source: Courtesy of the author, adapted by MCUP.            

In both EBMgt and MJDM, the associated properties serve as the foundation of the decision-making mechanism. In isomorphic sets, the properties that make up the mechanisms are interchangeable. In this case, the semantics that define the sets are used specifically to define a taxonomy. The properties within the MJDM set are expressed in terms common to military culture. Likewise, the terms within the EBMgt set are more familiar in a management environment. In both sets, the activities associated with each property will accomplish the same objective regardless of which set it is employed with and without compromising the process. The only inhibitor is in the bounded rationality of the practitioner’s learning curve in interpreting the properties.

Conceptual Framework
The properties that link MJDM to EBMgt will be used to determine if there is adequate relevant scholarly literature to inform MJDM. The concept is that there are isomorphic properties among the individual U.S. military Service component planning methodologies. There are also isomorphic properties between MJDM and EBMgt. Therefore, a single Service component planning methodology model can be used to explore the feasibility of integration between EBMgt and MJDM. The planning methodology used in this study is the Marine Corps’ 2019 Commandant’s Planning Guidance.23 Theoretically, these isomorphic properties suggest that if there is sufficient and relevant scholarship to inform Marine Corps planning, MJDM will also be informed overall. The vehicle in which to explore the potential isomorphic properties of these planning methodologies is the CAT (figure 3).

Figure 3. Presidency and isomorphic properties linkages diagram

Source: Courtesy of the author, adapted by MCUP.

Research Question Overview
General David H. Berger, the 38th Commandant of the Marine Corps, identified five focus areas of Marine Corps planning in the Commandant’s Planning Guidance: force design, warfighting, education and training, core values, and command and leadership.24 Each of the focus areas became a separate data set that addressed a single research question (RQ). A research question variable synthesis (RQVS) was introduced to address the previously stated paths of inquiry that explore the availability and quantity of literature to support EBMgt-MJDM integration. A quality appraisal was then conducted on each data set in two separate categories. The first category assessed rigor, transparency, validity, and reliability, while the second assessment measured relevance to the Commandant’s Planning Guidance. The quality appraisal addressed the path of inquiry on the relevance of literature to support EBMgt-MJDM integration.

Research Question Variable Synthesis
This subsection introduces the RQVS, a concept for developing search term options using RQ variables. Preceding the quality assessment in this study, the RQ was developed based on the five focus areas of the Commandant’s Planning Guidance. Developing the RQ was the antecedent to the application of the RQVS. In general, the RQVS summarizes, translates, and develops search terms for a database interrogation and refines the population of available literature into the data set for a study. This is done by developing word lists from a synthesis of codes identified in the abstracts and key terms in a sampling of literature. The RQVS is implemented in the following sequence: 1) identifying RQ variables and creating an initial list of terms; 2) database query; 3) sampling; 4) synthesis; 5) search terms; and 6) refining.      

As figure 4 illustrates, the concept underpinning the RQVS was to deconstruct the RQ in order to reconstruct search terms that link the RQ to the final data set. In this process, one of the many research framework logic approaches is used to conceptualize the RQ. Although these frameworks are applied with extreme latitude, they offer a series of checks and balances that ensure integrity between the included and excluded studies that comprise the data set.    

Figure 4. Sequence from logic framework to data set

Source: Courtesy of the author, adapted by MCUP.          

Although the portfolio of these framework logic approaches is continuously expanding, common identifiable variables include an impacted population, a method of exploration, and the desired outcome. After an RQ is crafted, it is deconstructed in terms of its variables, which are used as the search terms in the initial search string. In the RQVS method, each search improves on the previous one to either grow or narrow the field of available literature into a relevant data set. The searches continue until the researcher is satisfied that a unified final search statement has been developed that results in a relevant data set.      

The methods explaining the general application of the RQVS are the same methods used in this study. The concept behind this effort was to link the available scholarship resulting from the search with the five focus areas of the Commandant’s Planning Guidance. This effort revealed the volume and quality of scholarly research articles available that were related to each of the focus areas. The extent to which there are available scholarly articles for each tenet suggests the extent to which EBMgt can be applied to Marine Corps planning methodologies in particular and to MJDM in general.      

There are several distinct advantages of using the RQVS. The process offers rigor by using a standardized approach to seek integrity between the RQ and the data set. This approach offers transparency, in that it is recordable and publishable. Because of this transparency, the process lends itself to easy replication. This ease of replication increases validity, and the increased ability to validate increases reliability.

CAT Overview
The CAT is a method of research based on the systematic review of literature commonly used in dissertation and EBMgt research. This systematic review follows an objective process that is based on the scientific method of research. Both the systematic review and CAT methodologies provide rigor and transparency to decision-making processes. The CAT is executed in 11 steps, as outlined below. These steps are the methodology critical to identifying and assessing scholarly literature to answer a specified management inquiry proposed as a RQ.25

  1. Background
  2. Research question (RQ)
  3. Inclusion criteria
  4. Search strategy
  5. Study selection
  6. Data extraction
  7. Critical appraisal
  8. Results
  9. Conclusion
  10. Limitations
  11. Implications and recommendations      

Step 1, background, provides the rationale for the study. Step 2, the RQ, sets the parameters of the study by identifying specific paths of inquiry that answer the question posed. The paths of inquiry are similar to hypothesis in that they answer the RQ. In a CAT, answering the RQ is done by extracting data from the data set of scholarly literature that supports or disassociates evidence with the RQ. Steps 3, 4, and 5—inclusion criteria, search strategy, and study selection—create a data set of scholarly literature by data base searches in a transparent replicable process. Step 6, data extraction, creates evidence by extracting data that will support or disassociate the RQ and evidence. Step 7, critical appraisal, assesses the quality of the scholarly literature. In the CAT methodology, the critical appraisal of data set literature is often limited to methodological appropriateness and findings of each study in the data set. Step 8, results, explains how the evidence from step 6 supports or disassociates the RQ with each path of inquiry. Step 9, conclusion, is where the author states concisely whether the evidence supports the RQ. In step 10, limitations, the author discloses limitations impacting the study that could have influenced the findings. Finally, in step 11, implications and recommendations, the author recommends how the findings might be used based on what the evidence has implied.26      

Step 1. Background
The background should clearly state what the rationale for the CAT is and explain why the question being asked is important and how it might relate to a wider understanding of a general problem.27 The rationale for bringing EBMgt to military planning is to make decisions with the best evidence per President Biden’s directive on restoring faith in government. The vehicle for exploring this potential is a research design incorporating CAT, which is a tool of EBMgt under the umbrella of systematic reviews.

Step 2. Formulating the CAT Question
In general, formulating the RQ was done by defining the population, intervention, comparison, outcome, and context (PICOC) that help define study. Population refers to the groups or individuals impacted by the study, also called a target audience. Intervention refers to a mechanism that will lead to the desired outcome. Comparison speaks to the current state verses a state without the intervention. The outcome is the desired end state. Finally, context incorporates the type of organization and/or the extenuating circumstances being explored. According to CEBMa, PICOC helps determine how different the literature is from the PIOC parameters. The closer they are, the more generalizable the study should be.28      

In this study the PICOC logic was as follows:

  • Population: MJDM planners and decision makers.
  • Intervention: a study exploring the potential for integrating EBMgt and MJDM.
  • Comparison: decision making that does not include evidence from scholastic study.
  • Outcome: evidence of the potential for integrating EBMgt and MJDM.
  • Context: operational and strategic planning practitioners engaged in the arts and sciences of MJDM.

Research Question
What is the potential of available scholarship for integrating EBMgt into MJDM processes using the five focus areas found in the Commandant’s Planning Guidance?      

Exploring potential in this RQ initiates three paths of inquiry:

  • Is the scholarly literature sufficient in quantity to integrate into and inform MJDM?
  • Is the scholarly literature sufficiently relevant to inform MJDM?
  • Is the scholarly literature sufficiently available to integrate into and inform MJDM?            

To explore the quantity, relevance, and availability of scholarship under this RQ, a baseline of MJDM must be associated. In this appraisal, the Commandant’s Planning Guidance was used. This publication is ideal because it lays out issues that the Commandant of the Marine Corps has designated as priorities. In theory, if there is a sufficient quantity of scholarship that is available and relevant to these priorities, then the integration of EBMgt and MJDM is plausible.      

Step 3. Defining Inclusion and Exclusion Criteria
One of the distinguishing features of a CAT is the prespecification of criteria for including and excluding studies. These criteria determine if a study will be included in the CAT data set. The development of criteria is based on the CAT question and PICOC.29 The inclusion criteria determine whether a study will be included in the CAT by reviewing its abstract, key terms, and in some cases the full text. Within the PICOC, particular attention should be paid to including or excluding studies based on the outcome measures that will be considered when answering the question.      

In this study, the inclusion criteria were:

  • Date: studies conducted between 2015 and 2020.
  • Study type: peer-reviewed scholarly journals.
  • Systematic review: original research, qualitative studies, or quantitative studies.            

To exclude articles, the following criteria were met:

  • Language: studies in languages other than English.
  • Focus: weak nexus to the Commandant’s Planning Guidance’s focus areas, management, and/or planning.
  • Format: less than full text available.

Step 4. Search Strategy

Table 1. Initial RQVS search

Query #

RQ variable
(Commandant’s focus area)

Literary sources
available

1 Force design 124,435
2 Warfighting 603
3 Education and training 605,610
4 Core values 107,009
5 Command and leadership 3,072

Source: Compiled by the author, adapted by MCUP.      

The initial search terms of the RQVS are outlined in table 1. As evident by the heading of the center column, the initial search terms are the same as the five focus areas found in the Commandant’s Planning Guidance. The column titled “Query” contains a numerical identifier for each search. The column labeled “Literary sources available” represents the results of each of the five searches in terms of the volume of literature available. Each search represents the initial data set for each of the five focus areas. For example, the initial data set for search #1 is titled “Force design” and contains 124,435 sources. All of these searches were filtered for scholarly peer-reviewed journals.      

In each of the initial data sets, a sample of source literature was selected randomly, aided by the search engine that placed the most relevant sources first. Because of the volume of literature and the search engine’s functioning, the available articles were screened in the order they appeared in each search results list. The initial screening was conducted by scanning through abstracts of each article in the sample to identify the scope and purpose of the study.30 Key constructs describing the scope of each article and purpose were identified. The terms describing the key constructs were placed in the column with the heading “Summary of key terms,” as can be seen in table 2.      

A table allows RQ variables to be compared side-by-side with key terms. From this observation, the summary of key terms was synthesized in two actions: by creating search term options that are synonymous with the summary of key terms and by eliminating irrelevant terms. The search term options were captured in the column with a heading of the same name.

Table 2. RQVS development

Search # RQ variable Summary of key terms Search term options
1 Force design Military budgets; policy sciences; military policy; armed forces; national defense force; collision-free planning; complexity reduction; artificial force learning
  • Identifying personnel requirements
  • Complex adaptive systems
  • Military science
  • Organizational strategy
2 Warfighting National and international; military art and science; technological innovations; strategy; cyber terrorism; radical Islamist; command and control systems; coalition company; personnel management; United States Army human resource management; document delivery analysis; military doctrine analysis; armies: officials and employees
  • Defense arts and sciences
  • Doctrine
  • Operations
  • Combating terrorism
3 Education and training Addiction; crisis intervention; education; training; substance use disorder; disability, policy advocacy: interprofessional education; psychotherapy theory, research and practice
  • Diversity training
  • Tolerance training
  • Professional development
  • Motivation
  • Morale
4 Core values Corporate sustainability; organizational behavior; business ethics; social values; personality; sustainable development; human values; intercultural space; cultural groups; multiculturalism; economic value; cultural values; multicultural education; humans; Slavic culture; social interaction; cross-cultural studies; Indian culture; moral disengagement; moral identity, ideological conflict; aggression
  • Organizational culture
  • Diversity
  • Professionalism
  • Social choice
  • Ethical values
5 Command and leadership Command, leadership; military service, power, influence; military job descriptions; leadership (logic); professional associations; pressure groups, gender, leadership; military command, queer, sexuality; praxeology, organization, social systems
  • Leadership
  • Authority and responsibility
  • Conflict resolution
  • professionalism
  • Managing diversity

Source: Compiled by the author, adapted by MCUP.      

The search refinement process was conducted by combining search term options with “and” and “or” connectors, which are introduced and discarded in a trial-and-error process until the final data set is chosen. The refinement was conducted to reduce the number of articles in each data set. In this study, the goal was to obtain a data set of 10 articles, which is consistent with the rapid assessment of the CAT. These articles would then be used to explore the previously stated path of inquiry related to relevance of scholarship to support EBMgt-MJDM integration.

Step 5. Study Selection
Because of the volume of scholarly literature available, the CAT places importance on developing a relevant data set. The selection of articles included in the data was paramount to addressing the previously stated path of inquiry related to relevance that supports the RQ. To ensure that the best effort goes into developing the data set, a procedure was rigorously followed. The key to a relevant data set is the database search and search terms. The search concentrated on relevant bibliographical databases using clearly defined search terms from the RQVS.31 The search for this study was conducted using the ABI/INFORM Collection from ProQuest and Business Source Premier from EBSCO.      

As previously explained, in this study five data sets were developed from the five planning focus areas found in the Commandant’s Planning Guidance. The intent was to test the scope of each data set in terms of its potential to inform the associated focus areas. If the data sets contained sufficient scholarly studies, the potential to incorporate EBMgt into Marine Corps planning is high. The search terms for each data set are shown in tables 3, 4, 5, 6, and 7.

Table 3. Search strings for “force design” 

String # Search string Search string Literary sources available
A Identifying personnel requirements N/A 1,516
B Identifying personnel requirements AND: organizations 11,569
C Personnel requirements AND: organizational structure 984

Source: Compiled by the author, adapted by MCUP.

 

Table 4. Search strings for “warfighting”

String # Search string Search string Literary sources available
A Defense arts and sciences N/A 2,329
B Defense arts and sciences AND: doctrine 12
C Combating terrorism AND: doctrine 511

Source: Compiled by the author, adapted by MCUP.

 

Table 5. Search strings for “education and training”

String # Search string Search string Literary sources available
A Diversity training N/A 34,122
B Professional development AND: organizations 11,569
C Motivational training AND: diversity 6,395

Source: Compiled by the author, adapted by MCUP.

 

Table 6. Search strings for “core values”

String # Search string Search string Literary sources available
A

Cultural values; diversity; professionalism; social choice

N/A 233,308
B

Cultural values

AND: professionalism 814
C

Cultural values

AND: social choice 4,778

Source: Compiled by the author, adapted by MCUP.

 

Table 7. Search strings for “command and leadership”

String # Search string Search string Literary sources available
A Leadership N/A 579,398
B Leadership AND: authority and responsibility 1,420
C Personnel requirements

AND: managing diversity

692

Source: Compiled by the author, adapted by MCUP.

     

These tables illustrate the influence of the RQVS on narrowing and expanding the field of literary sources available. The five focus areas in the Commandant’s Planning Guidance all have additional categories and subcategories. These searches demonstrate the potential for integrating EBMgt and MJDM. In the RQVS, the optional terms provide an abundance of search string compositions and permutations for developing a data set. The RQVS demonstrates a high plausibility of an integration in terms of availability and volume.

Step 6. Data Extraction
Data extraction involves the collection of information in the studies to explore the quality and relevance of literature that supports the plausibility of an EBMgt-MJDM integration. Information from each study that is relevant to the CAT question should be reported, preferably in the form of a table.32 Table 8 shows the combined data sets. These studies will be assessed for quality. An identifier for the source article appears in the far-left column. The author, article title, and publication data are in the center column. A general description of each study appears to the far-right column.

Table 8. CAT data set

ID # Author, title, and publication data Description
1

Paul J. DiMaggio and Walter W. Powell, “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields,” American Sociological Review 48, no. 2 (April 1983): 147–60, https://doi.org/10.2307/2095101.

Once a set of organizations emerges as a field, a paradox arises: rational actors make their organizations increasingly similar as they try to change them. We describe three isomorphic processes that lead to this outcome: coercive, mimetic, and normative.
2 Joel Marcus and Jason Roy, “In Search of Sustainable Behaviour: The Role of Core Values and Personality Traits,” Journal of Business Ethics 158, no. 1 (2019): 63–79, https://doi.org /10.1007/s10551-017-3682-4. In two studies, the authors simultaneously assess the role of core values and personality traits in relation to a broad set of sustainability actions, both beneficial and harmful.
3 Stephanie Weber et al., “Fostering Disability Advocates: A Framework for Training Future Leaders through Interprofessional Education,” Psychological Services 17, no. S1 (2019): 120–27, https://doi.org/10.1037/ser0000386.supp. The article discusses the importance of incorporating advocacy training into preparation programs for future psychologists.
4 Debora J. Bell and Stephen R. McCutcheon, “Moving the Needle to Promote Education and Training in Substance Use Disorders and Addictions: Special Issue Introduction,” Training and Education in Professional Psychology 14, no. 1 (February 2020): 1–3, https://doi.org/10.1037/tep0000305. This article discusses some reasons for the profession’s relatively low involvement in addictions training and service delivery. It also introduces a special issue entitled “Education and Training in Substance Use Disorders and Addictions.”
5 Janna A. Henning and Bethany L. Brand, “Implications of the American Psychological Association’s Posttraumatic Stress Disorder Treatment Guideline for Trauma Education and Training,” Psychotherapy 56, no. 3 (2019): 422–30, https://doi.org/10.1037/pst0000237. This article discusses the New Haven Competencies for Trauma Training and Practice and reviews recently developed clinical and professional practice guidelines, with an emphasis on the American Psychological Association’s clinical practice guideline for the treatment of post-traumatic stress disorder (PTSD).
6 Julia Z. Benjamin et al., “Implementation of a Cross-Cultural Simulation Workshop: Feasibility and Training Satisfaction,” Training and Education in Professional Psychology 15, no. 1 (2021): 45–53, https://doi.org/10.1037/tep0000300. This article reviews the current state of multiculturalism education and highlights the many benefits of incorporating the use of simulation-based training into multiculturalism curricula.
7 Ryszard Kałużny and Piotr Pietrakowski, “Command–Leadership in Conditions of a Military Service,” Scientific Journal of the Military University of Land Forces 52, no. 1 (2020): 23–31, https://doi.org./10.5604/01.3001.0014.0257. Based on the analysis of the reference literature and their own reflections, Polish Land Forces officers address questions of whether students of military universities are being educated towards leadership or to become leaders.
8 Michael D. Matthews, Laura D. Strater, and Mica R. Endsley, “Situation Awareness Requirements for Infantry Platoon Leaders,” Military Psychology 16, no. 3 (2004): 149–61, https://doi.org/10.1207/s15327876mp1603_1. This study identifies seven goals and multiple subgoals of situational awareness among company-grade infantry officers operating in dynamic environments.
9 Stephen P. Hundley, “The Leadership Imperatives for Assessment Excellence: Imperative #2, Attracting and Retaining Talent to Support Assessment Excellence,” Assessment Update 31, no. 3 (May/June 2019): 3–14, https://doi.org/10.1002/au.30171. This article discusses leadership imperatives for assessing excellence for recruiting rewards and professional development.
10 Kathryn Steven et al., “Toward Interprofessional Learning and Education: Mapping Common Outcomes for Prequalifying Healthcare Professional Programs in the United Kingdom,” Medical Teacher 39, no. 7 (July 2017): 720–44, https://doi.org/10.1080/0142159X.2017.1309372. This study identifies the key areas of overlap in outcomes and standards expected of selected healthcare graduates in the United Kingdom. The mapping provides a framework for informing prequalifying curricula.
11 “The Implementation Methods of Fixing the Establishment and Identifying the Personnel for Posting of the Central State Organs,” Chinese Law & Government 36, no. 1 (2003): 72–75, https://doi.org/10.2753/CLG0009-4609360172. This article examines the work in developing a clean and fair crop of cadres and tests whether the cadres will accept the overall situation.
12 Karen Lee Ashcraft and Sara Louis Muhr, “Coding Military Command as a Promiscuous Practice?: Unsettling the Gender Binaries of Leadership Metaphors,” Human Relations 71, no. 2 (2018): 206–28, https://doi.org/10.1177/0018726717709080. This article treats the scholarly practice of coding leadership through gendered metaphor as a consequential practice of leadership unto itself. Drawing on queer theory, the article develops a mode of analysis called promiscuous coding.
13 Lei Liu, Rui Guo, and Junan Wu, “A Collision-Free Motion Planning Method by Integrating Complexity-Reduction SLAM and Learning-Based Artificial Force Design,” Robotics and Autonomous Systems 100 (February 2018): 132–49, https://doi.org/10.1016/j.robot.2017.10.017. This article proposes a nonholonomic integration of the Kalman filter-based SLAM (simultaneous localization and mapping) technique and governing force design for unmanned aerial vehicle (UAV) navigation.

Source: Compiled by the author, adapted by MCUP.

Step 7. Critical Appraisal
A CAT can be used to answer many different types of questions, including those involving the affect of an intervention, a factor, an independent variable, or the antecedents of a certain outcome. This study used the CEBMa rapid evidence assessment along with Mark Petticrew and Helen Roberts’s systematic reviews in social science classification system, combining and modifying them to assess the literature in the five data sets.33 These processes used six levels of appropriateness to assess rigor transparency, validity, and reliability. In addition, an assessment of the relevance of the study to the RQ was added. The levels appear in table 9.

Table 9. Research design index

Level Research design
AA Systematic review or metanalysis of randomized controlled studies
A

Systematic review or metanalysis of nonrandomized controlled and/or before-after studies randomized controlled study

Systematic review or meta-analysis of nonrandomized controlled and/or before-after studies

Randomized controlled study

B

Systematic review or metanalysis of controlled studies without a pretest or of an uncontrolled study with a preset

Nonrandomized controlled before-after study

Interrupted time series

C

Systematic review or metanalysis of cross-sectional studies

Controlled study without a pretest or uncontrolled study with a pretest

D Cross-sectional study
E Case studies, case reports, traditional literature reviews, and theoretical papers

Source: Compiled by the author, adapted by MCUP.

In table 10, the quality and relevance assessments are provided. The six levels of appropriateness in the far-left column are juxtaposed with the level of relevance in the far-right column. The levels of relevance to the PICOC and RQ are assessed. A numeric index associates the levels of relevance from 1 to 3, in which “1” indicates a high level of relevance, “2” indicates general relevance, and “3” indicates a marginal level of relevance. The abundance of available scholarship for EBMgt-MJDM integration that is related to the paths of inquiry stated in the RQ is also supported.

Table 10. Assessment

Level Author and title Main findings Relevance to RQ
D DiMaggio and Powell, “The Iron Cage Revisited.” Implications for theories of organizations and social change. 2
A Marcus and Roy, “In Search of Sustainable Behaviour.” The authors successfully replicate previous findings pertaining to values and find that controlling honesty–humility is the strongest negative predictor of harmful actions. 1
C Weber et al., “Fostering Disability Advocates.” Strategies are described in the context of Robert Gagné’s Nine Events of Instruction and include goal development and mentorship, experiential opportunities, and didactic teaching. 1
B Bell and McCutcheon, “Moving the Needle to Promote Education and Training in Substance Use Disorders and Addictions.” The special issue includes 10 articles that speak to how training can be implemented across the training sequence to improve knowledge and competency. 2
A Henning and Brand, “Implications of the American Psychological Association’s Posttraumatic Stress Disorder Treatment Guideline for Trauma Education and Training.” The authors conclude that applying the treatments identified by the American Psychological Association’s PTSD treatment guideline may inadvertently result in poor outcomes or even harm. Furthermore, the guideline does not adequately address aspects of treatment that are crucial to training about trauma. 1
D Benjamin et al., “Implementation of a Cross-Cultural Simulation Workshop.” Among the benefits of simulation-based training are the following: 1) access to diverse standardized patients with scripts that provide consistent cross-cultural experiences for discussion among educators and trainees, and 2) reduced risk of remarginalizing by moving away from practices that rely on patients from traditionally marginalized backgrounds as sources for multiculturalism training. 3
B Kałużny and Pietrakowski, “Command–Leadership in Conditions of a Military Service.” Based on the analysis of the reference literature and their own reflections, the authors define the notions of command and leadership, pointing to differences and similarities. In addition, they try to answer the questions of whether students of military universities are being educated toward leadership or to become leaders. 1
A Matthews, Strater, and Endsley, “Situation Awareness Requirements for Infantry Platoon Leaders.” Identified seven situational awareness requirements. 1
E Hundley, “The Leadership Imperatives for Assessment Excellence.” Provided an overview of the Leadership Imperatives for Assessment Excellence. 1
A Steven et al., “Toward Interprofessional Learning and Education.” Identifies the key areas of overlap in outcomes/standards expected of selected healthcare graduates in the United Kingdom. 2
E “The Implementation Methods of Fixing the Establishment and Identifying the Personnel for Posting of the Central State Organs.” Suggests implementing a “department leader responsibility system” (bumen lingdao zerenzhi). 3
C Ashcraft and Muhr, “Coding Military Command as a Promiscuous Practice?” Seeks to move scholarly practices of leadership toward queer performativity in the hopes of loosening other leadership practices from a binary grip and pointing toward new relational possibilities. 2

Source: Compiled by the author, adapted by MCUP.

Step 8. Results
The literary sources available from the search as well as the quality appraisal suggest that there is sufficient scholarship available to apply EBMgt to Marine Corps planning.34 The Commandant of the Marine Corps has outlined five focus areas in his Commandant’s Planning Guidance. In each case, there were more than 100 sources available that have the potential to inform issues within those five focus areas. The studies were representative of five of the six research design categories used herein. The variables explored in this appraisal were centered around potential, including the following:

  1. The potential of available scholarship in terms of volume of source articles.
  2. The potential of available scholarship in terms of quality of the source articles.
  3. The potential for integrating EBMgt into Marine Corps planning.

Step 9. Conclusion
Step 9 should make a concise statement on the main findings of the CAT question.35 The RQ asks, “What is the potential of available scholarship for integrating EBMgt into MJDM processes using the five focus areas found in the Commandant’s Planning Guidance?” Because the RQ seeks to ascertain knowledge on potential, the question is answered in terms of potential, comprising “high,” “adequate,” and “marginal.” The potential of available scholarship in terms of volume of source articles is high. The potential of available scholarship in terms of quality of the source article is high. Finally, the potential for integrating EBMgt into MJMD is high.

Step 10. Limitations
Step 10 should explicitly describe any limitations and discuss how they possibly impacted the findings of the assessment.36 The CAT is a tool of systematic review that is designed to quickly assess what is known and what is not known. At the inception of the appraisal in this study, it was not known that the availability of scholarship to integrate the scholarly evidence of EBMgt into Marine Corps planning existed. Although this appraisal affirmatively demonstrated that there was sufficient scholarship available for such an integration, the quick assessment of a CAT is limited in scope. The potential scholarship was made obvious by the results of the search process. However, this appraisal was limited by the small data set used in the appraisal in comparison to the abundance of scholarship available.

Step 11. Implications and Recommendations
Once enough evidence has been found to answer the RQ, the final part of the assessment should be used to relate the findings to the background of the CAT and the PICOC, as described in steps 1 and 2.37

Implications
The literary sources available from the search as well as the quality appraisal suggest that there is sufficient scholarship available to apply EBMgt to Marine Corps planning and MJDM. The Commandant’s Planning Guidance has outlined five focus areas. In each case, there were more than 100 sources available that have the potential to inform issues within these focus areas. A sampling of studies created five small data sets for each of the five focus areas. The studies herein suggest that EBMgt has a broad nexus to military planning. The specific implications from President Biden’s directive and this study are:

  1. More research will be conducted to verify and implement EBMgt.
  2. Evidence-based practitioners will emerge in the U.S. Department of Defense and its Service components.
  3. Training for evidence-based research, policy, management, and practice will emerge.

Recommendations
It is recommended that the Marine Corps continue to explore the potential identified within this article to further validate or discount this study. This should be done through original research and more detailed systematic reviews. If the potential is further validated, it is recommended that:

  1. Marine Corps University (MCU) develop an officers’ professional development program to familiarize commanders and staff with the administration of EBMgt and the application’s systematic reviews. The target audience should be commanders and staff at the battalion level and above.
  2. MCU develop an EBMgt curriculum for graduate-level courses.
  3. These EBMgt courses be integrated into intermediate-level education.            

Warfare is still the most significant human endeavor undertaken in terms of both mental and physical impact. The MJDM that guides this endeavor occurs in ever-adapting, complex environments. These recommendations will provide military planners and decision makers with the ability to make decisions with the best available evidence per President Biden’s guidance outlined in his memorandum on restoring faith in government.      

As stated earlier, this article is the first is a series of four. It has presented a systematic review research process that used a CAT format to explore the potential of integrating EBMgt into MJDM. This research demonstrates that such an integration is feasible. The intent of the following article within this series will not be to present research, but rather to present a detailed proposal on how the integration of EBMgt and MJDM can occur. In the next article, the proposed integration will be executed at the operational art level using Joint Planning, JP 5-0, as a primary reference.    


Endnotes

  1. Joseph R. Biden Jr., “Memorandum on Restoring Trust in Government through Scientific Integrity and Evidence-Based Policymaking,” White House, 27 January 2021.
  2. Biden, “Memorandum on Restoring Trust in Government through Scientific Integrity and Evidence-Based Policymaking.”
  3. Foundations for Evidence-Based Policymaking Act of 2018, Pub. L. 115-435, 132 Stat. 5529 (2019).
  4. Eric Barends, Denise M. Rousseau, and Rob B. Briner, eds., CEBMa Guideline for Critically Appraised Topics in Management and Organizations (Amsterdam, Netherlands: Center for Evidence-Based Management, 2017).
  5. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 3.
  6. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 9.
  7. Cynthia Grant and Azadeh Osanloo, “Understanding, Selecting, and Integrating a Theoretical Framework in Dissertation Research: Creating the Blueprint for Your ‘House’,” Administrative Issues Journal: Connecting Education, Practice, and Research 4, no. 2 (2014): 12–26, https://doi.org/10.5929/2014.4.2.9.
  8. Eric Barends, Denise M. Rousseau, and Rob B. Briner, Evidence-Based Management: The Basic Principles (Amsterdam, Netherlands: Center for Evidence-Based Management, 2014).
  9. Barends, Rousseau, and Briner, Evidence-Based Management.
  10. Margaret J. Foster and Sarah T. Jewell, eds., Assembling the Pieces of a Systematic Review: Guide for Librarians (Lanham, MD: Rowman & Littlefield, 2017), 38.
  11. Harold E. Briggs and Bowen McBeath, “Evidence-Based Management: Origins, Challenges, and Implications for Social Service Administration,” Administration in Social Work 33, no. 3 (2009): 245–48, https://doi.org/10.1080/03643100902987556.
  12. Grant and Osanloo, “Understanding, Selecting, and Integrating a Theoretical Framework in Dissertation Research,” 13.
  13. Grant and Osanloo, “Understanding, Selecting, and Integrating a Theoretical Framework in Dissertation Research,” 16–17.
  14. U.S. Army Leadership, Field Manual (FM) 22-100 (Washington, DC: Department of the Army, 1999); Navy Planning, Navy Warfare Publication (NWP) 5-01 (Washington, DC: Department of the Navy, 2013); Air Force Operations Planning and Execution, Department of the Air Force Instruction (DAFI) 10-401 (Washington, DC: Department of the Air Force, 2006); Gen David H. Berger, Commandant’s Planning Guidance: 38th Commandant of the Marine Corps (Washington, DC: Headquarters Marine Corps, 2019); and Joint Planning, Joint Publication (JP) 5-0 (Washington, DC: Joint Chiefs of Staff, 2017).
  15. The intent of this article is not to introduce or explain how EBMgt works or what it is. That would take an entire article in itself. Instead, this article considers the feasibility of integrating EBMgt and MJDM as well as the research into that potential.
  16. Barends, Rousseau, and Briner, Evidence-Based Management, 12–14.
  17. Some readers might think that doctrine would preclude the application of academic/scholarly research because the integration has not occurred. In the author’s opinion, the only integration of scholarship that takes place is the development of procedures that the planners use. That integration does not allow the planning process to be applied to a specified inquiry.
  18. The presidential memorandum on restoring faith in government mandates the use of science in decision-making for all federal agencies.
  19. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 3.
  20. Norman G. Lederman and Judith S. Lederman, “What Is a Theoretical Framework?: A Practical Answer,” Journal of Science Teacher Education 26 (2015): 597, https://doi.org/10.1007/s10972-015-9443-2.
  21. Lederman and Lederman, “What Is a Theoretical Framework?,” 593.
  22. Paul J. DiMaggio and Walter W. Powell, “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields,” American Sociological Review 48, no. 2 (April 1983): 147–60, https://doi.org/10.2307/2095101.
  23. Berger, Commandant’s Planning Guidance.
  24. Berger, Commandant’s Planning Guidance.
  25. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 3–4.
  26. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations.
  27. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 5.
  28. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 6–8; and Barends, Rousseau, and Briner, Evidence-Based Management, 12–14.
  29. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 9.
  30. This process can be done manually by a human reader, but there are many methods to conduct such a literature review extraction of terms automatically. All allow the creation of codes and themes, but the author knows of none that can create codes and themes automatically.
  31. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 12.
  32. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 13.
  33. Eric Barends, Denise M. Rousseau, and Rob B. Briner, eds., CEBMa Guideline for Rapid Evidence Assessments in Management and Organizations (Amsterdam, Netherlands: Center for Evidence-Based Management, 2017); Mark Petticrew and Helen Roberts, Systematic Reviews in the Social Sciences: A Practical Guide (Oxford, UK: Blackwell Publishing, 2006); and Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 15–17.
  34. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 18–21.
  35. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 22–23.
  36. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 24.
  37. Barends, Rousseau, and Briner, CEBMa Guideline for Critically Appraised Topics in Management and Organizations, 25–27.

 


                                            

MCU Press is a member of