Evaluating nurse practitioner students’ clinical performance is a vital and challenging role of nursing faculty. Identifying gaps in student knowledge and providing individualized feedback will enhance future clinical performance. Two methods of student evaluation were compared: preceptor clinical evaluation scores and open book exam scores. The two groups of evaluators scored student performance in an initial graduate clinical course, and the scores between evaluators were compared for the same first year course over a four year period (n= 242). For three of the four years, there was weak correlation between faculty scoring open book exams and preceptors scoring student clinical performance. One year (2015) showed a significant correlation between the two evaluation methods. The two evaluation methods provide richer data for student assessment than either method alone. Additional investigation of this potential theory/praxis gap is warranted.

Clinical evaluation, Open book exams, Preceptor evaluation, Nursing faculty

Hall MA, Hoebeke RE, Wooton AK (2017) Nurse Practitioner Student Clinical Evaluation: A Comparison of Preceptor and Faculty Scoring. Ann Nurs Pract 4(1): 1075.

NP: Nurse Practitioner; SPSS: Statistical Package for Social Sciences

Evaluating nurse practitioner students’ clinical performance can be challenging [1,2]. Improving accuracy of feedback to students about their clinical performance improves future professional performance [3-6]. The aim of the study is to identify if two clinical evaluation methods are reliable between groups of evaluators: preceptors in the clinical environment and full time faculty.

Literature supports clinical case studies is another method used to evaluate student clinical decision making [7]. Case scenarios require the same type of critical thinking used in realtime clinical experiences, but do not impose the same level of time constraint as with live patients. Open book case studies provide students additional experience in evidence-based patient management, similar to clinical environments. Literature was reviewed using databases including CINAL, ProQuest Health and Medical Complete, Medline, and ERIC, with search terms of clinical evaluation, open book exams, objective structure clinical exams, preceptor grading, and clinical education. While both open book case exams scored by faculty and preceptors in clinical environments are used to evaluate clinical competence, there is a lack of empirical literature comparing the reliability between the two methods.

Evaluating the similarities in grade assignment between open book exams and preceptor evaluations can help support accuracy of student assessment. Accurate assessment of student clinical performance is vital for patient safety [1-3,7]. Nurse practitioner students are assessed for a limited number of clinical semesters. After these clinical semesters are complete, students graduate and will function independently based on state nursing practice regulations. Because of the limited number of semesters to guide and evaluate nurse practitioner students, accurate and timely evaluation of clinical performance is an important role of graduate nursing faculty.

Purpose

The purpose of this study is to compare two methods of student evaluation: preceptor evaluations in face-to-face clinical experiences to case-scenarios using open book exams. A positive correlation between the two grades would support accuracy in student evaluation. The null hypothesis used for the study was: there will be no significant difference between a preceptor’s evaluation of a student during clinical compared to faculty evaluations using open book case studies. If the null hypothesis is not supported, then additional methods to improve accuracy in clinical grading should be implemented.

Theoretical framework

Patient care provided by nurse practitioners impacts individual patients, their families, and the health of communities. John Dewey (1938) emphasized the importance of experience with learning. “The value of the experience is to be judged by the effect that experience has on the individual’s present, their future, and the extent to which the individual is able to contribute to society.” (8, page 90).

Both face-to-face clinical experiences and open book exams using patient-scenarios provide experiential types of learning. Students have to select, sort, and prioritize large volumes of didactic content essential for advanced nursing practice. Actual experience in patient management is a valuable component to sustained learning [1,7-10]. Clinical experience is critical to students for experiential learning as well as its contribution to the health of society in reducing medical error.

Clinical grade inflation

Clinical grades are often higher than student performance in other methods of evaluation [1,11,12]. This grade inflation could result from preceptors who see students routinely and get to know them well. Clinical grades are assigned based on preceptor assessment of student performance while they actively caring for patients during an assigned time frame. Preceptors push students into independent performance at different rates based on their teaching styles and are key to preparing students for safe and independent practice as family nurse practitioners [1,2,13]. Some preceptors ask students to primarily observe their interaction with patients and discuss the diagnoses and management plans as co-managers of patient care. Other preceptors expect students to function independently, evaluating student performance on their ability to collect histories, perform exams, review labs and imaging reports, develop a prioritized diagnosis list, and a management plan by using references and previous course experience. Similar to the range of diversity in clinical teaching, is the assignment of the clinical grade [1]. Faculty discuss clinical performance with preceptors, but are not there in person to determine the student’s level of independence and their ability to integrate current didactic course material into patient encounters. For this reason, clinical grades can be falsely inflated based on preceptors’ and faculty’s perceptions of student performance.

Clinical Evaluation by Multiple Evaluators

Nurse practitioner students’ clinical performance is evaluated by preceptors and faculty. Preceptors are used in face-to-face programs as well as exclusively online programs. Little is published on the inter-rater reliability between clinical and practicum evaluators. Both faculty and preceptors are an essential component of nurse practitioner education, but consistency between these evaluators has not been supported by empirical research [14,15]. A correlation between part-time adjunct clinical faculty, full time faculty, and preceptors will help support inter-rater reliability of clinical evaluation, and provide consistent and accurate student feedback [1,15]. To further support preceptors, adjunct clinical faculty, and full time nursing faculty, ongoing training is essential to focus on student learning outcomes [13,16].

Several authors note evaluation of clinical performance has greater reliability when multiple evaluators are involved and occurs over a period of time [3,5,17-19]. With the use of video recorded Objective Structured Clinical Evaluation (OSCE), multiple examiners can evaluate students’ care of the same patient. Clark (2015) conducted a pilot study to examine if multiple examiners would have similar evaluation findings of students. The pilot included four faculty teaching in a nurse practitioner program. They used a 268-item checklist to evaluate student OSCE performance. Clark’s pilot study noted faculty agreement in scoring students on mastery of skills. LaRochelle et al. (2015) noted ongoing assessment beginning early in the curriculum can identify students at risk for poor clinical performance. Students should be mentored throughout clinical courses to strengthen their clinical reasoning and evaluated strategically by multiple evaluators [20].

Burglund, Sjorgen, and Ekebergh (2012), used a model where clinical evaluation was provided by both a faculty and a practitioner employed in the clinical site. The authors collected data from faculty, preceptors employed in the clinical site, and students. This model of teaching is similar to clinical teaching and evaluation of nurse practitioner students. Both the preceptor and the faculty evaluate students by completing a standardized clinical evaluation form. Faculty is responsible for clinical grade assignment, but preceptor input is vital in identifying student strength and weakness. Their study supported the value of student learning from multiple evaluators, primarily through formative evaluation. Preceptors stated they felt valued for their contribution to student evaluation [17]. Frequency of assessment from more than one expert was perceived to be positive as well. Students were more enthused to learn and felt they had clearer ideas on how to improve when the evaluations were from both faculty and expert practicing clinicians [21].

Case-based open book exams

Open book exams are another way to evaluate students’ ability to provide safe and accurate care. Open book has been perceived as less threatening by students, which could translate into a more accurate reflection of their knowledge [22-25]. Open book exam allow more time for students to reflect on their chosen responses before submitting for grading. Time to reflect on open book case scenarios has been noted to improve critical thinking [8,26-29].

Problem-based learning, open book exam instructional methods were used by Heijne-Penninga (2012) and demonstrated greater retention of learning compared with timed multiple choice exams alone By using open-book exams, the timed constraints of multiple choice exams are eliminated, leading to greater accuracy in evaluation of student knowledge [22]. Other authors have noted no significant difference between multiple choice exam scores and open book exam scores, especially when the multiple choice exam’s content is highly clinical-decision focused [34].

Human subjects protection

Submission to the university’s Institutional Review Board was requested prior to the study. The study was deemed exempt as only secondary data analysis was performed with no use of student, faculty, or preceptor personal identifiers.

Methods

This study was guided by Dewey’s theory, focusing on the importance of clinical experience to foster learning. Preceptors provide clinical guidance during live patient encounters. Faculty provides a similar type of experiential learning, but through the use of open book exams with patient case scenarios. Both methods immerse student into clinical encounters. Patient outcomes are supported by both evaluation methods, supporting a main tenet of Dewey’s theory [8].

To determine accuracy of the two clinical evaluation methods, grades were compared. Grades assigned by preceptors on clinical evaluation grids, and grades assigned on open book case scenarios were analyzed to determine if any significant correlation existed. Both types of evaluations assessed student performance in history taking, physical examination decisions, and their ability to diagnose and manage illness. Data collected over a four year period in the first clinical semester of the nurse practitioner program was used to compare grades on open book exams to preceptor evaluations.

To foster preceptor ability to evaluate students, preceptors received information about the student

evaluation processes early during the first clinical semester. Each preceptor evaluated students’ ability to develop an accurate diagnosis and management plan based upon patient assessment. Preceptors also assessed the ability to prioritize patient needs while functioning in a limited period of scheduled time. Preceptors evaluated student ability to collect an accurate history, perform a pertinent physical exam, and establish an accurate diagnosis with corresponding differential diagnoses. Clinical performance was directly evaluated by preceptors each clinical day, and summarized twice a semester on a standardized clinical evaluation form. The standardized evaluation tool was used for each clinical evaluation and was developed based on national guidelines for nurse practitioner programs.

Case studies were developed based on diagnoses previously covered in the semester. Students were provided with the patient’s past health history, their physical exam findings, and recent lab and imaging results. Students were asked to develop a prioritized diagnosis list for each case, as well as differential diagnoses for new problems, and cost-effective management strategies. Due to the program’s online format, case studies were used as an alternative to Objective Standardized Clinical Evaluations (OSCEs) as students were not expected to travel to campus.

A five-day timeframe was given for students to demonstrate their ability to recognize essential components of the history, physical exam, diagnostic results, formulate diagnoses, and establish an evidence-based management plan. This time frame is given to allow students to reflect on these “patient encounters” prior to submitting for a grade. Students’ references were evaluated for their selection of current evidence-based literature used to develop management plans. Fifty percent of the open book exam grade was dedicated to patient management and reference use. Assigning more weight to the management plans and reference section of the open book cases is essential to demonstrate the ability to apply evidence-based clinical literature is essential for accuracy in future practice [30-33]. Faculty reviewed both evaluations, but was responsible for grade assignment of the open book cases without preceptor input.

Sample

The sample used was a convenience sample of all graduate nursing students, and Post Master’s Certificate students in their first clinical semester of a family nurse practitioner program. The sample was limited to one family nurse practitioner program. No students were excluded from the study. Students ranged in age from 25 through 64 years old. Over 40 states across the U.S. were represented, so no geographic limitation existed.

Measures

Two measurements of student clinical performance were compared: clinical grades recommended by preceptors versus open book exam grades assigned by nursing faculty. Preceptors evaluated students using the program’s standardized clinical evaluation form, while nursing faculty used a standardized grading rubric for the open book case studies.

The IBM Statistical Package for Social Sciences (SPSS) software version 21 for Windows was used to analyze the data. Data were collected from 2012 through 2015 on 242 graduate students during the first clinical semester of a family nurse practitioner program. Class cohort sizes during this period ranged from 54 to 67 students. For each class year cohort, Pearson’s correlation coefficient and paired samples t-tests were computed on student scores on their first midterm clinical evaluation filled out by the preceptor and the first take-home case-based exam graded by faculty. All statistical tests were two-tailed with alpha set at 0.05.

Limitations of the study included using only one U.S.-based family nurse practitioner program. While the study was limited exclusively to one U.S. graduate nursing program, the program is offered online and included over 40 states across the U.S. in the sample. Other limitations included blinding the analysis to the type of clinical preceptor (physician preceptor versus nurse practitioner preceptor. Comparing student clinical evaluation performed by a physician verses a nurse practitioner preceptor would be of value to identify variances. A potential concern between preceptor clinical evaluation and scores on open book exams could have been due to bias in preceptor clinical grading. Preceptors responsible for completing a formal university evaluation form may feel intimidated in being honest as they otherwise would be if they were not going to see the student in the future. Some students select preceptors they have a personal or professional relationship with prior to starting clinical courses. Having intimate knowledge of a student’s hardships may also bias preceptors in inflating clinical evaluations.

Another limitation could be the clinical evaluation tool used to assess student performance. The tool used by the program was based on essential outcomes of a graduate nursing student and national recommendations for nurse practitioner programs. While the tool is not standardized across nurse practitioner programs, it is based on the American College of Nursing Essentials for Masters programs and the National Organization of Nurse Practitioner Faculty [37].

The data supported a lack of correlation between faculty grading and preceptor grading in three of the four years of the study. Evaluation methods that are valid and reliable between evaluators are the goal, and further investigation is warranted. Open book case exams have shown promise in other studies in meeting this goal. The open book case exams used in the study were evaluated using the Content Validity Index, using four content experts, with each question on the exam rated for relevancy by each of the four experts. A Cronbach’s alpha of greater than 0.8 was the goal for the exams, and each year the index of 0.8 or above was achieved. Each open book case exam was developed based on course content as well as frequent patient diagnoses treated by students in family practice clinical settings. Each student received the same open book case scenarios. New cases were developed annually; therefore reliability of each new exam was not established. If exam cases were to be repeated in future student co-horts, reliability could be assessed.

Student scores for the midterm clinical evaluation and takehome case-based exams were normally distributed. Table 1 shows the results of correlations and paired samples t tests for each cohort by year for comparing the mean difference in scores for the midterm clinical evaluation and the take-home exams.

Table 1: Comparison of Mean Scores for Clinical Evaluation Methods.

For the 2015 cohort there was no significant difference in mean scores for midterm clinical evaluations and take-home exams [t (57) = 1.371, p = .176]. The findings in 2015 did support the null hypothesis that there is no difference in student performance. However, the Pearson correlation coefficient was low for the relationship between these two variables (r = .139). For the 2014 cohort there was a significant difference in mean scores for midterm clinical evaluations and take-home exams [t (62) = 2.414, p = .019]. This same pattern was revealed when analyzing data from the 2013 cohort [t (53) = 7.123, p = .000] and the 2012 cohort [t (66) = 7.613, p = .000]. This showed there was a difference in how student performance was evaluated on direct observation of clinical practice compared to the case-based open book exams

The same pattern of a low correlation between the midterm evaluation and take-home exam scores persisted when analyzing paired samples t tests on these three cohorts of students for 2012 (r = .227), 2013 (r = .131), and 2014 (r = .103). These results may suggest even though preceptors used the same clinical evaluation performance tool with accompanying directions for scoring, and faculty used the same detailed grading rubric for the open book case study exams, preceptors and faculty may hold varied standards, and interpret performance differently, based on their experience and exposure to student learners. Methods to improve correlation of student evaluation could include updating the clinical evaluation tool used by preceptors, making it more specific to faculty expectations, and providing additional preceptor orientation for student evaluation.

Evaluating students accurately is the responsibility of nursing faculty, including programs that are exclusively online. While the findings noted minimal correlation between the two clinical evaluation methods of preceptor evaluation scores to open book exam scores, the two evaluation methods provide richer data for student assessment. One evaluation method is in a fast-paced clinical environment and the other is in a home environment with and open book exam over a five-day period to consider patient cases. Clinical experiences in an outpatient setting are varied, while case-based scenarios are controlled between students. For these reasons, the comparison findings should be considered, but neither method eliminated from use.

Each student’s ability to perform a focused history and targeted physical exam is evaluated on a day-to-day basis by preceptors. Faculty relies on preceptors to be their “eyes and ears.” Because these two methods of student evaluation are not strongly correlated, additional methods that are reliable must be used to determine student progression. Addition student evaluation methods include weekly clinical reflection journals, multiple choice exams, and clinical management plans. Essential skills required for independent practice like precision in differential diagnosis, priority of diagnoses, and formulating plans of care that are safe and reasonable need to be evaluated [4,31,35].

While student assessment can be challenging in an online nurse practitioner program, findings of the study supported minimal correlation between preceptor and faculty assessment of students, suggesting a theory-praxis gap. The study findings raise awareness that the two methods of student evaluation do not strongly correlate. Additional means to enrich validity of student assessment should be developed in improve accuracy in student clinical evaluation. Two suggestions include mandating clinical experiences on-campus using standardized patients or using two-way video technology.

Clinical competence is vital to providing safe and accurate patient care. Preceptors and other methods to immerse student in clinical decision-making are an invaluable resources to student learning. A student’s ability to hone to a correct diagnosis and use appropriate evidence-based references is one way to provide faculty with insight to individual student decision-making. Both preceptored clinical experiences, and open book case exams, provide valuable clinical experience. Grave omissions in care can be identified during these experiences, prior to a student graduating to practice independently. Continued improvements in clinical grading methods can lead to improved evaluation techniques, translating into improved and safer patient care.

1. Fazio S, Papp K, Torre D, DeFer T. Grade inflation in the internal medicine clerkship: a national survey. Teaching and Learning in Med. 2013: 1: 71-76.

2. Oermann MH, Gaberson KB. Eval and Testing in Nursing Ed. 4th ed. New York: Springer; 2014.

3. Clark C. Evaluating Nurse Practitioner Students Through Objective Structured Clinical Examination. Nursing Ed Perspectives. January 2015; 36: 53-54.

4. Gorton K, Hayes J. Challenges of Assessing Critical Thinking and Clinical Judgment in Nurse Practitioner Students. J Of Nursing Ed. 2014; 53: 26-29.

5. Garoushi S, Taher S, Al-Tawaty A. Dental students perception and attitude towards their first exit objective structure clinical examination. Dentistry. 2014; 4: 243.

6. Oermann MH. Reflections on clinical teaching. Nurse Ed. 41: 165.

7. Weeks K, Clochesy J, Meriel Hutton B, Moseley L. Safety in numbers 4: The relationship between exposure to authentic and didactic environments and Nursing Students’ learning of medication dosage calculation problem solving knowledge and skills. Nurse Ed In Practice. 2013; 13: 43-54.

8. Dewey J. Education & Experience. New York: Touchstone. 1938.

9. Brownlee S. Situational learning in midwifery. South Africa Nursing J. 1977; 44: 9-10.

10. Kubin L, Fogg N, Wilson CE, Wilson J. Comparison of student learning among three teaching methodologies in the pediatric clinical setting. J of Nursing Ed. 2013; 52: 501-508.

11. Donaldson JH, Gray M. Systematic review of grading practice: Is there evidence of grade inflation? Nurse Ed in Practice. 2012; 12: 101-114.

12. Paskausky AL, Simonelli MC. Measuring grade inflation: A clinical grade discrepancy score. Nurse Ed in Practice. 2014; 14: 374-379.

13. Lasater K. Clinical judgment: the last frontier for evaluation. Nurse Ed In Practice. 2011; 11: 86-92.

14. Dyer J, Black A. “What do I do or say?” Guiding clinical preceptors. Nurse Ed. 2015; 40: 274-275.

15. Johnson K. Improving adjunct nursing instructors’ knowledge of student assessment in clinical courses. Nurse Ed. 2016; 41 : 108-110.

16. DeBeer M, Martensson L. Feedback on students’ clinical reasoning skills during fieldwork education. Australian Occ Therapy J. 2015; 62: 255-264.

17. Berglund M, SjÖGren R, Ekebergh M. Reflect and learn together - when two supervisors interact in the learning support process of nurse education. J Of Nursing Management. 2012; 20: 152-158.

18. Traynor M, Galanouli D. Have OSCEs come of age in nursing education?. British J of Nursing. 2015; 24: 388-391.

19. Wunder L, Glymph D, Newman J, Gonzalez V, Gonzalez J, Groom J. Objective Structured Clinical Examination as an Educational Initiative for Summative Simulation Competency Evaluation of First-Year Student Registered Nurse Anesthetists’ Clinical Skills. AANA J. 2014; 82: 419-425.

20. LaRochelle J, Dong T, Durning S. Preclerkship assessment of clinical skills and clinical reasoning: the longitudinal impact on student performance. Military Medicine. 2015; 180: 43-46.

21. Burgess A, Ramsey-Stewart G. Elective anatomy by whole body dissection course: what motivates students? BMC Med Ed. 2014. 14: 272.

22. Duty S, Christian L, Loftus J, Zappi V. Is cognitive test-taking anxiety associated with academic performance among nursing students? Nurse Ed. 2016; 41: 70-74.

23. Heijne-Penninga M, Kuks J, Hofman W, Cohen-Schotanus J. Directing students to profound open-book test preparation: The relationship between deep learning and open-book test time. Med Teacher. 2011; 33: 16-21.

24. Heijne-Penninga M, Kuks J, Hofman W, Muijtens A, Cohen-Schotanus J. Influence with PBL on open-book tests on knowledge retention measured with progress tests. Adv in Health Sci Ed. 2013; 18: 485- 495.

25. Hrynchchak P, Takahashi S, Nayer M. Key-feature questions for the assessment of clinical reasoning: a literature review. Med Ed. 2014; 48: 87-883.

26. Dunn L, Musolino G. Assessing Reflective Thinking and Approaches to Learning. J of Allied Health. 2011; 40: 128-136.

27. Bernabeo E, Reddy S, Ginsburg S, Holmboe E. Professionalism and Maintenance of Certification: Using Vignettes Describing Interpersonal Dilemmas to Stimulate Reflection and Learning. J of Cont Ed in the Health Prof. 2014; 34: 112-122.

28. Seanlan JN, Hancock N. Online discussions develop students’ clinical reasoning skills during fieldwork. Australian Occ Therapy J. 2010; 57: 401-408.

29. Stirrat GM. Reflections on learning and teaching medical ethics in UK medical schools. J of Med Ethics. 2015; 41: 8-11.

30. Blanco M, Capello C, Dorsch J, Perry G, Zanetti M. A survey study of evidence-based medicine training in US and Canadian medical schools. J of the Med Library Assoc. 2014; 102: 160-168.

31. Facchiano L, Snyder C. Evidence-based practice for the busy nurse practitioner: Part four: Putting it all together. J Am Assoc NursePractitioners. 2013; 25: 24-31.

32. Marshall T, Straub-Morarend C, Handoo N, Solow C, CunninghamFord M, Finkelstein M. Integrating critical thinking and evidencebased dentistry across a four-year dental curriculum: a model for independent learning. J Dental Educ. 2014; 78: 359-367.

33. Schmidt HG, Mamede S. How to improve the teaching of clinical reasoning: a narrative review and proposal. Med Ed. 2015; 49: 961- 973.

34. National Task Force on Nurse Practitioner Education. Criteria for evaluation of nurse practitioner programs. 5th ed. 2016