Wayne T. McCormack, Ph.D.
Professor
Department of Pathology, Immunology, and Laboratory Medicine
University of Florida College of Medicine
Gainesville, FL
With the support of the Arnold P. Gold Foundation, we developed a peer nomination survey that has been useful as a selection tool for the Gold Humanism Honor Society. A 12-question survey was tested at the University of Florida, Tulane University, and University of Michigan during 2003. Factor analysis of the peer nomination survey data revealed three significant factors by which medical students could be ranked for the identification of a top 10-15%: clinical competence; caring (interpersonal skills & humanism); and community service. Based on these results, a shorter, 6-question survey (please see link below) was developed for testing at the same three medical schools in 2004. In other research, we have demonstrated that simply counting the number of nominations for subsets of questions can be used to generate virtually the same rankings as those generated by factor analysis, thereby simplifying the data analysis using this survey. The results of this work have been published:
McCormack, W.T., C. Lazarus, D. Stern, C.B. Stevens, P.A. Small, Jr. 2007. Peer nomination identifies medical student exemplars in clinical competence and caring at three medical schools. Academic Medicine 82(11):1033-9.
Please note that this peer assessment format (peer nomination) does not provide detailed information on every student. Peer rankings can provide data on every student, but tends to be less discriminating (e.g. everybody is “above average”). The greatest utility of peer nomination is in effectively discriminating the extremes within a group, i.e. the top ~15% of a medical school class in this application.
How and When To Administer the Survey:
Peer nomination forms may be administered as web-based or paper forms. You may get a better return rate when administered as a paper form at a meeting of the entire class, when the purpose of the survey can be explained, instructions given, and sufficient time given to fill in names. Web-based forms are convenient to administer and have the advantages of being available at the students’ convenience and data collection & reporting may be automated, but medical students often need multiple reminders to complete the form in a timely fashion. If on-line evaluation is part of your local culture, then the use of web-based forms can result in good response rates (85-99%).
The survey period can be at any time, but we believe that good peer assessment requires good prior peer interactions, so we administer the form as late as possible to have the data in time for GHHS selection and preparation of MSPE’s, which is during the summer preceding the fourth year at Florida. The instructions should explain briefly what the data will be used for at your institution, e.g. recognition for ~15% of the class by election to the GHHS, mention in the the MSPE, other honors or awards, etc.
How To Analyze the Data:
Factor analysis can be used to analyze relationships among survey items and to identify latent variables or “factors” that account for correlations among the survey items. Factor analysis can also be used to assign scores to individual students based on the major factors identified. For many years we have used the SAS factor procedure and principal components analysis (SAS Institute Inc, Cary, NC), with an eigen value greater than one to identify major factors. The Harris-Kaiser rotation was used to determine standardized regression coefficients, which were used to plot rotated factor patterns. The SAS score procedure was used to list student scores for each factor, which were then sorted in descending order to determine student rankings for each major factor.
If you do not wish to use factor analysis, we have shown that you can obtain virtually the same rankings of students by simply counting the number of nominations that each student receives for each question. It is not necessary to ask the students to rank their nominations 1st, 2nd, and 3rd, as it will make no difference whether you weight the nominations or simply count them. You can sum the numbers of nominations for subsets of questions to arrive at rankings for the factors that have been identified by factor analysis.
Factor 1 – Clinical Competence: Questions 1, 4, and 5
1. The classmates you would like to have work at your side in a medical emergency.
4. The classmates who would be the best choices for a highly desired residency. Residencies for future graduates will depend on these persons’ performance.
5. The classmates you would want as the doctor for yourself or a loved one.
Factor 2 – Caring (Interpersonal Skills/Humanism): Questions 2, 5, and 6
2. The classmates who best personify the quote “the secret of good patient care lies in caring for the patient”.
5. The classmates you would want as the doctor for yourself or a loved one.
6. The classmates who have the best listening skills with patients.
Factor 3 – Community Service: Question 3
3. The classmates who have shown exceptional interest in service to their communities.
As described in our manuscript, Question 5 is interesting, as it has characteristics of both Factor 1 and Factor 2.
What’s With These “Factors”?
Review of the peer assessment literature reveals that two major characteristics are consistently identified by factor analysis of peer ratings by medical students, residents, and physicians: (1) medical knowledge or technical skills; and (2) interpersonal skills or patient relationships. Our peer nomination results for three medical schools revealed the same pattern. We call Factor 1, which accounts for the most variance, “clinical competence”. One of the goals of our Gold Foundation-sponsored study was to determine whether medical student traits related to humanism and professionalism in medicine, such as caring, respect, and altruism, would be identified as an independent factor in peer assessment. We found that new peer nomination items related to humanism and professionalism factored with interpersonal skills, prompting us to suggest that a better description of Factor 2 may be simply “caring”. We concluded that medical students display their humanism via their interpersonal skills, thus humanism could not be identified as an independent factor. Community service was identified as a less significant third factor.
So Which Data Do I Use for GHHS Selection?
That, of course, is the $64,000 question, and one that your GHHS selection committee must decide. Your selection committee should decide how to use the data, and what additional data you want to use in your selection process. Some medical schools rank students based on the grand total of nominations each student receives for all of the questions. Others, including UF, rank students for each of the three factors, and consider all three rankings, as they provide information about different aspects of being a humanistic physician. We also consider other data, such as peer ratings of professional behavior, faculty ratings of professional behavior during clerkships, and personal knowledge about the students among the selection committee members, however the most weight is given to the peer nomination rankings.
Opinion:
I am posting a few comments about my personal opinion, because it is a question I am often asked about how to use this peer nomination survey. Obviously you and your GHHS selection committee should make your own decision about how to incorporate such data into your selection process.
We have found that one can be overloaded with data when you compare various rankings and other student data. Different subsets of students are identified when students are ranked separately by Factor 1 and 2, with a few students appearing on both lists of the top ~15%. Students who are ranked in the top ~15% for both Factor 1 and Factor 2 are outstanding candidates for GHHS selection. Beyond them, which rankings are the most useful? After all, Factor 1 and 2 are both important – the best physicians are both clinically competent and caring. Doctors may be extremely competent at medical procedures but not very caring, and that might be OK in some situations, but I don’t think that is the kind of future physician we are trying to recognize. The question is how to select students who are ranked high according to one factor but not others. This is where additional data comes into play.
I believe that the peer nomination process is a good way to get reliable data about caring behavior, and that the Factor 2 ranking may be the most relevant as a GHHS selection tool. We expect all of our medical students to be clinically competent, and peer nomination can identify the standouts. Other measures of performance and behavior can help ensure that those ranked high for Factor 2 (caring) but not standouts for Factor 1 (clinical competence) meet the standards you desire for GHHS selection.
Another debate your selection committee may have is how much emphasis to place on Factor 3, community service. This ranking provides additional information about one aspect of humanism in medicine. If a student is ranked high for community service but does not appear in the top ~15% for clinical competence and/or caring, other sources of information should come into play in your committee’s decision about GHHS selection. This ranking may also be useful for selection of students for other awards specifically dedicated to recognizing outstanding community service.
What Happens If We Change the Survey Questions?
We have tested several other survey questions, and most of them displayed remarkably similar factor characteristics at all three medical schools. However, some were not consistent across all three medical schools, such as those related to likeability, discussing personally disturbing events, and respect. Based on those observations, the six items on the current survey were selected because they best defined each of the three factors and resulted in consistent results by factor analysis at all three medical schools. If you are concerned with having some uniformity with other medical schools in the selection tools you use for the GHHS, I would certainly recommend using the survey posted here, as we now have good data from three different medical schools to validate it (soon to be published we hope!).
It is inevitable that some medical schools will change the survey questions to best fit their local culture. If you plan to substitute questions, my guess is that you are trying to tweak the survey to get at elements of humanism you feel are most important for your institution. If you plan to rank students according to individual factors, I recommend performing factor analysis to ensure that new questions have the characteristics you desire (and we would be happy to help you with that if you cannot do it yourself). Based on our experience and results, I predict that most substitute questions that ask about behavioral characteristics related to caring, altruism, and empathy will be weighted towards Factor 2 when subjected to factor analysis. Adding different questions will likely have less impact on the survey results than substituting questions. If you plan to sum all of the nominations into a single peer nomination ranking, ignoring the individual factors, changing the questions may have little effect.
Several medical schools currently use this survey, or variations of it, as a selection tool for their chapters of the Gold Humanism Honor Society. Please feel free to use this survey at your institution. Reference to our work (when published) would be appreciated. If you have any other questions about how best to administer the survey or interpret the data, please contact Dr. McCormack at (352) 273-9042 or by e-mail at mccormac@pathology.ufl.edu.