“Look at every path closely and deliberately, then ask ourselves this crucial question: Does this path have a heart? If it does, then the path is good. If it doesn't, it is of no use.” ~Carlos Castaneda

Sunday, January 6, 2008

Are we too nice?

“High achievement always takes place in the framework of high expectation.”
~Charles F. Kettering

When I went through all of our applications, it was hard to separate one application from another. The white pages and black lettering blended together after the about 15th application. For a majority of the applications, all I had was a name, school, and their basic statistics.
John Doe
  • Medical School: State University of X or X University
  • USMLE step 1: 235
  • Clinical Rotations: 1/2 clinical honors
  • Orthopaedic Rotations: honors
  • LOR's: good to excellent with some stating he is in the top 10% of students rotating this year
  • Personal Statement: "... Since I injured my knee playing football, I have always wanted to be an orthopaedic surgeon. ... I have played sports all of my life ..."
I can't count the number of times that I have read this type or similar application. The names of the applicants are sometimes interchangeable. What aspect of their application tells me that this is going to be a stud or a dud? I have received a number of emails and have read plenty of forums that ask the same question, "what do I need to get into orthopaedics?" If I could tell you the exact recipe, I would, but I don't think there is one. In truth, we all know the recipe, good grades, good scores, good rotations, and a little luck. I think what is more important is the special ingredient or special sauce you bring to make you different.

Many applicants get caught up in the numbers of the game. Time and time again the question is asked, "what score do I need to get into orthopaedics" or "what grades do I need" or "how much research do I need"? I can give numbers of the typical solid application, as I have listed some above, but does that guarantee a spot? Not so much. The quoted figures on scores, grades, research, and AOA are just guidelines, not a guarantee. Every year there are applicants with strong numbers that don't get into a residency. So, why didn't they match? Typically, when I have reviewed these non-match applicants, the reason was either glaringly obvious (USMLE score 205, bad letters, failing a subject in medical school) and on other occasions, it was not. When there was no glaring flaw, the applicant looks like every other applicant. There is nothing in their application that makes him/her stand out.

I have asked myself why is it that that most of the applicants so similar. Is it because they have all used performance enhancing drugs to make them all academic superstars? All jokes aside, I think it may have more to do with the way we grade. I blame it on kids soccer, where everyone gets to play and in the and they all get a trophy. Many parents display their honor role student sticker on the rear window of their SUV. All the children are A and B students. Many go to a 4 year university and expect the same. They argue for a better grade and petition for grade changes when it doesn't meet their expectations. The professors that grade on a true Bell Curve are not liked or considered "hard" because they give out fewer A's and B's. Over the past 10 years, I have noticed this trend and I have wondered if we (educators) are too kind in our evaluations?

With a student's application, we receive a copy of the grade distribution for that medical school. When comparing applicants from one school to another, there are definite differences in grading philosophies. I don't know if this is a problem with the grading set up or that we are "too nice." In my experience of clinical grading, unless the student does something drastically wrong (like never show up or cursing out a patient), the student will at least receive a pass. The question is who receives the highly coveted Honors. Each School varies in their grading system. The grades can range from pass/fail only to honors/pass/fail to honors/high pass/fail to honors/high pass/marginal pass/fail and the always popular A/B/C/D/F (with +/-). What puzzles me about all of these systems is that the average tends to be a B or high pass. There are schools with greater that 50% honors in some subjects. You may say, well are these the "lesser schools"? Not so fast young patawan. In my limited research (okay not really research but observation), it is more common for the "very competitive schools" to have more of a top heavy grade distribution and the "less competitive schools" to have a more even grade distribution. It is not uncommon for a school to have grading distribution (in the clinical years) with greater than 50% honors and less that 30% passes. How does this allow for us to assess these applicants? If you score only gives out 20% honors and you received a high pass, should you be penalized? On the other hand, if you went to a school that gave greater that 50% honors, should you be given bonus?

With competitive specialties' concentrating on USMLE scores, students have been crushing this test. The USMLE is one of the only tools we have to compare applicants from different schools and areas of the country. Because the USMLE "powers that be" don't want the test to be used in the manner we use it, they do not provide us with the distribution of scores. In the old days, the mean was in the low 200's (205 when I took it) with a standard deviation of 20. Today, the mean is in the mid 210's. Therefore, a score in the 90's of 225 is equivalent to a score of about 235 in today's scoring (I am guessing). Most of the applicants I have reviewed have an average of a 230 (just a guess, again no true data). Again, when trying to create separation like Randy Moss from a corner back, it doesn't happen. The applicant's are all bunched together like 6 year old children playing soccer.

You may say, "then look at the letters of recommendation (LORs)." This is less helpful than the grading. Most folks have the prerequisite letter from their program chair that says he or she is a supernova or has star like qualities. There are usually 1-2 letters from surgeons that are not known by most interviewers and 1 from a well known surgeon. Although the letters are helpful when pointing out top end and lower end, they to not create the needed separation to differentiate one applicant from another. There have been occasions where I have read the same recommendation on 2 or 3 applicants from the same physician. Although we think we know the code words, I think we kid ourselves at thinking we can read into another's recommendation like it is Morse Code. Usually the true meaning is missed, except when comments are blatant like, "we recommended that he look into other specialties ... "

What is the answer? I have recently begun to reevaluate my own grading system. How is my grading? Am I too nice? The answer is yes. I believe that many of us don't want to be the bad guy. Who wants to be the professor who fails most of his/her students? I don't think that there are many who would answer yes. I believe we do need to re-center. In the clinical setting, the average grade should be a pass. The excellent grades should be give to those who truly stand out for the rest of the students. As an educators, we must communicate our expectations are and explain what passing grade means. Is this a student problem, I would propose it is not. It is a educator problem. We have evaluate honestly. No more just checking the 4 out of 5 box. If they have met expectations, then they should get a pass. You may read this and think I am arguing for more strict grading, but I am not. I think that our grading should be fair. Lumping the average around above average is not fair to those who are truly above average.

“Success is simple. Do what's right, the right way, at the right time.”
~Arnold H. Glasgow

14 comments:

Anonymous said...

While I agree with some of your points, I think your perceptions are clouded by the fact you are looking at orthopaedic applicants who are by and large the cream of the crop. If you could look at the average or even star applicants to a psychiatry or family medicine residency, you would see quite the difference. A single honor would make many of them stand out, along with a Step I score above 210.

While certain applicants will certainly make better residents and surgeons, for the most part there are not many truly bad ortho applicants, I'll put the 10 random ortho applicants up against even 10 random gen surg applicants any day in terms of residency performance. Who I feel really bad for is the attendings in the less desired fields who really have to find diamonds in the rough.

Someonect said...

anon: thank for your comment. i think you are right. we do have a skewed sample. that being said, when i speak to my college professor friends, they do have similar opinions. may be it is a generational thing, i am not sure.

in our program, we look for the diamonds in the rough. we look more for a fit than a number. i think by the numbers many orthopaedic applicants blow away those from other specialties, but the question is who will do well in your program. who has the complete package?

the sad thing is to see good applicants not match. why? usually because they didn't stand out enough.

Bongi said...

beautiful

Midwife with a Knife said...

Your statement is so true. Lumping everybody around the above average mark does a real disservice to those who are truly above average.

Anonymous said...

Hmm… this entry frustrates me (due to it’s accuracy). Of course, there is a problem with the grading system. However, I’m not sure the answer to rectify it is as clear as “grading harder”. There are only 5 grading categories. Due to the shear quantity of students you will only “shift” wherein the majorities lie. Yes, it is true one would elicit the few that are truly gifted, which emphasis of few. Because these few are very sparse, you will now have to discern from equally large group, just with lower scores. What has this accomplished?

Someonect said...

anon#2: i was waiting for a comment like yours. i knew that someone would read this as if i was making a statement to grade harder. it is not. it is to make average, average. to make above average, above average. i vote for a normalized curve. having seen the grade distributions for many different universities, i can tell you that there are many schools that have near honors, or high pass (meaning B) as the average for a class. that is why the distribute the grade distributions for each school. there are some schools are very strict on honors, only allowing 20-30% honors.

it is interesting that this comment would come today. today we interviewed applicants. my chairman made a comment about a applicant's grades, the schools grade distribution, and grade inflation. (he did not read this entry).

there is grade inflation. i would just like to see fair grading not harder grading.

Anonymous said...

I don’t think I read into your entry and equated grading fair to grading harder. If one as an individual starts “grading fairer” this would perceived to others “as grading harder” and in the long run doing your students an injustice. Their passes compared to others who have honored (graded by someone else) might cause them to be overlooked for a slot. Would that be fair?
anon#2

Someonect said...

anon#2: maybe i should say schools should all grade equally. the differences on grading already exists. speaking only for the clinical grading which is more subjective (with objective components), not every school's honor or high pass is the same. my school only had only pass/fail for a while, how does that compare to a school with honors/pass/fail? for the schools with only honors/pass/fail, how do they compare to the schools with honors/high pass/marginal pass/fail? is the one school's high pass better than my school's pass?

i have just looking at a number of school's grade distributions, i can tell you that a common grade distribution is 20% pass, 50%high pass, and 30%honors. when i see that the high pass means nothing, it is the average, and a pass is below average. on the other hand, there are schools with very strict grading, 20% honors, 30% high honors, and 50% pass. is this not injustice to students? why would the schools need send out the grade distributions with an applicants grades?

i think we should be fair. we should not inflate the grade. if the do what is expected, pass; if they did more, high pass; and if they rocked, honors.

for our orthopaedic oral boards, they actually normalize the examiners. if there is someone who is always a high grader, they normalize the scores; and conversely, if someone is a hard grader, they do the same. it is almost like they a handicap in golf. i wish we could do this for medical school subjective grading.

Anonymous said...

I see your point. I also want to apologize. It’s been an arduous week, and my comments above might have seemed a bit harsh. I look forward to reading your blog entry’s when time permits. Keep up the good work.
anon #2

Someonect said...

anon#2: no apologies needed. differences of opinion are appreciated and encouraged.

Anonymous said...

The problem with grading harder is that the separation between the students you speak of is so small that the grade distribution would be large even though the ability and achievment distribution is so small. If you have an exam with an average of 85 but a std dev of 4 then it is obvious that the students are all lumped together. The difference between an A (one std above) and a C (one std below) is little more than a few questions. In a situation with such a tight distribution how can you make the difference between a C and an A so small? Additionally, should a med student attending a tougher med school with more intelligent students lose all his honors when he could have gone to a less competitive med school and had a greater chance at achieving higher scores? If inflation were removed, the quality of med school attended would have to be next to Step 1 score for determining factors. Thus, the residency program would have to accept students with lesser marks at greater schools. When a residency program lists the statistics for their acceptances this clarification will not be present and their stats will appear worse. Additionally, a medical school that grades harshly while other inflate will not be a popular choice for undergrads. A medical school that eliminates inflation would also appear worse statistically if the rest of the medical schools did not stop inflating grades. I believe the solution lies in the best medical schools initiating the elimination of grade inflation. The trend can only start in a top down fashion because the prestige and name recognition of these medical school will allow them to overcome the period when they are the only schools without grade inflation.

Someonect said...

anon: i don't think i said we should grade harder, we should grade fairer. meaning average should be average. the mean grade should not be a B. If things are to be graded on a curve, it should be a normal distribution.

as far as honors go, for most of th so called less competitive schools is is harder to get honors than the so called more competitive schools. there are top 10 schools who give out greater that 40% honors on some rotations. is that fair to the less competitive school that gives only 20% honors?

Unknown said...

Howdy,
Just started reading this blog, think its FFT :), and have caught up to January. I thought I'd chime in on the subject of grade deflation.

To start with I definitely agree with you in premise. Rampant grade inflation starts in high schools and extends all the way through the highest levels of education and it is bad for everybody involved. The problem though is that addressing it is a classic 'tragedy of the commons'ish situation. If one school, at any level, puts in a grade deflation policy it is going to hurt their students prospects upon leaving that school, even if it improves the educational atmosphere within the institution.

I went to undergrad at a top ranked college that instituted a grade deflation policy (though not a true return to average = C grading). As far as the academic experience was concerned, the grade deflation policy definitely pushed people to work harder and did help to differentiate top achievers from the rest of the pack. From a med school/law school etc. admissions standpoint, the grade deflation policy definitely hurts students who are being compared against students from other institutions with generally higher GPAs. Whats the answer? no idea, but I think that short of making a certain grade distribution mandatory for accredation, I don't really see any solution for grade inflation in med schools.

Anonymous said...

I propose actually reporting usmle scores as strictly pass/fail. Then, much like the corporate (real) world candidates would be interviewed based on demonstrated commitment to a field as well as skill set. We are to awash with applications to not do this. Screening candidates based on numbers and scores does a disservice to the system as students are more focused on numerical achievement rather than personal and intellectual growth, growth that would serve everyone once in a residency and beyond.