Kafka Goes To Washington

[Yeats]

I think that at the end of the day, the only rational response to this process is that it is not a rational process, because in all likelihood, the testing process has no external validity and no inter-rater reliability. Indeed, I believe that the lack of transparency at OPM about the process shows that they realize this as well. How else can explain their unwillingness to provide any breakdown in the score, any answers to the questions they ask on the written test, or even to provide any real guidance on how one might appeal a test result?

The score I received that puts me in the conjectured upper quartile, but how this happened is quite beyond me. My answers on the lifetime achievement portion of the exam were no longer than five sentences—on the premise that if you have real achievements it should be readily apparent. This, according to general consensus, is a bad strategy.

On the written portion of the exam, I did not understand that there was an automatic return set until half-way into the second paragraph of response. The end result was a half page of word salad sentences, which in theory should not have made a good impression on the graders.

In the structured interview, at one point, I informed the interviewers that I thought the question they were asking me was poorly worded and reflected a set of assumptions that showed it could only have been drafted by a non-lawyer—a response that I thought was true to my feelings of extreme frustration with an interview that was really not an interview and if one, certainly not one calculated to produce much in the way of reliable data, but hardly diplomatic.

I know many gifted state administrative judges in a number of states who are far more able than I am but who have received inconceivably low scores given their intelligence and their experience. I have little doubt that a lawyer in ODAR for 25 years is infinitely more qualified for this job than I am.

There is little question in my mind that this process represents the height of bureaucratic arbitrariness and silliness and that there is little gained in trying to understand or to game it. It is sort of like trying to figure out a winning strategy for roulette. I think the seemingly random results this testing process produces say far more about the people who created it then they could ever say about the people who are participating in it.

[pm]

The basic OPM process is quite transparent. It is the scoring that is not transparent. That people believe they are entitled to transparency in the scoring is interesting to say the least. This entire process is more transparent, and more fair, than about 99% of the hiring decisions made outside government.

There is no general consensus on how to write an AR accomplishment. One of my answers was quite short also. I'm not quite sure why you think there is a consensus. There is not.

With respect to your ALJ friends and their scores, NO ONE was scored based on their experience nor on their IQ. You were scored based on your answers to questions. No one was scored higher because you felt they were gifted. It is a sad truth that some experienced state ALJs thought that their experience alone would buy them a good score. In that conclusion they were incorrect.

People keep posting that they don't understand this and that regarding the testing process. That is very true - they do not understand. It might however be more productive to learn the process rather than complaining about their failure to understand it.

There is nothing kafkaesque about this process. We just don't know how they score. That's it. But I bet if you knew, there would be very little unusual about it. OPM tried to make this process simple for OPM and I'm quite sure they did.

As for randomness, I have seen no randomness and I have been following this process closely on this board and among my many friends who are involved in it (including my wife).

What I have seen is people who took little time and little effort to understand the process who now feel justified in complaining about their own lack of understanding.

[pm]

Here's something that might raise some kafkaesque concerns:

I know of thousands of people who all had the same training, the same experience, and most were quite bright, yet when given the same tests, they had widely varying scores. Scores ranged from the 40s to the 90s. Sometimes, the highest scores did NOT go to the brightest test takers. What was this bizarre and irrational process? Who were the evil doers that did not grade all tests the same?

IT WAS LAW SCHOOL (and later the bar exam)

So believe it or not, people with the same background, the same experience and the same training, and even the same IQ, can have very different scores, because they answer the questions differently. It does not mean that the people doing the scoring were stupid or evil.

[Yeats]

If you do not know how they score the test, and you do not know whether the test has any predictive validity or inter-rate reliability, then how can you possibly say that the process is fairer than “99% of the hiring decisions made outside government?” And given that we are talking about a lifetime appointment here with very little ability seemingly to punish or dismiss poor performers, perhaps, government should not be setting the bar this low? In private industry, one has the ability to get rid of hiring mistakes. It does not look to me that you can do this with an ALJ.

And if you do not know whether the test has both internal and external validity, then you certainly cannot exclude the hypothesis that it is producing random outcomes. The fact that you have “seen no randomness” is not surprising since it fairly common for people to impose patterns on outcomes which are truly random, e.g. the hot hand in basketball. And even if your perception is correct, and the process is not producing random outcomes, then that still leaves open the question about whether, these patterned outcomes correlate in any way to job performance.

If you are correct when you say that “OPM tried to make this process simple for OPM and I’m quite sure they did,” this is hardly a ringing endorsement for the folks at OPM and certainly seems faintly reminiscent of the intentional distortion of reality by powerful and anonymous bureaucrats that so frightened Kafka.

I think you should be very careful not to impliedly mischaracterize my position. I do not think that the people who designed this process are “stupid or evil.” I think the process seems quintessentially bureaucratic and that just because a process produces a result does not mean the result correlates with much of anything except its ability to produce a result.

Having gone through this process, I certainly understand how it works so far. What I do not understand is how it correlates in any way to making valid choices among candidates, i.e. predicts job performance, and your response has done little to increase my comfort level.

[nonamouse]

The OPM process is generic for all 30+ agencies that hire ALJs. The goal is not for OPM to have their process making predictions about job performance at a specific agency. The agencies that ask for OPM to send them a list of names of people who are "qualified" at some basic and generic way for the position will make the determination of how to choose between the many "qualified" candidates. It is the hiring agencies who sift through the remaining pile (after OPM culls out hundreds of people) to look for candidates who can do the job for their particular agency. I have had conversations with "oldtimers" who have been involved in the hiring process as interviewers/selectors at a hiring agency and they agree that the OPM score merely gets one's foot in the door. Unless a vet/non-vet head-to-head comes up, the agency can skip over a high scorer from the OPM process and hire someone they believe is better suited to their particular needs.

Edmund, I don't understand why you believe that OPM should provide the scoring methods to the public. I have worked for various governments my entire adult life (city, state, fed) and none of them made their hiring methodology known to candidates or the general public. Government work often requires crossing into Bizarro World and learning to function in that environment. If OPM is irritating you now with their lack of transparency, then ODAR would likely make your head pop off or give you a stress-related condition. There are things that I shake my head about daily at ODAR. They should pass out copies of the Serenity Prayer at the door. Some of the very characteristics that have helped people to excel in law school or in a litigation practice will not serve them well in an ODAR office.

Good luck with the rest of the process and think carefully about whether you want to come through the looking glass to our side where transparency and "fairness" are not generally priorities. Even if you have worked in an ODAR office as an attorney, the view from the other side of the bench is quite different.

Don't get me wrong. I enjoy my job, but I recognize that it is not a good fit for many people who are excellent attorneys on the other side of the bench.

[comfortablynumb]

PM, I am new to posting on this board but certainly not new to reading the multitude of postings over the last few years (all with an eye toward gaining an understanding of the process). My viewing pleasure was only one aspect of my attempt to understand the process. With all due respect, your opinion on the transparency of OPM's testing process, is nothing more than a distinction without a difference. The process cannot be transparent when the all-important end product, the scoring, is cloaked in mystery.

As Mr. Burke has pointed out, this is further evidenced by the inability for an individual to even know what portion of the process to appeal, if any, because of OPM's lack of disclosure on the scoring of the various components. Perhaps an explanation for OPM's determination to not provide detailed scoring results or usable appeal guidelines can be found in the following quote from Mr. Burke at a different time and place: "They defend their errors as if they were defending their inheritance." By the way, you also make an assumption as to the nature of Mr. Burke's association with the various state ALJ's with whom he has come in contact. To have knowledge of or respect for an individual does not make them a friend.

In my opinion, your position that there is no visible randomness in the process is simply incorrect. I certainly did not realize that the OPM process is such an exact science as to be able to quantify, for example, how one person might have received a 76.90 total score and someone else a 75.90, a 65.90, or even a 55.90, when the individuals conducting the SI portion varied greatly and undoubtedly came with their own set of biases and preconceived notions of what might constitute appropriate answers to a "structured" set of questions.

Finally, no doubt there are individuals who have posted on this board that clearly support your observation of the lack of time and effort expended by some in attempting to understand the process. However, it is also my opinion that the scoring is very arbitrary and I have made a concerted effort to gain knowledge of the process and the end result. I suspect many others with similar opinions of the whole OPM/ALJ register process have done the same.

[elizjuge]

Mar 3, 2009 0:41:44 GMT -5 nonamouse said:

Edmund, I don't understand why you believe that OPM should provide the scoring methods to the public. I have worked for various governments my entire adult life (city, state, fed) and none of them made their hiring methodology known to candidates or the general public.

I have seen the same thing. With significant experience working on federal employment cases, and expert knowledge of federal hiring practices, I can tell you that the federal agencies have a crediting plan for each job announcement it posts and it is never made known to the applicants/public. It's often known only to the HR specialists doing the review--not always to the selecting official. So it's not surprising that the ALJ exam scoring methods are not disclosed. Nor do I imagine that they will be. Doing so would give future applicants all the "answers" to the qualifications and competencies that they are looking for.

And I agree with other posters who reference the subjectivity of the reviewers. I have seen HR reviewers with the exact crediting plan rate an applicant's application in two totally different ways just based on the subjective interpretation of the crediting plan.

[jagghagg]

Careful, careful, EdmundBurke - you are wading into the territorial waters guarded with great ferocity by PM and a select few. They do not seem to ascribe to the admonition from Gunter Grass that "the first job of a citizen is to keep your mouth open.” For them, "honest criticism is hard to take, particularly from a relative, a friend, an acquaintance, or a stranger."

[fisherwoman]

This is a bizaare process. Take it for what it is and move on. I hope the hiring numbers are such that .25 difference among us gets us all jobs, for which we are all suited. I hope, in turn, we make better decisions and a better system. That's what this ultimately should be about.

Me? I am going to send suggestions about this process to the new President's still extant hiring team. I think HR ought to be abolished in most Federal Agencies, as well as the private sector. The HR people are generally people who do not have a clue about who they are hiring beyond dummed down matrices that applies a methodology to an other wise prejudiced and irrational process. Was Wall Street truly filled with the "best and brightest"? Or the Wall Street law firms that colluded to allow the financial meltdown? Hey guys. This is no more rational or irrational a process than anything else out there that purports to be "fair" or set standards. In MHO, all of that was lost a long time ago.

[jcse]

I think we are all old enough here, and have had enough experience, to be honest with ourselves. That is what makes it so difficult. If I had winged it through the whole process, I could honestly say, "Well what did you expect?" when I got my score. But the truth is, I prepared for every segment of the exam, the same way I prepared for taking the bar or preparing for a trial. The more specific people get in describing their experience on the exam tells me that there was randomness in the scoring; and that is what is so unnerving.

PM, you have said in other posts that you are graded based on how you answer. That has already been proven wrong, based on the experience of Edmund and others here. You also told someone the reason they got a low score was because they had worked for ODAR for 25 years. But that was wrong too~many with ODAR experience got offers last round. You have also admitted above, "We just don't know how they score". So do you understand what the process is anymore than the rest of us?

We have a wealth of legal experience and wisdom on this board, why should we not keep asking questions and seeking answers? De-mystifying the scoring process for some is no different that zealously representing a client or winning a difficult case. Some of the greatest cases I studied in law school were about people that got the proverbial door slammed in their face and challenged the process. This is what we do as lawyers, and we owe it to ourselves to find the answers.

[pm]

Mar 3, 2009 9:40:58 GMT -5 jcse said:

I think we are all old enough here, and have had enough experience, to be honest with ourselves. That is what makes it so difficult. If I had winged it through the whole process, I could honestly say, "Well what did you expect?" when I got my score. But the truth is, I prepared for every segment of the exam, the same way I prepared for taking the bar or preparing for a trial. The more specific people get in describing their experience on the exam tells me that there was randomness in the scoring; and that is what is so unnerving.

PM, you have said in other posts that you are graded based on how you answer. That has already been proven wrong, based on the experience of Edmund and others here. You also told someone the reason they got a low score was because they had worked for ODAR for 25 years. But that was wrong too~many with ODAR experience got offers last round. You have also admitted above, "We just don't know how they score". So I am not sure you understand what the process is anymore than the rest of us.

We have a wealth of legal experience and wisdom on this board, why should we not keep asking questions and seeking answers? De-mystifying the scoring process for some is no different that zealously representing a client or winning a difficult case. Some of the greatest cases I studied in law school were about people that got the proverbial door slammed in their face and challenged the process. This is what we do as lawyers, and we owe it to ourselves to find the answers.

The people here who have "described their experience" this year have mostly complained that they don't understand the experience. And it is absolutely clear that they do not. The only thing that proves is that they do not understand the experience. It says nothing about the process itself.

The fact that you prepared means what? That you are entitled to a certain score? Again, I return you to law school. Everyone I know prepared. Everyone got different scores. What does that prove? That everyone is different and that everyone gave different answers. It proves nothing about any alleged problem with the process or the scoring by the test givers, absolutely nothing.

You state that "The more specific people get in describing their experience on the exam tells me that there was randomness in the scoring; " People are not specific at all about the testing because we have yet to see a single answer. They have only complained that their experience should entitle them to a certain score, and it absolutely does not. We have seen no specificity at all to enable us to judge or compare the test answers. We have seen no evidence of any kind to suggest randomness in grading answers because we have seen no answers.

You state that I was wrong about someone being scored low by OPM because they had too much time with ODAR. You try to prove this by showing that some people with ODAR experience were hired by ODAR. You confuse two separate concepts, as many are doing this year. ODAR is not OPM. OPM clearly has scored some people low and ODAR has clearly hired them anyway. They are two different agencies. Why are so many having trouble with that concept this year?

I never said anyone should quit asking questions. The problem is not people asking questions, but people answering their own questions when they lack an understanding of the testing, scoring and hiring process. It's very clear that many of the posters complaining about the process are not looking for answers. They are looking for excuses. They don't want to read that people with the same experience and same education can have wildly different answers. They want to read that OPM is horrible and the process is flawed. Yet they have produced no proof of any kind that the process is flawed. Their only "proof" is that people with similar experience have different scores, which proves absolutely nothing. Every year people take the California bar exam that have very similar experience and every year people get very different scores. Many fail compeltely. People from Harvard and Stanford law, fail. That is no proof that the process is obscure and no proof that the scoring is flawed. It is proof that people are different and give different answers to the same questions.

The biggest problem with this entire issue is that normally intelligent attorneys have substituted their emotions for their powers of reasoning. These things affect you personally and like JH, you have lost all semblance of objectivity. While it is clear your objectivity will not be returning soon, it is heartening that some on this board do understand both the process and the true nature of these continuing baseless complaints.

[hod]

PM-I can accept my score for what it is. What I cannot accept is the backward logic that no one has "proof" of anything when the reason no one has proof is that the process is cloaked. Even when I took the bar, which admittedly was a couple years ago, if one failed, one got a chance to look at and appeal the score. And before you tell me that these OPM scores can be appealed, the board score appeal results occurred in someone's lifetime. Justice delayed is justice denied. OPM takes the position that if they wait long enough the appeals become moot. Granted, I never heard of anyone winning a law board appeal (at least not in my state) but they got to see the process.

I do not intend to appeal. I do not think I intend to ever go through the process again either.

People paid a lot of money for the "privilege" of going through this process. I have interviewed for jobs outside my state before and when it got to the point where an interview had to be face to face, the company paid. Now I am not saying that the governement should have to pay for everyone during the OPM process that you seem to understand so well, but I do think that they owe people some transparency. Further-these are governemnt jobs with taxpayer money fudning the process-maybe that alone should be enough of a reason for transparency. Private employers do not have to explain anything other than EEO fairness.

In any event-your opinion that the complaints are "baseless", and lack objectivity would be stronger if the process provided requested information. The secrecy is unnecessary. It is not necessary for OPM to use the same question time after time-bar exams change the questions each time they give the test. OPM can afford to let some light shine on the process without damaging the desired outcome.. Maybe then your point will be correct. Information may show that OPM is the only honest and fair component of the government that does not make mistakes and it is totally unnecessary to ever question anything that they do.

I cannot understand why you feel such a need to defend the system. Even if you think everyone who questions the process is an idiot-why do you care so much? People get to ask. Lawyers get to push for answers. That is what we do. The answers sometimes do not support the original theory-but that is all right also. We all gain from knowledge.

[interested]

How does one know that OPM scored some people low and they were hired by ODAR? I thought the point was that some people with ODAR only experience scored high and others with ODAR only experience scored low. I would also be curious what the result would be if all applicants scored below 60 had never applied. Would all the higher scorers still have gotten their higher scores? That would be a fair test of the integrity of the examination.

[zero]

I know somebody who predicted his/her score within 8/100th of a point. I was about 2-3 points off my prediction for my own score.

We both used a process that involved chicken bones and chanting to reproduce, as accurately as possible, the method we anticipated OPM would use.

[southernmiss]

I have enjoyed this thread. When it was serious, it said things that needed to be said; when it was funny, I laughed.

I am one of the low scorers, so I am obviously more dissatisfied than someone who got a high score. But let me get this across- my score may have accurately reflected what they were testing, which I obviously did not zero in on, but my score did not reflect my ability to be an ALJ. I would have been a fair, knowledgeable (at least for ODAR) and productive judge. So, if the test is to get the best judge, I have to think it is defective. Not that those that did better are necessarily going to disappoint, but there are many who got a poor score who would have done as well, or better.

[zero]

Mar 4, 2009 16:48:32 GMT -5 southernmiss said:

I have enjoyed this thread. ... when it was funny, I laughed.

I'll claim my share of the compliment and say "THANK YOU"!

[Yeats]

Because I did well in the testing process, I would like to think this post is not a whining exercise (as some seem to imply) on my part, but rather a truth exercise, in the sense that for me the real question is whether this process does what it is purportedly designed to do, which is to find individuals who will make good federal ALJs.

I would emphasize that question of external validity is different than the question of whether OPM should provide its scoring methods to the general public. My suspicion is that the reason it will not do so or provide any real detail about the testing process is that to do so might raise real questions about the testing process’s external validity. (Would it really destroy the testing process to outline the procedures that are used to secure inter-rate reliability, for example?) This is why I find PM’s claim that, “We have seen no evidence of any kind to suggest randomness in grading answers because we have seen no answers” to be quite beside the point, a position I might add that is somewhat at variance with a post in September of 2007 in which PM expressed the opinion, “Of course how OPM grades this year is a completely random factor.” In fact, it is the very fact that OPM takes no steps to justify the validity of its process that raises the legitimacy concern in the first place.

Determining external validity is not an impossible task. Psychologists do this sort of thing all the time. In theory, you could compare the test results against ALJ outputs that ODAR values. For example, you could try to determine whether a higher score on the test correlates with ALJ productivity. You could have supervising judges in each office rate existing judges, and then see how evaluation and test results correlate. If you wanted, you could compute a regression and see what test taking characteristics correlate with test scores and if anything pops out that does not look like it relates to judging ability that would be instructive and raise important validity concerns. At this point, given the perception of how the test operates, and given he how coveted these jobs seem to be, a study like this is probably warranted and would be relatively inexpensive.

It is also apparent to me that there is probably good reason to doubt this testing process has much external validity. First, I am troubled by Barkley’s recent post on the “Scoring and Entitlement” thread in which he notes his failure to pass the AR in 2007 and attributes his phoenix like resurrection in 2009 to changing the way in which he wrote his answers—an assertion that derives some added authority from PM’s claim recently that the lower scores on the current exam are attributable to the test takers failure to “prepare.” (Prepare for what, one might ask? Certainly, OPM does not tell you what to prepare for. How does one prepare for something if you do not know what the rules of the game are in the first place? By engaging in an internet scavenger hunt on this forum and on eBay?). And if the AR counts for more than the other parts of this testing process (as some on this forum have asserted) then this is very troubling.

Second, there is a great deal of research (originating I believe with Paul Meehl’s work on clinical versus statistical prediction in 1954) that strongly suggests that interviews are not valid predictors of future job performance. I suspect that this is especially true when you have interviewers, like those who interviewed me, who did not appear to be particularly well trained in the use of the assessment tool, and never seemed quite sure whether the script required them to follow-up on an answer and sometimes did so inappropriately, e.g. I had already answered the follow-up question before it was asked.

Third, it is hard for me to understand how what is essentially law school exam (and a not particularly clear or well-written one at that) translates into judging skills. And if as I fear, the exam is graded by a lot of different people and no real attempt is made to secure any inter-rater reliability between them, then this is also quite the recipe for randomness.

Measured against any statistical standard of proven validity, I find many of the arguments that I have read on this forum wanting—rhetorically attractive, but logically deficient. Claiming that the testing process is the same or even better than other exams given to pick administrative judges in other states simply begs the question. The fact that a bad process is replicated elsewhere does not make the process any less bad. Asserting that the testing process is fair and just based on one’s experience (while at the same time claiming there is no evidence one way or another to prove or disprove random results and
vehemently denying that other poster’s claims based on their own experience have any validity) seems equally unavailing and is arguably quite self-contradictory.

I think this will be me last post about the testing process, because I think everybody needs to move on. Short of a public campaign (which would be doomed to failure) for truth in federal ALJ testing based on a small number of disappointed job seekers there is really little to be done. The best that can be said for this process (absent any proof to the contrary) is that it winnows and chooses. Sadly for most government bureaucracies this is enough, but it should not be. We deserve better.

[nonamouse]

Interesting post Edmund, but OPM is not ODAR and ODAR is not OPM. I have no doubt that you or many other thoughtful individuals who actually give a rat's you know what could make improvements to OPM's part of the process. However, it would require all parties to cooperate to get the best results for all of the hiring agencies and not merely something "good enough" to pass muster and cover OPM's behind when Congress wants a progress report on the SSA case back log.

I don't know of any way that OPM can force a study at ODAR or any of the other 30+ agencies that have ALJs. IMO, OPM has no particular incentive to study their own process since they don't have to "live" with the results. It is the hiring agencies that deal with whomever they hire as ALJs whether from the register or by poaching from another agency. I think that even if ODAR wanted the managing (never say supervising or you will see fur fly) ALJs to study the other ALJs in the field offices, there would be an incredible stink once the news leaked. There is not exactly a huge bond of trust between the line ALJs and those pushing them for more, better, faster at the behest of those on high.

[valkyrie]

I won't argue with anything that edmundburke posted. It just needs to be viewed from the perspective of a sytem that ignores reality on its face. The OPM system is inherently flawed because of its failure to acknowledge the difference between its overwhelmingly largest customer, ODAR, and all of the rest of the federal agencies that use ALJs.

We are talking about two drastically different forms of legal practice, one very informal, high volume, and nonadversarial, and the other formal, low volume, and adversarial. Yet in practice, OPM assesses candidates on the latter, ODAR wants experience with the former, and then, as noted before, other agencies using the latter recruit from ODAR, which practices the former. Will testing methodology really improve this system?

Also, what is ODAR's definition of an "effective" ALJ? I think we can all agree that productivity and numbers is what ODAR wants. However, as we can tell from insider experience and some of the posts on this board, there is heavy resistance from the ODAR ALJ core on the productivity issue, some legitimate, and some not. While it appears that ODAR is currently winning the productivity battle, it isn't over, and as nonamouse indicated, any kind of productivity study would be a very explosive topic in management/ALJ relations. So there is currently no scientific and/or objective opportunity to determine how candidates with particular experience sets perform in the field.

To sum it up, I think some people, including edmundburke, have some valid complaints and corresponding ideas for improvement, but these represent wonderful ways of treating the bunions of a patient on the heart transplant list.

[Legal Beagle]

One fallacy in testing that I note in the WD is the fact that the ALJ candidates are judged on how well they write, but aren't all of the actual written decisions that come out of ODAR or other agencies - written by people with the title 'decision writers'?