understanding your alj final numerical rating

[allrise]

I just got this email and it is slightly different from the explanation I received in 2007. At the bottom, it lists these competencies:
Competencies: 1) Decision Making, 2) Interpersonal Skills, 3) Oral Communication, 4) Writing, 5) Judicial Analysis, 6) Judicial Management, 7) Reasoning, 8) Self Management, and 9) Stress Tolerance.

The first 6 are from AR. Are the last 3 from the wd and si?

[maxlaw]

I'm just going to post the contents so people can see what is being discussed. This is my first rodeo, so I can't compare it to previous ones. Based on the previous emails though, the AR was reasoning, and the SI added self-managment and stress tolerance.

The "??" in the fourth paragraph were present in the original email (at least as forwarded to my Blackberry).

Subject: Understanding Your ALJ Final Numerical Rating Dear ALJ Applicant, You recently received a Notice of Results (NOR) indicating that you had successfully completed all parts of the Administrative Law Judge (ALJ) examination and that your name and final numerical rating would be placed on the current ALJ register being maintained by the U.S. Office of Personnel Management (OPM). The purpose of this notice is to provide a brief explanation to help you understand how the ALJ final numerical ratings were calculated. As stated in the vacancy announcement, the purpose of the ALJ examination is to evaluate applicants on the competencies essential to the work of an ALJ. For the Accomplishment Record (AR), applicants provided written statements of their accomplishments, which were scored. All applicants were ranked based on their AR scores, including any applicable veterans' preference points. The applicants who were among the higher AR scores from all eligible applicants were identified and moved on to the next stages of the assessment process. Because your AR score was among the higher AR scores, you were invited to participate in the Written Demonstration (WD) and Structured Interview (SI) portions of the ALJ assessment. The competencies measured in the AR, the WD, and the SI were combined to determine a total competency score. For this step, any veterans' preference points added to the AR score (as discussed in the previous paragraph) were not included; instead veterans' preference points were added to the numerical rating at the end of the scoring process, as indicated below. To compute this total competency score, the following steps were taken: ??The competencies were made equal in importance, so that each competency carried equal weight, ??The points earned on each competency were added together to determine the total competency scores for each applicant, and ??The total competency scores of all applicants were put on a 1 to 100 scale to establish each applicant's numerical rating, excluding veterans' preference. In this scale, the lowest possible score is 1 and the highest possible score is 100. This numerical rating, plus any applicable veterans' preference points, is the final numerical rating noted on the NOR. Every applicant on the register has already exceeded an initial cut-off score because OPM identified the higher scored applicants after rating the AR. Therefore, OPM did not employ any subsequent screen-out or passing score when adding new names to the current ALJ register. As stated on the NOR, your name will be placed on the current register in rank order based on your final numerical rating. This list will be used as a source of names to make referrals to agencies for employment consideration. Competencies: 1) Decision Making, 2) Interpersonal Skills, 3) Oral Communication, 4) Writing, 5) Judicial Analysis, 6) Judicial Management, 7) Reasoning, 8) Self Management, and 9) Stress Tolerance. We hope this information is helpful to you.

[allrise]

I looked further at old emails and in previous emails OPM stated that WD measures: (1) Reasoning, (2) Decision Making, (3) Judicial Analysis, and (4) Writing. OPM stated that SI measures: (1) Reasoning, (2) Decision Making, (3) Self-Management, (4) Interpersonal Skills, (5) Oral Communication, (6) Stress Tolerance, and (7) Judicial Management. I had always assumed that there was a seperate score for each of the 3 parts, WD, SI and AR. It looks like 1/9th of your score is made up of Decision making for example, which is measured on all 3 parts of the test.

[71stretch]

May 12, 2010 14:30:49 GMT -5 allrise said:

I just got this email and it is slightly different from the explanation I received in 2007. At the bottom, it lists these competencies:
Competencies: 1) Decision Making, 2) Interpersonal Skills, 3) Oral Communication, 4) Writing, 5) Judicial Analysis, 6) Judicial Management, 7) Reasoning, 8) Self Management, and 9) Stress Tolerance.

The first 6 are from AR. Are the last 3 from the wd and si?

So it would appear.

[Propmaster]

May 12, 2010 15:12:10 GMT -5 71stretch said:

May 12, 2010 14:30:49 GMT -5 allrise said:

I just got this email and it is slightly different from the explanation I received in 2007. At the bottom, it lists these competencies:
Competencies: 1) Decision Making, 2) Interpersonal Skills, 3) Oral Communication, 4) Writing, 5) Judicial Analysis, 6) Judicial Management, 7) Reasoning, 8) Self Management, and 9) Stress Tolerance.

The first 6 are from AR. Are the last 3 from the wd and si?

So it would appear.

Maybe Chinook got bonus points for handling his extra stress so well at the WD.

[chinook]

Maybe.

[allrise]

I think I may have figured out how much weight they give to each part of the application process. Interview: 48.1%; application: 31.45% and written test: 20.35%. In the explanation it states that the 9 “competencies” are given equal weight in the scoring so each competency is 11.1% of the grade. OPM has told us which competency is measured in each part of the process. Self management, for example, is only measured in the interview so 11.1 points can be earned in the interview for this competency. Reasoning is measured in the interview and the written test so you can get 5.55 in this category during the test and interview. This is how I calculated it:

AR WD SI
Reasoning 5.55 reasoning 5.55
Decision making 3.7 Decision making 3.7 decision make 3.7
Judicial analysis 5.55 Judicial analysis 5.55
Writing 5.55 Writing 5.55
Selfmanagement11.1
Interpersonal skills5.55 Interpersonalskill5.55
Oral communication5.55 Oralcommunication5.55
Stress tolerance
Judicial management 5.55 Judicial management 5.5

Totals:

AR: 31.45
WD: 20.35
SI: 48.1
I don’t know where the .1 went

[maxlaw]

May 13, 2010 8:47:51 GMT -5 allrise said:

I think I may have figured out how much weight they give to each part of the application process. Interview: 48.1%; application: 31.45% and written test: 20.35%. In the explanation it states that the 9 “competencies” are given equal weight in the scoring so each competency is 11.1% of the grade. OPM has told us which competency is measured in each part of the process. Self management, for example, is only measured in the interview so 11.1 points can be earned in the interview for this competency. Reasoning is measured in the interview and the written test so you can get 5.55 in this category during the test and interview. This is how I calculated it:

AR WD SI
Reasoning 5.55 reasoning 5.55
Decision making 3.7 Decision making 3.7 decision make 3.7
Judicial analysis 5.55 Judicial analysis 5.55
Writing 5.55 Writing 5.55
Selfmanagement11.1
Interpersonal skills5.55 Interpersonalskill5.55
Oral communication5.55 Oralcommunication5.55
Stress tolerance
Judicial management 5.55 Judicial management 5.5

Totals:

AR: 31.45
WD: 20.35
SI: 48.1
I don’t know where the .1 went

This is such good information. I think your reading is consistent with the email, and may well be correct. Still, there is a large caveat in that while the emails say the 9 competencies are equally weighted, they don't specifically state that the break down within the competency is even. For example, AR, WD and SI measure decision making, but it isn't clear to me that it's an even split among all three parts.

If this chart is correct though, it's quite interesting in that my understanding of the previous conventional wisdom was that the AR was the largest percentage of one's final score. I'm rather shocked that the SI seems to be so important (although perhaps that nondisclosure agreement makes more sense now).

Anyway, I've taken the liberty of formatting your info in forum-friendly way. The missing .1 is because you can't evenly divide 100 by 9, so you reasonably picked 99.9 when you crunched your numbers

Competency	AR	WD	SI
Reasoning	---	5.55	5.55
Decision making	3.7	3.7	3.7
Judicial analysis	---	5.55	5.55
Writing	5.55	5.55	---
Self management	---	---	11.1
Interpersonal skills	5.55	---	5.55
Oral communications	5.55	---	5.55
Stress tolerance	---	---	11.1
Judicial management	5.55	---	5.55
Judicial analysis	5.55	5.55	---
Totals:	31.45	20.35	48.1

[allrise]

If this info is correct, it would help explain the wild swings in some people's scores. I know a couple of people who threw together the ar in a few hours and scored very well. Doing well on the wd and especially the si may be key.

[71stretch]

I'm one of those people who did the AR on the fly in a matter of hours, without stopping, (it's been so long since November I don't remember how long it actually took me) because I was afraid they would cut off apps before I finished. Could it have been better if I'd worked on it ahead of time? Surely.

I know I lost some points on the SI with one of the questions. Overall, it probably went better than I felt like it did at the time, but I could have wished to have a different panel, or at least that question/answer done again. I have no idea how the WD was scored, but I felt better about that than the SI. Doing both the WD and the SI in a little more than 24 hours wasn't easy, but it saved $$.

[Deleted]

What I am having trouble understanding is why the SI, which was short and an awkward format and didn't seem to correlate well with the skills to actually be an ALJ, would count so much. After all, the AR isn't supposed to be evaluated on the format itself but is supposed to take into account the totality of our career and therefore our qualification to be an ALJ. It seems like it's just become form over function with the AR. Honestly, I find it just odd that the SI would count so much, whereas our whole career would count so little. IMHO, at least the WD was related to one of the skills of being an ALJ -- writing. Granted the set up was very artificial, but at least they had the right skill. The SI remains a mystery to me. I must have done poorly, since I got a stunningly low overall score, but I couldn't pinpoint why that's true.

[maxlaw]

May 13, 2010 11:15:38 GMT -5 @shanti said:

What I am having trouble understanding is why the SI, which was short and an awkward format and didn't seem to correlate well with the skills to actually be an ALJ, would count so much. After all, the AR isn't supposed to be evaluated on the format itself but is supposed to take into account the totality of our career and therefore our qualification to be an ALJ. It seems like it's just become form over function with the AR. Honestly, I find it just odd that the SI would count so much, whereas our whole career would count so little. IMHO, at least the WD was related to one of the skills of being an ALJ -- writing. Granted the set up was very artificial, but at least they had the right skill. The SI remains a mystery to me. I must have done poorly, since I got a stunningly low overall score, but I couldn't pinpoint why that's true.

Remember, we're not certain of any of this right now - we could be way off. I will say that HR types seem to love structured interviews in general, and a lot of ink has been spilled in scholarly journals about why they are good tools. Not sure I buy it, but what do I know? I'm just a fellow lab rat.

[tricia]

Issuegirl, you said: "I also was short one panel member last year during the SI, which may have hurt me. " I think everyone in the SI process the day I was there agreed to waive the presence of the third interviewer. How could this hurt the applicant?

[kilgore]

Here is how. With a full panel you have three individual raters who give you a score. For arguments sake lets say with the three member panel you get 60 points. Assuming the overall score is divided by three your SI would be 20. Now you have a two member panel. Your score from the two members is 40 obvioiusly your overall would be 20 if you divided by two. However, say the GS-5 clerk tallying the scores just assumes three members for every panel. Your 40 is divided by three and your overall score becomes 13.3. I think you have been hurt.

Oh an by the way I received a NOR despite the fact my WD was not rated due to a malfunction of the laptop I was using. My rating was based purely on the SI and AR. It was 38.97. I am assuming I received a zero for my WD. I am appealling. I am also contemplating other actions.

[issuegirl]

Kilgore:
I'll have a tequila shot for you. That is about the worst thing I have heard on this forum.

[kilgore]

Yes this was a lousy outcome, but not the end of the world. Other opportunities open when others close. You just have to keep your radar adjusted.

I would join you Issuegirl, but tequila and I have somewhat torturous history. I love it and it makes me do crazy things. So that said I will consume an 18 year old single malt scotch, which is arguably a more civilized drink.

[71stretch]

May 13, 2010 14:04:22 GMT -5 kilgore said:

Here is how. With a full panel you have three individual raters who give you a score. For arguments sake lets say with the three member panel you get 60 points. Assuming the overall score is divided by three your SI would be 20. Now you have a two member panel. Your score from the two members is 40 obvioiusly your overall would be 20 if you divided by two. However, say the GS-5 clerk tallying the scores just assumes three members for every panel. Your 40 is divided by three and your overall score becomes 13.3. I think you have been hurt.

Thanks for posting this. I had a two person panel. Now you have me wondering. However, I am still not inclined to appeal solely to check this math.

[kilgore]

You are welcome. It happened to me in 2007. My appeal was not successful. No reason was given and it took nearly 14 months to process.

[Well]

May 13, 2010 11:15:38 GMT -5 @shanti said:

After all, the AR isn't supposed to be evaluated on the format itself but is supposed to take into account the totality of our career and therefore our qualification to be an ALJ. It seems like it's just become form over function with the AR. Honestly, I find it just odd that the SI would count so much, whereas our whole career would count so little.

Going back over my materials, I don't believe career counts for much at all in the process beyond getting to the minimum qualifications.

Think about the AR essays. It is a series of snapshots of examples of specific skills. All could have happened in the last month or last decade.

WD places no weight on career or achievement.

SI places no weight on career or achievement either.

When someone has 20+ years of broad relevant experience at the highest levels of the profession and wonders how they scored so low, I think the answer is simple. You've got to score well on the essay, score well on the WD, and on the SI because all the experience does is get you past that minimum qualification threshold and does nothing else for you.

The agency interviews I suspect do take into account more of a person's career but we are playing a giant TV reality game show where the first key to winning is to not be voted off in the current stage. First meet the deadline to apply, second score well enough to move to the WD/SI, then score well enough to make the cert, then perform well enough to win a location battle. Just like American Idol the producers pretty much know in advance how many advance to each round and some winners are going to be Taylor Hicks and some "losers" are going to be Daughtry or Jennifer Hudson.

[tricia]

Kilgore, just when I thought that I couldn't get any more depressed about this, you give us your input.
First you said: "However, say the GS-5 clerk tallying the scores just assumes three members for every panel. Your 40 is divided by three and your overall score becomes 13.3. I think you have been hurt. "
Okay, well I'm hoping that since so many of the applicants had two interviewers, they would have figured this out and done the math properly. But I'll still be worrying about this in the middle of the night.

Then you said: "Oh an by the way I received a NOR despite the fact my WD was not rated due to a malfunction of the laptop I was using. My rating was based purely on the SI and AR. It was 38.97. I am assuming I received a zero for my WD. I am appealling. I am also contemplating other actions."
Okay, now based on what I've read on this board, you just might have a good appeal issue. The - it was your fault and I can prove it - argument has been a winner before. Good luck