How examination is scored

[shadow]

To start a new thread, as Learnedhand suggested, does anyone know or care to speculate:

1) How much weight is assigned each of the 3 parts of the exam in the final score [application, WD, & SI]?

2) How does VP affect the final score?

3) How, if at all, will references affect the final score?

4) When the process is complete, will we each be told what our score is, how it was arrived at, and where we rank vis-a-vis the other candidates? Some of these things; all of these things; none?

Thanks everyone. ;D

P.S. Why is everything so secret this time around?

[learnedhand]

According to the ALJ manual, the application counted for 50 percent of the final score. The WD and SI were each worth 20 percent and the personal references were 10 percent. That was true for the examination in the 90s and I don't think anyone knows what the numbers are now.

[Pixie]

As a guess, I would say that the new exam will track pretty closely the scoring on the old exam. Most things in government change little. even when forced to change. The pieces of the puzzle are swapped around, but the puzzle still looks about the same.

I take it from the comments that the "Accomplishments" or "Qualifications" section, or whatever they are calling it these days, was included in the application section? That is the section where many would spend weeks preparing for in the prior examinations. Pix.

[Pixie]

In the old exam, the qualifications section was called the "SQS" or supplemental qualifications statement (just now looked it up!). Don't know what it is called this time around, but probably something close. In the old exam the areas one had to discuss were:

+Knowledge of Administrative Procedures, Rules of Evidence and Trial Practices.
+Analytical Ability
+Decision Making Ability
+Oral Communication Ability and Judicial Temperament (seems like two separate areas to me)
+Writing Ability (of course this would be tested more fully later in the process, if the candidate made it to the next step) and
+Organizational Skills.

Seems as if this format was altered just a bit sometime around 1999, but I'm not sure.

I am giving this to you so can see how the exam has changed, if any, from the old version. If it is similar, then it is a lot to think about and to write about in such a short period of time. I understand that many got the applications in before midnight on the day the exam was announced. No wonder there were so many zeros handed out; easy to make fatal mistakes when rushing like that.

As to secrecy this time around, I would have thought that the scoring method would have been discussed in the job announcement. Guess not, huh? Pix.

[shadow]

Yes, Pixie, that's all in the application now. The new application is actually quite a bit simpler than it used to be. For example, a candidate doesn't have to list his or her ten most important cases, and a lot of other things that used to be required.

The only reason I know that is because about two years ago I put together the entire package [based on the 1997 version of the ALJ Handbook], mistakenly believing the ALJ examination was about to open up then, and that it would remain the same as in the 1990s. I spent a solid week compiling all of that stuff, and most of it wasn't needed in the new application.

The exercise wasn't entirely fruitless. I did use a lot of the information I compiled then in the new application.

[odarite]

And, again, your mileage may vary, but you will almost certainly be given your final score. You will not, however, be told how you rank, especially since this will be a moving target as new people are scored and as the register is reopened in the future.

[chris]

That's how it works with many state civil service exams. In that situation you talk to friends on the list and try to figure out what the high score was, so you can see where you rank. That's going to be difficult to do here. I don't know how many of the 650 or so contenders will be on the register but I assume most if not all will be. So even if 400 of us compare notes and find out that our highest score is 94, there could be some in the remaining 250 with scores from 95-110. The wannabes are just going to be hoping that someone with inside info tells us what is going on.

On the other hand, if we get called for agency interviews this year we will know we are high enough. But if we don't get called in the first round it could be because of a low score or a high score combined with no openings in our chosen cities.

[odarite]

excellent precis, Chris. That is pretty much how the thing will play out.

[lark]

I am new to this board, in private practice and am currently awaiting the results of the written exam and SI. I have been taking comfort from all of your comments.
I am wondering whether OPM intends to contact any verifiers. To date, none of my references have been contacted by OPM.
Thanks and good luck to all!
Lark

[testtaker]

I, too, have taken comfort from the information on this board and from some of the posts on the old board. I found the information in the ALJ manual to be helpful, but out of date for this exam. It appears that the old SQS is now the accomplishment record (AR).

I am in private practice and did not know about the posting until that Tuesday. I drafted my "USA Jobs" resume as fast as I could, ripped through the appliction and then worked on completing the AR. As it neared midnight, I thought about leaving the application open overnight, so I could proof it in the morning with a clear head. No way! I was afraid the posting would close that evening. So, I hit "send" at 11:56 pm. (Turns out the posting closed on Wednesday, but I figure better safe than sorry.)

I then learned about the ALJ handbook and bought it. I was floored when it said that it usually takes weeks to compile the information for the application. It made me second guess myself (If I did the resume, application and AR in 6 hours, how could it possibly be any good?). What a relief when I got my appointments for the WD & SI.

I have no idea how they will score things this time around. As someone else said, it was 50, 20, 20, 10, but since it appears that references are not being scored this time around, its anyone's guess as to how the percentages will work out.

I also wonder if the test will be weighted as it was in the past. The ALJ handbook stated that each section had a certain score, but then that was weighted. For instance, the WD itself was scored on a scale of 100 points, but was only worth 20% of the total rating. That was because the score was weighted. The handbook then states that once each score was weighted, they were added up and the total was "transmuted" [OK, I have to admit that they lost me at transmuted.] Once that was done, the Veteran's Preference was applied.

I wonder if they will still use that same formula of weighting and transmuting. I also wonder why the scoring process hasn't been more out in the open. Its been all very secretive and odd. If it hadn't been for the handbook, I would not have had any clue as to what to expect for the WD or SI. [sorry for such a long post - its my first on this board].

[testtaker]

lark said:

I am wondering whether OPM intends to contact any verifiers. To date, none of my references have been contacted by OPM.
Lark

None of my references or verifiers were contacted.
Anyone have any information about this?
Thank you.

[learnedhand]

The application, if I remember correctly, did not ask for references. It did ask for verifiers for each of the competencies. The competencies were part of the application process but it is not clear to me of they were included in the AR scoring, which you had to pass with a high enough score to go forward. Any thoughts?

[zugswang]

I would guess that OPM is not planning to contact verifiers, given that that OPM has to turn around 650 of these files by October. Most people who have done even a smidge of hiring can pretty easily tell when they need to contact people to verify the legitimacy of claims, so that step could easily be left to the hiring office when then get a slate of candidates.

[tootsie]

I looked at my copy of the application, and in the USA Jobs Resume, just after "Professional Publications" there was a space to enter references. It was right at the end, before "Additional Information" -

In addition, the remainder of the "package" asked for "verifiers" for the 6 competencies. I re-used the references, and added a couple more.

[testtaker]

learnedhand said:

The application, if I remember correctly, did not ask for references. It did ask for verifiers for each of the competencies. The competencies were part of the application process but it is not clear to me of they were included in the AR scoring, which you had to pass with a high enough score to go forward. Any thoughts?

It was the USA Jobs resume that gave you the option to list references.

The application and the Accomplishment Record (AR) were all done on the computer and submitted as one document through a site called "Application Manager". The application portion consisted of basic info, your geographic preferences, etc. and the "Assessment Questions", in which you had to describe your experience as it related to that required to be an ALJ (administrative/litigation, 7 years, etc.).

Once that was done, you went onto the AR portion. There were six competencies defined. For each competency, the applicant was required to provide a statement of how he/she achieved/accomplished that competency. In addition, for each of the six competencies, the applicant had to list the name and contact information of someone who could verify the information provided by the applicant for that particular response.

You'd think they'd have at least checked up on the verifiers. Perhaps that will happen at before the agency interview.

[Pixie]

Under the prior exam the "verifiers" or perhaps they were called references, were sent a form questionnaire to be filled out and returned. This form rated the applicant in the six areas of competency and was used as part of the grade. I believe it counted as 10%. Don't know how they are doing it this time around. Pix.

[doctorwho]

Remember that applicants were given the choice to upload their resumes rather than to go through the USAJobs resume process. Those of us that did that, did not necessarily include resumes in addition to verifiers. I suspect that at this stage in the game, OPM is not going to be contacting anybody -- as they don't have anytime to do that AND get a cert list out by their October deadline. The time is ticking away. If anybody contacts any sort of references (and I suspect they will), it will be the hiring agencies.

[chris]

I suspect doctorwho is right. OPM is not going to do anything with the references and the verifiers. They will not be part of the score. First, they don't have time. Second, nothing in the application process stated they would be part of the score. Third, I think OPM is tired of the criticism for not having a new list, especially criticism from SSA, so they are going to provide the verifier/reference information to the agencies and let the agencies do their own legwork. Although I'm very new to this federal civil service stuff, I just get the feeling that OPM is going to pay SSA back for all the criticism by letting SSA do some of the work.

Chris

[lee]

Under the old test procedure, weren't the verifiers contacted to confirm information regarding the 10 significant cases that the applicant listed in the SQS?

[Pixie]

Yes, they were contacted and given the questionnaire to fill out and return. The rating from the questionnaire counted toward the total score. From the sound of things, that's not the way it will work this time around. Pix.