scoring & entitlement

[carrickfergus]

As a long time lurker and first time poster, I've gotten to the point where I just can't refrain (disclaimer: adjudicated a hearing today where the agency only got through 1/4 of its presentation, with no documents submitted as evidence - all testimonial - so needed a glass of wine or two upon returning to home & hearth) from commenting on the angst about the scoring rationale (or lack thereof).

We are fortunate to have even been able to submit an application within the deadline.

We are fortunate that the AR was written reasonably well enough to have been invited to the WD/SI,

We are fortunate if we had the means to attend both the WD and SI.

We are fortunate to attend law school, pass the bar, and practice law or adjudicate.

But we are not entitled to an ALJ position unless we are ALJs, and are not competent to criticize the scoring process, unless we've designed and implemented a better way to go about it. I've yet to see an alternative posed on this board to the existing system.

When we hired adjudicators in our State, we were forced to consider union members first, under the contract, regardless of their quals. Imagine if that were the case here.

We designed a system to flush out those individuals with a three-part process: (1) a multiple-guess test that filtered out the droolers; (2) an essay test that filtered out the illiterate; and (3) an SI that filtered out the sociopaths. I don't see how the ALJ process is much different. Fair? Who knows. Did it serve our purpose? Yes.

We are all human and subject to the foibles and defects that are inherent with our condition, and for those of us who made it this far, congrats are in order and be grateful for the opportunity; and, for the love of Pete, quit whining!

I'm enjoying the roller-coaster ride up to this point, it has really intensified my existence.

Cazart!

C.

[roggenbier]

I've taken the AlJ exam 1999 and now. I've designed ALJ exams at the state level. It has been a well designed test. Not perfect. Amen.

[pm]

I've taken and/or scored ALJ exams in different states. They are all pretty similar to the new OPM exam. I even had exactly the same question on the OPM SI as I had on a state test (it didn't help-it was one of those questions where your immediate answer was "huh?"). They all have their little quirks and could all be tweaked a bit, but they are about as fair as you are going to get. People have been invited on this board and TOB to come up with something better and they have not.

I didn't take it, but the old Federal ALJ exam appeared to have been somewhat unique and full of interesting quirks. You could tell it was designed some time ago.

[flannery]

Carrickfergus--is that far from the Lake Isle of Innesfree? If not, why do I agree with so much of what you stated? I agree completely: be grateful and thankful for each bit of progress we make in life, whether it be passing the bar, getting a decent government job, or getting one of these ALJ positions. And yes, we should all try to keep the whining down. If we don't make it this time, we should save our energy (and our blarney) for the next application...

[interested]

I don't really believe that I have seen anyone upset with the examination itself. The problems are with the secrecy of how it was scored. I don't know much about state ALJ examinations, but are the methods of scoring secret? With the prior ALJ examination, we were provided with a breakdown of our score. Not so this time. How is that for a suggestion? Another suggestion would be to weigh agency experience in the score relative to certifications. How about allowing minimum Agency experience to be appointed as an ALJ?

[ssaer]

Mar 5, 2009 8:24:04 GMT -5 interested said:

The problems are with the secrecy of how it was scored. I don't know much about state ALJ examinations, but are the methods of scoring secret? With the prior ALJ examination, we were provided with a breakdown of our score. Not so this time. How is that for a suggestion? Another suggestion would be to weigh agency experience in the score relative to certifications. How about allowing minimum Agency experience to be appointed as an ALJ?

I don't understand how anyone could reasonably disagree with this comment. When we are not informed about the relative weights accorded the different parts of the examination process, not provided with any of the specific criteria used to evaluate performance in that process, and not provided with itemized scores differentiated among the various parts of the examination process, on what basis could an individual -- even theoretically -- challenge his or her score?

Further, since most of those hired will work for SSA, why shouldn't SSA-specific experience be a relevant and meaningful factor? And why should both the Written Demonstration and the Structured Interview focus upon adversarial types of proceedings that have little relevance to the work of SSA ALJs?

[Legal Beagle]

I kind of agree about the agency specific experience. I have both SSA, NLRB, and other agency experience in my private career, and understand that some ALJs who were hired last year doer ODAR did not have SSA ecperience - and I hope I do not get them on my cases. Perhaps once we are on the Register, OPM could ask us to list the agencies that we do have experience in - as a way of certifying to those agencies.

[jagghagg]

The regulations governing the employment practices of the federal government and that of individual agencies is that the recruitment and selection of individuals for initial appointment . . . in the competitive service is to be done without discrimination because of non-merit grounds. (See generally 5 CFR 300 et seq.) Prior agency-specific experience is a non-merit factor. If SSA wants it to be a valid factor to be considered the requirement that would allow such a preference is that the test validating it be professionally developed. Without professional research conducted to validate a preference for Social Security Administration law-specific experience, and utilization of that as a criteria for selection violates current regulations.

It’s not that the SSA does not and has not wanted their ALJs to have agency-specific experience. The Social Security Administration has argued in favor of granting bonus points to candidates who have subject matter experience “to provide such candidates with a reasonable opportunity for selection.” As far back as 1988, the public was being informed that the Social Security Administration still preferred it’s own. In October of 1988, SSA Commissioner Dorcas Hardy sent a letter to Constance Horner, OPM Director, wherein she was seeking OPM certification of ALJ candidates with SSA experience. A year later, in 1989, SSA Commissioner Gwendolyn King addressed her letter to Constance Newman, OPM Director wherein she was attempting to select candidates with agency-specific experience from the ALJ register. (Interesting that OPM had two “Constances” in a row, huh ?) In 1996, SSA Commissioner Shirley Chater continued to express the high value placed on the specialized expertise of SSA ALJ's by the Commissioner’s Office when she testified before a Congressional subcommittee and observed that agency-specific experience was essential for quality and efficiency of SSA adjudication. In 1997, the SSA Associate Commissioner, Rita Geier, restated the agency's intention to conduct a portion of its 1997 ALJ hiring with attention to agency-specific experience. The current Commissioner himself has published his own belief that the agency requires ALJs with agency-specific experience.

The problem is that OPM has been unwilling to allow this emphasis on agency-specific experience. OPM has promulgated policies regarding its employment practices and they can be found in the CFR referenced. Agency-specific experience cannot be used unless or until the efficacy of such experience has been validated by independent professional research. To my knowledge, this has not yet been done.

[valkyrie]

You can get a decent score with only agency experience. There are plenty of examples out there. The trick for everyone, inside or outside of ODAR is how you fill out the application.

The key to this process is to remember that the point of the entire endeavor is for OPM to provide other govt agencies with a register of highly qualified ALJ candidates from which openings can be filled. Other than basic protections against discrimination, there is no mission to assist each candidate in achieving his/her highest possible score. It sounds very cold, but again, ask yourself, "what does the govt owe me in this process?" They owe everybody equal consideration within an imperfect system, and that's it. Then, from an objective standpoint, ask yourself if OPM has failed to provide a register of qualified candidates. Don't ask if OPM has gotten EVERY highly qualified candidate on the register, that's perfection, and its not their job. There are only two ways for OPM to fail; putting poorly or unqualified candidates on the register, or failing to fill the register with enough candidates.

In terms of fairness, everybody has the opportunity to re-compete in any of the areas after one year, whether you know the breakdown of your score or not. People have appealed and gotten onto the register with decent scores. There is a process for them to "find" a good candidate they previously missed.

When I first applied for the job as an SSA staff attorney, it was one of many applications for govt attorney positions that I filled out. Several months down the line I would receive an occasional rejection notice, and then of course I received an offer. However, during all those months of waiting, I had know idea in hell what was going on with my applications, scoring, etc, for any of the positions for which I applied. I wasn't angry, frustrated, or furious with OPM then. I just knew that my applications were one of many thousands, so I certainly didn't take it personally. Should I feel that way now, where OPM has actually given me a better idea of where I stand in the hiring process?

[interested]

I had it wrong. I expected OPM to evaluate the applicants, rate the applicants based on reasonable criteria and provide a score relative to other applicants that is consistent with the criteria. I would expect similarly situated people would at least be scored somewhat similarly. I would also expect absent stark differences, the scores would be reasonably close.

Do you seriously compare this process to submitting a resume through USAJOBS? People went through great effort, investing significant amounts of time and spent a large amount of money. They should just wait a year and do it all again so OPM can get it right the second time?

I can't disagree with any of the prior post if that is really what this was all about. OPM just getting together a list of enough people that were minimally qualified. Just toss in widely disparate scoring to keep everyone's life more interesting.

I guess that's it. You either trust that OPM developed and fairly applied criteria or you don't. Who could ever question OPM's fairness or competence? But it is what it is.

[valkyrie]

"Do you seriously compare this process to submitting a resume through USAJOBS? People went through great effort, investing significant amounts of time and spent a large amount of money. They should just wait a year and do it all again so OPM can get it right the second time?"

If you recall, thats where this whole process started, and where, for a lot of the people involved, the process ended too. Despite everybody's effort, time, and money, nobody is special, and based upon the number of people going through the process, anyone who gives up on the process in disgust only makes OPM's process of elimination easier. As I have noted before, OPM did not fail. Nobody has suggested that anyone selected, or even on a certificate does not belong there, or that there are not enough people on the register. OPM may have failed some candidates individually, but it achieved its mission.

I actually put a lot of time and effort into the attorney applications I submitted to OPM. I believe I was successful based upon my qualifications, my effort, and, most importantly, luck. With the number and quality of people of people involved in this process, it is ridiculous to think that luck cannot be a part of the process.

There are people who have spent over ten years trying to get an ALJ slot. Sometimes it comes down to a test of stamina involving multiple trys on the interviews, apps, and tests. At 150 selections per year, this is a more exclusive process than gaining entrance into the service academies or the ivy league law schools.

[jagghagg]

Mar 5, 2009 13:59:31 GMT -5 valkyrie said:

I believe I was successful based upon my qualifications, my effort, and, most importantly, luck. With the number and quality of people of people involved in this process, it is ridiculous to think that luck cannot be a part of the process.

It never, ever comes down to "luck." It can't. A process in the federal sector has to be defensible and "luck" just isn't a factor. Luck, perhaps, from the perspective that your timing was right and the Forces of the Universe converged to make you the best qualified amongst qualified candidates; luck, perhaps, that someone better qualified than thou did not apply, but as a "most important" aspect, I'd probably opt for qualifications.

[privateatty]

When I started this process in 2007 I knew nothing. AR? Automatic rifle., etc. I put down one city. Why more? How we did the written exam? What, requires skills we were supposedly taught in high school, but were never encouraged to sharpen as associates. Whine Whine.


I was counseled to take the exam again if I could. I did and raised my score 18 points. Middlin' to top 10% (by this Board). And it was all due to what I learned on this Board. As I say to my daughter, there is a moral to this story...Thank you Board!

The process OPM has devised is not necessarily going to identify good Judges. It is game we must play by knowing the rules and applying oneself. Sorta like litigating in a new jurisdiction.

I feel for those with 25 years of lawyering under their belts and low scores. I would strongly recommend trying again if you can and having your AR "scored" by friends. Do some mock interviews too. Can't hurt.

[ishmael]

Mar 8, 2009 14:59:21 GMT -5 privateatty said:

I feel for those with 25 years of lawyering under their belts and low scores. I would strongly recommend trying again if you can and having your AR "scored" by friends. Do some mock interviews too. Can't hurt.

I know of "rumors" of many attorneys who have acknowledged that they had helped in preparing their ARs, including being given ARs from those who had made it through 2007 so that they can see what needs to be done to score high enough to get through the first eliminiation.

However, the honor code and responsibility that bind all of us attorneys from revealing material facts about the WD/SI also bind us during the AR process.

Before you can submit your ARs, you have to certify several facts under the "Certification of Information Accuracy." Such facts include not making any false statements or commit deception or fraud.

Another such fact that you must adher to before you can submit your application clearly states -- "I certified that I have worked independently and did not get any form of assistance while completing the Accomplishment Record."

It is clear to everyone that you should not be seeking "assistance" during the WD . . . from OPM's perspective, the same seems to be true for the ARs . . . for the ARs, OPM does not only want to know how competent you are in each competency, but OPM also wants to know your skill (not someone else's) in drafting such competency . . .

Why so many attorneys miss this part of the certification is beyond me. I would strongly advise anyone against getting any form of assistance in completing their ARs per the certification that you must signed prior to submission of your ARs . . . at the very least, I would not acknowledged such in public* nor advised anyone to do so . . .

If you did receive "any form of assistance," you must check the option of "N -- No, I do not certify/understand the information provided above."

We can guess what would probably happen to your ARs if you were honest and check such an answer . . .

I have been asked many times by those seeking assistance of what I discussed in my ARs. I have always politely declined to answer (for their sake) and gently remind those who asked of the certification requirements . . .

* if you did this, you may have violated the "fraud," "false statement," or "deception" criteria

[privateatty]

Thanks ishmael for pointing this out. Obviously, I gave bad advice and need to address it here.

I need to point out that writing a better AR was an issue that was addressed last year on this Board--the thought being that extolling one's virtues in the first person sometimes needs a third person perspective. Some people can step outside of themselves, others can't. I did not mean to infer that one would use another's words or direct thoughts. Never even thought of that. I know no one else who did, so my experiences are different than yours.

Other posters here have spoken of "different answers" to life experience descriptions. Is it improper for some one to read your AR and tell you to "beef it up"? Under your definition probably, although I don't think you would find much support for that.

That having been said, I think that it is paramount that we all operate from the same ethical base we were taught in law school. Frankly, what you caution has not been well addressed on this Board because we almost all assume a high degree of ethical behaviour.

My problem throughout this whole process is that it is skewered to those who know what they are doing. Who understand how to interview, do an effective AR and write a good exam in the format demanded. I do appreciate your thoughtful response and admonitions.

[pm]

Mar 8, 2009 18:40:03 GMT -5 ishmael said:

Mar 8, 2009 14:59:21 GMT -5 privateatty said:

I feel for those with 25 years of lawyering under their belts and low scores. I would strongly recommend trying again if you can and having your AR "scored" by friends. Do some mock interviews too. Can't hurt.

I know of "rumors" of many attorneys who have acknowledged that they had helped in preparing their ARs, including being given ARs from those who had made it through 2007 so that they can see what needs to be done to score high enough to get through the first eliminiation.

However, the honor code and responsibility that bind all of us attorneys from revealing material facts about the WD/SI also bind us during the AR process.

Before you can submit your ARs, you have to certify several facts under the "Certification of Information Accuracy." Such facts include not making any false statements or commit deception or fraud.

Another such fact that you must adher to before you can submit your application clearly states -- "I certified that I have worked independently and did not get any form of assistance while completing the Accomplishment Record."

It is clear to everyone that you should not be seeking "assistance" during the WD . . . from OPM's perspective, the same seems to be true for the ARs . . . for the ARs, OPM does not only want to know how competent you are in each competency, but OPM also wants to know your skill (not someone else's) in drafting such competency . . .

Why so many attorneys miss this part of the certification is beyond me. I would strongly advise anyone against getting any form of assistance in completing their ARs per the certification that you must signed prior to submission of your ARs . . . at the very least, I would not acknowledged such in public* nor advised anyone to do so . . .

If you did receive "any form of assistance," you must check the option of "N -- No, I do not certify/understand the information provided above."

We can guess what would probably happen to your ARs if you were honest and check such an answer . . .

I have been asked many times by those seeking assistance of what I discussed in my ARs. I have always politely declined to answer (for their sake) and gently remind those who asked of the certification requirements . . .

* if you did this, you may have violated the "fraud," "false statement," or "deception" criteria

You have to read and consider the entire admonition, not just parts of it. The critical language is that you "did not get any form of assistance while completing the accomplishment record." Then of course, you have to interpret the word "assistance" correctly. If your wife reminded you of your bar number and gave you a cup of coffee, was that assistance?

[ishmael]

Mar 8, 2009 21:36:37 GMT -5 pm said:

Mar 8, 2009 18:40:03 GMT -5 ishmael said:

I know of "rumors" of many attorneys who have acknowledged that they had helped in preparing their ARs, including being given ARs from those who had made it through 2007 so that they can see what needs to be done to score high enough to get through the first eliminiation.

However, the honor code and responsibility that bind all of us attorneys from revealing material facts about the WD/SI also bind us during the AR process.

Before you can submit your ARs, you have to certify several facts under the "Certification of Information Accuracy." Such facts include not making any false statements or commit deception or fraud.

Another such fact that you must adher to before you can submit your application clearly states -- "I certified that I have worked independently and did not get any form of assistance while completing the Accomplishment Record."

It is clear to everyone that you should not be seeking "assistance" during the WD . . . from OPM's perspective, the same seems to be true for the ARs . . . for the ARs, OPM does not only want to know how competent you are in each competency, but OPM also wants to know your skill (not someone else's) in drafting such competency . . .

Why so many attorneys miss this part of the certification is beyond me. I would strongly advise anyone against getting any form of assistance in completing their ARs per the certification that you must signed prior to submission of your ARs . . . at the very least, I would not acknowledged such in public* nor advised anyone to do so . . .

If you did receive "any form of assistance," you must check the option of "N -- No, I do not certify/understand the information provided above."

We can guess what would probably happen to your ARs if you were honest and check such an answer . . .

I have been asked many times by those seeking assistance of what I discussed in my ARs. I have always politely declined to answer (for their sake) and gently remind those who asked of the certification requirements . . .

* if you did this, you may have violated the "fraud," "false statement," or "deception" criteria

You have to read and consider the entire admonition, not just parts of it. The critical language is that you "did not get any form of assistance while completing the accomplishment record." Then of course, you have to interpret the word "assistance" correctly. If your wife reminded you of your bar number and gave you a cup of coffee, was that assistance?

pm, you are correct in the quote of the language, thus I quote myself in stating that -- "I would strongly advise anyone against getting any form of assistance in completing their ARs per the certification that you must signed prior to submission of your ARs . . . " (see paragraph six of my post). Please noticed that I tailored my advice to closely mirror the certification requirement. I purposely did not attempt to try to guess and interpret what these words mean -- I am not so foolish. I also must note that to have a better grasp of the terms used by OPM, one must also read the entire application -- which clearly shows that the listing of your 7-year experience as a lawyer, bar number, and inclusion of a resume are not part of the AR.

Also, my post was in DIRECT response to the post above me giving advice that someone should have their AR "scored" by others . . .

I understand your wife example, but it does not apply in this case nor to my post.

My quoted post deals with ONLY the AR and not to any of the parts of the application outside of the AR -- the bar number, the 7 years requirements, etc. As defined under "Qualification", the Assessment Questionnaire consists of three distinct parts: "Experience (7 years), "Licensure" (licensed to practice with bar number), and the "Competitive Examination", which deals with the competencies (AR).

If you read the ALJ application, it clearly distinguishes the part concerning the Experience and Licensure as the "Assessement Questions" (AQ). Once the AQ have been answered, the application clearly moves to "Examination Part A. Accomplishment Record" and starts with "Competency 1: Decision Making". . . ". Thus, the AR does not begin until Competency 1 and does not include either your 7-year exprience or bar number. In fact, the title "Accomplishment Record Instructions" does not start until after the completion of the bar number (Assessment Questions).

Moreover, I again must quote the "Certification of Information Accuracy." It starts with "I certify that, to the best of my knowledge and belief, all of the information included in this Assessment Questionnaire is true, correct, and provided in good faith." Please note the terms "Assessment Questionnaire" which as defined under "How to Apply" includes the the Assessment Questions and Accomplishment Record.

The sentence RIGHT after that is interesting. It states that "I certify that I have worked independently and did not get any form of assistance while completing the Accomplishment Record." Please note the change from Assessment Questionnaire to Accomplishment Record. This change, under any rule of statutory interpretation, clearly means that OPM intended to make the distinction between the two.

The distinction is also clear to those who were denied based on failure to correctly complete the Assessment Questions, because you will not even get a score. If you go to "How You Will Be Evaluated: Basis for Rating," you will noticed that the application once again distinguishes the Assessment Questions and Resume from the AR. First, your AR is not graded until after OPM determines that you had correctly completed the AQs. Second, if you failed to meet the requirements of the Assessment Questions . . . you do not get a score of "0" but an "ineligible" score instead . . . the "0" is given to those who successfully completed the Assessment Questions but not the AR.

Because, as defined by OPM, the "competitive" part of the application does not begin until the AR, the certification places a greater burden as to independence and assistance on completion of the AR than the AQs -- the actual scoring begins with the AR and not the AQs. As noted in my first post, I suspect that for the AR, OPM is not only concern with what you did but also on how YOU (and not someone else) wrote up your experience. With scores stretched out to the 1/100th of a point, it doesn't take much "assistance" to tip the balance in your favor over someone else.

Many on this Board are very careful about offering too much advice/assistance on the WD and SI. A similar standard of caution may also be appropriate for the AR. Per the application instructions, all three sections (and only these three sections) are part of the "competitive examination" and the scores in each are combined to come up with a numeric rating. One could arguably say that the AR is even more important than the other two because if you do not do well on the AR, you don't even proceed to the WD and SI. Given the unique role of the AR over the WD/SI, OPM's desire to place a greater burden (limitation) of what constitute assistance/independence on the AR over any other parts of the application is understandable.

(Please also note that your resume is also a part of the application distinct and apart from the AR and AQs. There is no such prohibtion for someone assisting you on your resume either, much like the AQs -- I suspect OPM also considers the resume as a "non-competitive" part of the application.)

What if you wrote your AR now and have it scored by friends who have made it through the process and then submit them when the application process reopens? Would this action occur during the "completing" phase of the AR? . . . Does the application process need to be open for you to be in the process of "completing" your AR . . . Is it possible to work on your AR without being in the process of "completing" the AR? Is it possible to have others help and not be under the "any form of assistance" requirement? If the term "assistance" was meant to be narrowly defined, then why did OPM use the phrase "any form of"?. If you seek and receive help, can you still work on your AR "independently"?

I understand that we are lawyers and can play games with words and definitions all day long . . . no one is really sure what OPM means by "independently," "any of form of assistance," or "completing" . . . until I receive/see/hear any official announcement, however, for my sake, I am going to keep my definition of OPM's words/terms using the KISS principle.

The certification part of the AR relies on one's honest interpretation and honor code . . . Because I do not exactly know how OPM wants to define these words/terms, I will just err on the side of caution when giving advice on what constitutes assistance and being independent during the completion of the AR . . .

Obviously, self-policing requires some amount of common sense (the wife example is not needed). My first post was not meant to cover every conceivable scenario concerning this matter nor was the intent of my first post was to offer any specific advice (which I didn't) on what to do. My first post's intent was just to throw some caution to people to not treat assisting others on the AR so cavalierly.

Although the AR is the "take-home" part of the exam, it is still part of the competitive exam process and should be completed as independently as possible.

[ishmael]

Mar 8, 2009 21:08:41 GMT -5 privateatty said:

Thanks ishmael for pointing this out. Obviously, I gave bad advice and need to address it here.

I need to point out that writing a better AR was an issue that was addressed last year on this Board--the thought being that extolling one's virtues in the first person sometimes needs a third person perspective. Some people can step outside of themselves, others can't. I did not mean to infer that one would use another's words or direct thoughts. Never even thought of that. I know no one else who did, so my experiences are different than yours.

Other posters here have spoken of "different answers" to life experience descriptions. Is it improper for some one to read your AR and tell you to "beef it up"? Under your definition probably, although I don't think you would find much support for that.

That having been said, I think that it is paramount that we all operate from the same ethical base we were taught in law school. Frankly, what you caution has not been well addressed on this Board because we almost all assume a high degree of ethical behaviour.

My problem throughout this whole process is that it is skewered to those who know what they are doing. Who understand how to interview, do an effective AR and write a good exam in the format demanded. I do appreciate your thoughtful response and admonitions.

Thanks privateatty. I reread my first post and didn't intend for it to attack anyone specifically. I was just trying to warn some who may take the "assistance" too far . . . I understand your clarification, appreciate your kind response, and agree with you.

[Legal Beagle]

Whew! I was worried that my having my resume professionally done would be considered "getting assistance." Since the Register opened for applications on one day, I spent the better part of that and the next working on the AR, submitted it, and that day it closed, I could not see how anybody had time to get "assistance"!