As an addendum to the Summary Report for Social Security Administration Contract #600-95-21914, Analysis of Functional Assessment Instruments for Disability/Rehabilitation Programs, the Department of Physical Medicine and Rehabilitation was requested by SSA to review and comment on the Westat protocol for the proposed Disability Evaluation Study (DES) to be conducted by SSA or its contractors (Task 2 of Modification #7). This report summarizes that review. First, we will respond directly to questions presented by SSA under Task 2 of the contract modification #7. Second, we provide specific comments to the Westat protocol as required under Task 2, Subtask #1. Comments are organized by (1) section of the protocol and (2) page number references or, for instruments appended to the protocol, item numbers.

I. Responses to SSA Questions Regarding the Westat Protocol

A. Will the proposed screening interview screen in wide ranging functional impairments, as well as medical conditions as per the original goal of the DES?

The screening instrument would appear to adequately screen individuals with physical or health impairments. However, the screening instrument may underreport the following conditions: muscle and pain disorders, mental disorders and disorders with fatigue as a prominent symptom, as will be explained in the following paragraphs.

A subject with a musculoskeletal injury or disorder which causes pain may respond that they have no difficulty with the individual activities listed in section [c] such as reaching over the head, bending down to pick up an object, etc., if the subject perceives the question to be: can they perform this activity as a one time event? Yes, they can do this action although it may cause pain at the point of doing the activity or at some point post activity. However, required repetition of the activity or combined activities will multiply the effect to the point where the subject is unable to perform the activity due to severe pain. Also, in musculoskeletal disorders the subject may be able to perform these activities repetitively and in combination for several days but then cannot perform any of the activities without several days of rest. This is why many functional capacity evaluations which are job specific will evaluate the subject over one to two weeks to assess the cumulative effect of the activity on the subject's level of functioning. Inability to do specific activities consecutively and consistently may be due to many causes such as low cardiovascular endurance, poor muscle endurance and strength, limited muscular flexibility, as well as due to the specific injury or muscular disorder such as myofascial syndromes, fibromyalgia, etc.

Subjects disabled from disorders with fatigue as a prominent symptom may be underreported on this screening because the screener focuses on individual activities and on diagnosis/conditions. It does not allow for assessment of performance of multiple tasks in succession, repetitive tasks, or for evaluation of consistency of performance over days and weeks. This issue of fatigue impacting ability to do multiple tasks in succession, repetitive tasks, and multiple activities in one day or consecutive days is very important. Fatigue can have a significant impact in many medical disorders including all connective tissue and autoimmune disorders (rheumatoid arthritis, lupus, Sjogerns etc), multiple sclerosis, post polio syndrome, neuropathies and myopathies. Energy conservation techniques must be learned and implemented in these conditions. Typically the fatigue is unpredictable and inconsistent. Another reason that the screener may underreport related to subjects with limitations from fatigue is that it is such a nonspecific symptom that the subject may not have a diagnosis or may not have pursued medical assessment. Since the screening does not allow for fatigue-related inability to perform ADLs and IADLs, if the subject does not have a diagnosis then these limitations/disorders will not be reported.

Subjects may reject completing this screener because is it lengthy and very repetitive and must be done for each member of the household. Repetition could be reduced by orienting the questions around function as opposed to orienting around condition/diagnosis. The opportunity to get specific information on the condition/diagnosis is fully available on the Medical History Assessment. So if the purpose of the screening form is to screen in any conditions which may have functional limitations the screener should focus on identifying functional limitations in all ways possible. To ensure accurate answers the wording should be changed to more functionally oriented answers in layman's terms. For example items B4 and B5 page 3 ask the following:

"Does any impairment or health problem now keep you from working a job or business? What condition causes this?"

More understandable wording would be "What prevents you from working?" This gets down to the "real cause" of inability to perform occupationally. The subject may have a condition or diagnosis, but the primary factors which prevent them from working may be transportation related, no jobs available in the area, or discrimination due to disability. This screening form does not identify what prevents the subject from working but instead focuses on the medical diagnosis which may be misleading and lead to underreporting.

For example, the subject may say they have chronic fatigue but no specific diagnosis has been given. It may prevent them from working but the subject may answer on the specific activity related questions that they can perform the activity independently such as eating, dressing, bathing. So this subject may be viewed as able to work and assumptions made about lack of motivation. A more functionally oriented screener may identify that, although the subject is able to do specific activities in isolation, due to the chronic fatigue condition they cannot bath in the same day that they must walk up ten steps.

The screener may overreport inability to work depending on how the data are analyzed from the form in regards to cognitive limitations compared to physical limitations. Question: Will cognitive limitations be separated from physical limitations?

The section on ADLs/IADLs does not allow for specifics on assistance needed. A subject may not be able to eat independently and require assistance but the reason may be as simple as needing set up or opening of containers as opposed to the more intensive need for supervision of swallowing to prevent aspiration. These are examples at the opposite ends of the spectrum and will obviously impact a subject and their ability to work very differently. This screening form does not allow identification of where the subject is on the spectrum of needing assistance.

B. Does the DES main questionnaire and performance protocol contain all the self-reported items that are important within the domain of self-reported functional assessment instrumentation, particularly in relation to the self-reported instrumentation identified as promising in the earlier work performed by the contractor?

The DES performance protocol, we feel, is limited by the following overriding factors:

Measures included in any functional capacity battery should have sufficient statistical power (i.e. validity, reliability, sensitivity). It is not clear that this has been established for the data collection protocol prior to its use in a large-scale national prevalence study.

Nonspecific tests such as on the performance protocol do not reflect actual demands of a job. Subjects may not exert themselves to a maximum effort due to pain or fear of further injury.

Performance measures are the product of both physical and psychological factors. Subject effort can also be influenced by fatigue, depression, and pain. This is not addressed.

The performance protocol requires professional skills in performing and analyzing the data and requires the ergonomic expertise in linking the findings of the performance protocol to job based demands based on ergonomic job analysis. There is no allowance for these factors.

The performance protocol is a one-time evaluation and findings therefore reflect an individual's abilities at the time of testing.

The performance protocol does not address psychosocial factors. Some authors emphasize that psychosocial factors are more important than physical factors in the development of disability. Many authors have recommended use of FCA protocols which are based on psychophysical techniques and the concept of "acceptable maximal effort," or AME This concept refers to quantitative methods of measuring functional abilities in chronic pain. AME is the highest level of voluntary effort that a person can achieve without inducing unacceptable pain. Matheson states that measurement of physical factors alone and apart from psychosocial factors is "fiction" and any assessment of range of motion, or strength has significant bias and subjectivity.

7. A weakness of the DES main questionnaire and performance protocol is that they are performed at separate times by different individuals. This approach is common; however, the performance protocol gives no indication of state of mind, presence of pain, fatigue or depression which the subject might be experiencing at the time of the protocol administration.

8. The main questionnaire is too exhaustive in its collection of history, conditions, etc. We are concerned that subjects will become too fatigued or irritated by the questionnaire and respond inaccurately, even if compensated for their participation as suggested in the protocol. The medical questions are extensive and range from general nonspecific symptoms to system based reviews to toxic exposures, psychiatric disorders and injuries and seem to be based on standard medical student training questions as opposed to a more sophisticated disability-based medical history questionnaire. However, the medical history questionnaire is quite complete in its attempt to capture information on medical conditions, their symptoms, impact on ADLs, medications, related hospitalizations and surgeries, side effects of medications, effectiveness of medications, treating physicians, treating institutions and onset, frequency of symptoms, etc.

In comparison with the assessment instruments which have been reviewed, we believe that the influence of behavioral and environmental factors are being overlooked in this assessment. Neglected in the main questionnaire are assessments of coping skills, pain levels, associated distress, social supports, any tendency to somaticize, familial perceptions of the individual's disability, and return to work expectations. Rudy et al have reviewed the current inadequacies of FCAs when it applies to chronic pain and reviewed the many internal and external environmental factors that can significantly influence FCAs and identified some psychosocial factors that play a role in subject's performances.

Functional capacities assessments are considered by many to be behavioral observations. Not included in the protocol assessment are any instructions as to whether family members can watch the assessment or provide supplemental data. This may influence performance both positively and negatively but perhaps mostly in a negative sense. Regardless of the influence there should be consistency in how the protocol is administered and guidelines given. In some cases where subjects have communication deficits a family member may be crucial in effective communication between the examiner and the subject.

Another consideration is this: For what reason is the subject told the performance protocol and main questionnaire are being administered? Purpose for performance can affect motivation and effort. This needs to be clarified and specific consistent scripts provided in this area.

We have noted that the main questionnaire and performance protocol have limitations regarding the assessment of individuals who experience chronic pain. Individuals may be able to complete some or all of the tasks independently, but would not be able to complete them to the demands of a job or to daily functioning in the home.

In addition, not included in the DES main questionnaire and performance protocol within the domain of self-reported functional assessment are the following measures:

Behavioral Measures

• time spent lying down, sitting, standing for routine day
• observable pain and illness behaviors ( guarding, bracing, grimacing, vocal complaints)
• medical measures related to pain
• intensity, impact on ADLs, patterns,
• tests to assess pain behaviors, exaggeration or magnification of symptoms
• sleep patterns

Psychosocial Measures

• social activity
• leisure activity
• marital and family difficulties
• psychological distress related to disability
• coping strategies
• denial, self blame, irritability

C. Are the performance protocols administered as part of the main interview adequate and appropriate, particularly in relation to the performance-based functional assessment instruments identified as promising in the earlier work performed by the contractor.

The performance protocols appear to adequately address the range of performance-based physical functional assessment instruments reviewed in our earlier work. We have concerns, however, about specific task performance protocols and assessment criteria as compared to those which were reviewed previously. These concerns will be addressed here.

There is no identification of reasons why the subject "cannot do" certain tasks in the mechanical, physiological and psychological/motivational areas. For example, for Task 6 (upper extremity strength) the subject has the opportunity to identify that they did not attempt to lift because they felt it was unsafe. Also, Task 10 (putting on blouse) gives the examiner the opportunity to ask the subject to explain why the task was not attempted. However, in general there is inadequate behavioral and psychosocial assessment of why tasks are not completed. Nor are there indications of what the subject has been instructed by health care professionals that the subject "should not do" because doing such activities will aggravate the underlying medical condition, delay healing or constitute risk to the subject and others.

There are no overall cardiovascular/respiratory or other disease-related symptom endpoints used.

There are no task specific endpoints identified which will negatively impact the reliability and validity of the physical assessment testing. The lack of general and task specific endpoints is considered a major deficiency in the physical performance assessment from an accuracy standpoint and from a medical legal liability standpoint for SSA.

While there is lifting evaluation there is absence of carrying evaluation - the subject's ability to carry the loads that he was able to lift needs to be assessed through the use of a structured task simulation. One standard method of conducting a structured task simulation is done by the subject carrying a load he was able to lift over a flat and unobstructed surface on a 100 foot course at approximately 3 miles per hour. After a 20 second standing rest, the load is incremented by 10 pounds and the subject completes another cycle. The test progresses in this manner until the maximum acceptable weight is reached.

The strength testing is not dynamic strength testing but static and thus less useful and less indicative of true function than dynamic testing.

Inadequate flexibility testing is performed.

The "Bending Over" assessment could be considered a spine and hamstring flexibility test but there are no side bending or spinal rotation assessments for range of motion or flexibility.

There is an absence of position tolerance testing; the main questionnaire does address this but relies completely on self-report, which is very subjective and often inaccurate.

The endurance testing (measured walk and rapid walk) are very limited and in no way reflect the demands of an 8-hour work day.

There is an absence of observations in regards to the subject's motivation, maximal effort, consistency of performance of activities, pain behaviors and illness behaviors, safety of function and noting of specific abilities or deficits demonstrated. Effort can be evaluated with a variety of scales (Hazard Scale; Mayer, Jamar dynamometer) or a three-level rating scale which focuses on effort during the test. The scale ranges from "reliable effort" to "questionable effort' to 'unreliable effort'. Examples of each would have to be included. This is more subjective than the more objective Jamar dynamometer testing of grip strength for effort. Task 7 (grip strength) does use the Jamar dynamometer but it is not clear whether the data are analyzed for effort assessment.

No allowance for impact of repetitive activity or the effects of sequential activity over time.

We have concerns that physical performance testing appears to be performed prior to a clinical assessment by a physician. This might put SSA at risk should individuals experience an injury or aggravate an existing injury during the performance testing.

D. Is there sufficient observational data recorded by the interviewer during the performance protocols used in the main interview?

The interviews and performance protocols are extremely thorough. As mentioned previously, we are concerned about the issue of fatigue developing during the administration of these lengthy interviews and functional performance. There were no opportunities to document amount of time spent in the interview, whether rest periods were taken, or whether the protocols were administered one after another vs. on different days, or whether the onset of fatigue or other factor are observed which might impinge upon the accuracy of the data.

This could greatly impact performance on the performance protocol testing. For example, it will obviously take several hours to complete the screening medical interview, the functional assessment interview and others. If the functional performance testing is done last the exertion required to answer and interact with interviewer for other assessments could have a major negative impact on the performance of cognitive tasks (e.g., check book balancing) depending on the disability, illness, and overall condition of the subject.

We repeat our concern that there is no observational measures of maximal effort or identification of why the subject cannot perform certain tasks in the mechanical, physiological and psychological/motivational areas (with the previously mentioned exceptions of Tasks 6 and 10). In general there is inadequate behavioral and psychosocial assessment (completed through either observation or from the subject's self-report) of why tasks are not completed. Nor are there indications of what the subject has been instructed by health care professionals "should not do" because doing such activities will aggravate the underlying medical condition, delay healing or constitute risk to the subject and others.

E. Does the medical exam protocol obtain sufficient physician input with respect to ability to function, especially in context of physician completed functional assessment instruments identified as promising in the earlier work or as they might be used in order to corroborate or validate self-reported information?

The medical history and exam protocol are not going to be as useful in validating self-reported information as was used in many of the instruments identified as promising. The information obtained in the Westat medical exam and history are in such a form as to be a dictation of subject self-report. Also, the way the information is categorized and separated in the medical history and exam sections vs. the functional assessment questionnaire makes is difficult to get an accurate assessment or prediction of function.

The medical exam protocol is extensive in its collection of physical medical data. The medical history regarding function is strictly the subject's self-report and there were no instructions seen which would have the physician or interviewer obtaining reported performance from subjects in any indirect measures or manner. The functional assessment questionnaire is, again, a subject self-report, which is extensive. The functional performance testing gives parameters, definitions, and scripts. It covers a wide range of functional activities as found in our review of assessments, primarily the assessments of physical functioning, cognitive functioning, and ADL/IADLs. However, the functional assessment is limited in its assessment of pain associated with physical activities, and assessment of non-cognitive mental functioning is almost completely ignored. Physicians and psychiatrists will therefore have little information on which to base clinical decisions for individuals with these impairments.

It is not clear how the functional assessment questionnaire and the functional performance data will be presented to the physician. The visual format in which the physician reviews this data, the opportunity for the physician to review this data prior to completing the functional assessment predictions, and the amount of time spent interpreting this data and comparing it to the findings on the physical exam, all will impact the accuracy of the physician's predictions.

The medical exam protocol by itself does provide a great deal of physical and health-related information about the subject. The amount of information it provides to the physician with respect to ability to function will depend on the training, experience, and abilities of the physician to use physical exam data to predict functional abilities. Predicting functional abilities, in fact even thinking of the subject from a functional capacity standpoint, is not taught in medical school or most residency training programs. The addition of self-reported functional performance information and the results of the functional performance testing will greatly enhance the ability of the physician to use the medical exam protocol to predict function.

The issue of effort expected to be exerted by the subject or amount of effort felt to be exerted by the subject during testing is not addressed. It could be addressed during the grip strength measurement using the dynamometer. The issue of physician and interviewer impression of effort expected to be exerted by the subject is important from the standpoint of a subject not giving full effort due to fear of injury, exaggerating symptoms, fear of giving full effort during this testing because the subject knows/fears they could not sustain the same level of effort consistently throughout the day or from one day to the next. The issue of not giving full effort needs to be documented and the interpreted reasons for not giving full effort explored. The issue of effort is also important for the subject who may overexert due to desire to impress or please the examiner, wanting to show they are trying hard, poor judgment for safety and ergonomic issues. The overexertion/extreme effort issue should be recognized and documented by the examiner related to risk of injury.

II. Specific Comments on the Westat Protocol

Section 1. Introduction

Section 1.3

All research objectives refer to comparing data to the pool of people who meet current standards for SSA disability. Unfortunately, from a pragmatic standpoint there are NO clear standards. For example, all chronic pain patients are initially denied disability on first application, i.e. not meeting SSA standards. Any patient persistent enough will continue to reapply and will eventually be awarded compensation. Usually there has been no significant interim change in medical status. Which STANDARD is to be used, the initial one or the eventually-revised one (which actually represents those out on disability)?

Section 2. Research Plan

Section 2.3

Page 2

The plan states the respondents will be asked to participate in physical performance testing. Who will perform these tests? What are their professional expertise? This physical performance testing appears to be done before the physician evaluates the individual. Will the person doing the performance testing have enough knowledge to recognize subjects at risk for cardiovascular events and musculoskeletal injuries secondary to the performance testing? What endpoints are used (both for general ending of the testing due to cardiovascular risk and specific endpoints for each function due to risk of musculoskeletal injury)?

How much time is allotted/expected for the in-house physical performance testing? The length of the screening questionnaire combined with the in-house physical performance testing will probably be very fatiguing for many subjects. The amount of fatigue experienced by each individual and the impact of that fatigue on the physical performance testing will make standardization difficult and accuracy of assessment more complicated.

What is meant by "manipulative" in the physical performance testing?

The plan states that following the interview and performance testing an appointment will be made for a medical examination.

Will medical history questions on screening be used to rule some individuals out for physical performance testing and are the qualifications of the individual conducting the interview such that they can make that determination prior to evaluation by a physician?

This reviewer is very concerned about physical performance testing being done on individuals prior to evaluation by physician and feels this places SSA at some risk and liability for injury sustained. Injuries could be due to worsening of disease process due to over fatigue, cardiovascular event, and musculoskeletal injury.

Does the experience of the examiner allow for adequate evaluation of exertional and postural functions as well as cognitive evaluation, especially in a disabled population where evaluation of these functions are very complex and difficult to assess accurately? In addition to potential liability for SSA there is significant potential for under reporting of functional capacities by incorrectly assessing function in some disabilities.

Is there a consent and removal of liability form which the subject will sign prior to performing the physical performance testing? What decision points are used to determine which individuals interviewed will have an evaluation by a physician? Are these decisions reviewed by anyone else to identify incorrect decisions?

The plan states that medical records will be obtained and summarized for use by the team conducting the physical examination.

Who will obtain these records? How will these records be summarized and who will summarize the records? What are their qualifications? Medical records are notoriously difficult to obtain in their entirety even when individuals are applying for SSA Disability and have incentive to accelerate the process. This plan grossly underestimates the difficulty in obtaining complete medical record information. Summarizing appropriate medical record information is difficult to do. Individuals with significant experience in the medical field and disability field will be needed to do an appropriate job; for example, a rehabilitation physician, nurse practitioner or rehabilitation nurse/case manager.

Section 2.4 Medical Examination and Psychological Assessment

Page 2-3

The plan states a physical examination by a physician will consist of review of each body system. An exam that evaluates each body system will be quite long and complex - prohibitively so.

Is review of each system really required? How much will be left to the judgment of the medical practitioner? Will the purpose of the exam be to review each body system and document all medical illnesses and potential functional limitations or to identify illness/disorders which are causing functional limitations. The difference in these two approaches may impact amount of time required for the assessment and the training/experience of the medical examiner.

The plan states that in addition to the medical history data obtained from the in-person interview that appropriate medical and hospital records will have been collected. Again, this is very difficult to do in reality. After completing 1200 evaluations of chronic pain patients in the Pain Assessment Instrument Development Project and as clinicians routinely consulting on patients, we find it alarming the number of participants for whom we were unable to obtain medical records in a reasonable amount of time. In some areas where medical records are automated and moving to a paper less process perhaps this may be less of a problem. However, there needs to be a recognition and contingency plan for not having an individual's medical records available for review.

2.5.1 Use of Traveling or Local Interviewers and Medical Teams

Page 2-3

The training provided for either local or traveling interviewers and medical teams must be delineated completely before assessment of adequacy can be evaluated.

2-4

What level of education, experience and training will be required for the team of interviewers? "Centralized training" needs to be defined. We agree that a traveling team would become "experts" in administering the questionnaires and physical performance testing and the data from this team would be more consistent and reliable than that of local teams.

Is there any contingency for assessing "drift" in the team from original training and for doing appropriate follow up when this occurs?

Please define "refusal conversion".

2-4 It is agreed that traveling teams would be more dedicated to the study, more easily scheduled to complete examinations, adequately trained and the collection of standardized data would be more likely.

Is there a plan to assess reliability of assessments? Both test-retest and interrater reliability measures are essential to the successful completion of the DES objectives.

2-5 Use of local physicians is considered to be very difficult because of differences in physician training and experience in evaluating function as well as the fact that physical exam techniques are not standardized.

There is no mention of payment/incentive to the local physician to do these exams. The broad scope of the evaluation required as opposed to a more problem focused assessment will be quite time consuming for the physician and might produce low participation rates unless a considerable financial incentive is provided that would come close to typical reimbursement for the amount of physician time required.

Using local doctors may not be feasible for proper standardization because the evaluation is too complex and lengthy. The protocol as written will require trained physicians, each doing a large enough set of assessments to become proficient; this would be very costly. We do not see how local physicians can perform more than 1 or 2 assessments per day, in addition to their other duties. SSA may have considerable difficulty obtaining local physician agreements. Also, given a rough physician cost of at least $200/hour, the cost of the examinations could be prohibitive.

Section 2.5.2

Page 2-9

Plan states incentives should be offered to study participants to encourage participation in the DES. This will probably have to be broken down into incentives for the screening questionnaire, the physical performance testing and the medical examination.

Amounts will differ and there should be consideration for the need to increase incentives in order to get an appropriate sample. The starting rates of incentives seem accurate.

The plan states the MES study population will consist of persons who are seriously disabled. Shouldn't this study evaluate a broader population? What is the definition and range of " less seriously disabled" in this study? On page 2-15 the plan states that the sampling plan for the MES includes 3 samples (severely disabled - meet SSA disability eligibility criteria, less severely disabled, and nondisabled).

For this instrument, Non-response will definitely be a problem, for both the subjects and for the physicians. Unless the physicians are paid significant hourly fees (with the exorbitant cost as noted above), there is no way an instrument this long will get complete valid responses.

2.5.3 Consent/Human Subjects Issues

Obtaining informed consent is complicated by the physical performance testing component of the evaluation. The consent form should include language regarding risks of physical performance testing and waiving of liability for the agency/team.

2.5.4 Using Proxy Respondents

Page 2-11

Although the plan states that research has shown that proxy respondents are much more accurate in responding to questions about specific physical conditions than to more subjective questions, the PAID project found that significant other data did not add to the reliability or validity of the data collected as a whole and this should be taken into consideration in decisions regarding use of proxy respondents.

2-11 The protocol discusses using proxy data and medical exam and tests to help, but will they compare this to proxy data obtained from patients who can give responses?

2.5.6 Home Examinations

Page 2-12

The protocol states that a nurse would be used to do home examinations and that given the nature of the exam a nurse could complete only one per day. This seems to be a very low number and emphasizes the considerable size and complexity of the data being collected. An RN would probably not be adequately trained to do home examinations. A nurse practitioner would have to be considered and even then there are issues related to experience which will be addressed in other sections.

The forms to be completed in general are very long and repetitive and consideration must be given to ways to decrease the size. Studies eliminating items based on poor reliability and usefulness of data could be considered.

2-14 Regarding testing arthritis, the examinations may get different results at different times of day. For any patient, motivation is probably biggest determinate. How can these factors be built into the DES format?

2.5.8 Determining Disability from MES screener

Page 2-15

Section 3. Sample Design

3.1 Introduction

It is heartening to note that the report actually attempts to relate the sample design to the research objectives. This seemingly obvious task is all too often ignored in the sample selection and sample size calculations of research studies. Within this section of the plan four possible approaches to the second phase of sample selection are identified, each with concomitant problems and expenses. Multiple design proposals are prudent at this point, but do not allow an adequate identification of potential costs. Rather than provide a detailed review of each of the designs presented, except to say that they are well thought out, the reviewers focus our attention on the relationships between phase one and phase two of data collection and how they in turn relate to the research objectives.

Research Objective #1 is to "identify the total number and distribution of persons with relevant medical, vocational, and functional limitations and impairments as a basis for estimating the universe of potentially eligible applicants who might apply and meet or equal the listings and other standards now in use, regardless of current employment above SGA." All subsamples to be used in Research Objectives #2 through #5 stem from this identified population. It therefore logically follows that this population would be identified prior to performing the sample selection for the subsequent research objectives. However, the proposed study design apparently tries to skip a step and conducts sample selection for all research questions by using the prevalence estimate of 3.5% for disabled nonbeneficiaries. Although this estimate is a good initial estimate, it would be much more cost efficient to design the study with a staggered sample selection, allowing for the fulfillment of Research Objective #1 prior to the creation of samples to test the subsequent research objectives.

Partially because the authors are uncomfortable with the prevalence estimate upon which they are basing their proposed designs, they have identified four possible approaches to sample selection. The decision of which design to use for the second phase must be made at some point. As presently written, it appears the protocol presents the alternatives and leaves it to SSA to make the design decision. If Research Objective #1 is viewed as a separate phase, to be fulfilled between phase one and phase two of data collection, the researchers will be able to base the second phase sample design decision on real prevalence figures rather than estimates.

We will more fully develop this sampling methodology for the Task 2, Subtask #2 report.

Section 4 Data Collection

4.2 Screener

Page 4-1

The reported goal of the MES household screener is to determine eligibility to participate in the interview and examination phases of the MES study based on data collected over the telephone or in person on reported disabilities, medical conditions and functional limitations. The plan states that for individuals who screen into the MES study, data collected from the screener will be used to identify areas where more detailed information will need to be collected during the interview or examinations.

Will an algorithm be developed that will identify which interview questions and which physical examination sections will be performed from items identified in the screener? If so, how would this work? When and how will the algorithm be developed and validated? If there is such a process it would be an important part of this study in helping to decrease the significant amount of information required much of which may be inappropriate or redundant, for example: what does the presence of genital warts have to do with impairment, disability, functional capacity or the ability to work?

4-2 The plan states that "measurement of ADLs and IADLs in the screening phase of the study will ....". Do writers really intend to do actual measurement of ADLs in the screening phase of the study? It appears this is simple self-report of ability to do ADLs and IADLs as is the rest of the screening form.

4.3 Main Questionnaire

Page 4-5

A number of different sources to identify appropriate measures to be included in the questionnaire to meet the research objectives are identified. Are these sources/questionnaires included in their entirety in the main questionnaire or are pieces used? Have these sources/questionnaires been shown to be valid and reliable? Who will be administering the questionnaire? Will it be broken down into sections and different sections administered by different individuals? Will the main questionnaire be completed at one sitting/evaluation period?

It is extremely complex and lengthy in order to obtain all the information desired. The element of fatigue for the subject must be taken into account as well as the impact on accuracy of information.

4-9 The plan states that a knowledgeable member of the household will complete all the questions related to certain areas for each person in the household. What are the qualifications of the individual collecting this information? It is not clear whether this part of the interview will be in person or over the telephone. If it is in person, will multiple members of the household be expected to be available for this interview. The most knowledgeable member of the household for this section may well be employed and thus access to this individual for lengthy interview limited. Also, the "most knowledgable" member of the household may not be aware of the more personal data collected by the questionnaire.

4.3.2 Medical History Section B

Page 4-9

The protocol states the medical history section is designed to collect "detailed information on each participant's history of diagnosed illnesses and chronic symptoms experienced within the past year". This is an extremely broad statement and the medical history form is an extremely long form (consisting of 110 pages of very small type and 748 questions). Attempting to collect data on all chronic symptoms regardless of whether they have any impact on function or work performance will be very cumbersome and data intensive. This data, for example may include rhinitis (runny nose), post nasal drip, vaginitis, etc., which, while chronic and conspicuous to the individual, may have no impact on function.

The protocol also states that these data will guide the physical examination and will help identify the body systems requiring in-depth examination and identification of conditions for which medical records will need to be obtained. Again the question arises, what will be the training and experience of the person collecting the data and making these decisions, especially regarding identification of conditions for which medical records will need to be obtained and which areas need to be pursued with in-depth examination?

For example, an individual may mention that they have flat feet which may be considered to be a benign, common condition which could be considered to have no impact on function. Pursuing this information further with the correct specific questions may disclose that the individual has shin pain and back pain with prolonged standing and physical exam may reveal "miserable malalignment syndrome" (a functionally important disorder) or merely pes planus (flat feet). If there are alignment and lower extremity mechanical problems associated with the flat feet such as in miserable malalignment syndrome the individual may need in-shoe orthotics, may be unable to walk long distances, or stand on hard surfaces such as concrete for prolonged periods of time. These issues may or may not have been identified to the subject as related to each other and may or may not have been adequately evaluated and treated. Flat feet with no symptoms or other musculoskeletal mechanical dysfunction have no impact on function or health status.

Another example is a high arched foot which may be considered also to be insignificant if collected by an inadequately trained individual. Appropriate further questioning might reveal a family history of high arched feet, progressive distal extremity neurologic dysfunction with muscle atrophy which would reveal a congenital neurologic disorder which has functional implications. Thus, the qualifications, training, and experience of the interviewer will be key to the successful identification of disorders, and especially if the remainder of the medical evaluation is to focus on only the items/disorders considered important and identified in the main questionnaire.

There is concern for underreporting of certain diagnoses or symptoms and over reporting of symptoms and diagnoses both of which may impede the purpose of the project. The knowledge base and experience of the interviewer is key for these reasons unless the medical evaluation will be starting from scratch. However the plan seems to imply that the medical evaluation will follow the data collected on the screening interview so the qualifications, training and experience of the person administering the screening interview are also very important to meet the research objectives of this project.

4-10 The medical history sections identified on which data will be collected are interesting in their selection and we are curious as to the reason for each selection. Some are standard medical history questions but which have no direct functional or performance implications such as tobacco use. While tobacco use is believed to lead to the development of certain respiratory disorders and the development of certain cancers, tobacco use does not directly impact function and performance unless these disorders are present or the individual is asked to work in a smoke free environment and is unable to do so.

Medical history conditions reviewed on the medical history questionnaire: fever, chronic fatigue, dizziness, fainting spells, headaches, blood disorder or diseases, change in weight, cancer, surgeries; muscle, bone, and joint problems (pain, spasm, swelling, stiffness, weakness, disc problems, fractures, limited ROM), history of connective tissue diseases and other musculoskeletal conditions; amputations; conditions which impact ADLs; vision speech, hearing,; review of systems for respiratory conditions (symptoms, onset, frequency, diagnosis, doctor, medications for, impact of medications on condition, impact on ADLs) heart, circulatory , GI, renal, dermatological ,endocrine, neurological (seizure, CVA, TBI etc), toxic exposures; psychiatry; aids; other; accidents & injuries. This is comprehensive in range of topics.

Regarding side effects of medications - patients may not realize what they are experiencing is a side effect of medications. Terms used such as circulatory problems may be too technical for many subjects and need to be put in lay terms; again, the training of individual doing interview is key. SSA might want to add appropriate lay terms beside each condition or a list of terms commonly used to describe these disorders by the public (for example, for HTN, high blood; for anemia and for low blood pressure - low blood; urination - make water; brain injury - concussion; hydrocephalus - water on the brain) both for diagnosis & symptoms.

It is interesting that a large number of questions deal with cancers with specific questions regarding primary and secondary sites, when cancer patients are not listed as the more costly patients on the disability roles, as they do not remain on the rolls for lengthy periods of time. In contrast, there is a relative paucity of pain-related questions whereas chronic pain has a major impact on an individual's ability to function and work and is most often related to nonmalignant syndromes which can keep the claimant on the rolls indefinitely.

The medical questions are extensive and range from general nonspecific symptoms to system based reviews to toxic exposures, psychiatric disorders and injuries and seem to be based on standard medical student training questions as opposed to a more sophisticated disability-based medical history questionnaire. However the medical history questionnaire is quite complete in its attempt to capture information on medical conditions, their symptoms, impact on ADLs, medications, related hospitalizations and surgeries, side effects of medications, effectiveness of medications, treating physicians, treating institutions and onset, frequency of symptoms, etc.

4-11 Data will be collected on accidents and injures and physical and occupational therapy. It is interesting that there are no questions regarding response to therapy (outcome), pain related to accident or injury, or whether the person is doing an exercise program of any kind (for general health or prescribed for a specific musculoskeletal disorder). These questions would be more helpful in determining potential long term disability claimants than data on chewing tobacco, cigars, and pipes (tobacco use as one question would provide the information that this individual is at higher risk for cancer and cardiovascular disorders than the average population. It seems inappropriate and unnecessarily lengthy to go into the specifics of cigar vs. pipe vs. chewing, etc).

4.3.3 Functional Assessment

Page 4-11

The plan states the data from the functional assessment section of the main questionnaire will be used to determine the prevalence of functional limitation in the study population and to determine the relationship between self-reported functional data and findings from the physical performance testing and physical examination. The functional assessment is completely self-report and is derived from a number of commonly used assessments. The DES functional assessment will need to be validated with the intended administrators and with SSA target populations (i.e., working-age adults), perhaps during the pilot test phase.

4-12 While activities evaluated include lifting weights varying from 10-50 pounds, pushing and pulling, etc., the section does not address the following: the body level at which the lifting will be done (from ground to waist, shoulder or above head); whether twisting and turning while lifting is required; the number of repetitions required/performed; repetitive above head activity (very important due to the high rate of degenerative shoulder disorders which cause pain with over the head activity). If the examiners are not aware that these criteria must be specificed in order to obtain reliable and valid data, then these are major deficiencies in a basic functional assessment.

4-13 The walking and standing variables do not address surfaces (wood, asphalt, concrete), the need or ability to change positions, or the need/ability to rest. Again, these are substantial deficiencies in a functional assessment.

The additional questions which are to accompany the specific activities seem very laborious to administer and to assess.

4-14 The questionnaire asks if the respondent can work at a job in extreme heat, cold, humidity. The respondent may not know whether they can work in this environment unless they have had the opportunity to do so. Also important is whether the environment is stable or shifting - does an individual work for long periods in a meat locker or walk in and out frequently? A person with arthritis disorders in which thermal regulation is impaired (Multiple sclerosis, Chronic Fatigue Immunodysfunction Syndrome) and other connective tissue and endocrine disorders would not tolerate these working conditions but may not be aware unless they have actually worked in that environment.

While questions are included which deal with high levels of tension or stress, the questions are not specific regarding distractions, ability to perform multiple tasks at once, or the ability to work with dangerous and fast moving machinery.

The functional assessment questionnaire is lengthy, repetitive, and consists of 57 pages and 51 major section of questions with approximately 376 individual questions. There are approximately 19 opportunities to jump to the next section if the answer is no to the preceding question. While the medical history questionnaire is lengthy and repetitive, it collects more specific data than the functional assessment questionnaire. The length and repetitiveness of this assessment will not necessarily lead to accurate useful information on the subject's self-report.

Questions related to onset of functional problems will be difficult for the subject to answer accurately and likely to extend the time required for the interview significantly. While the reviewer realizes it is important to document for disability regulations that functional disabilities have been present for certain lengths of time it might be more useful to have multiple choice answers such as less than 6 months, greater than 6 months, greater than 1 year, greater than five years. The assessment should be dated for when it was completed and the estimated date of onset could be determined from the comparison of date of administration and length of time present if this information is needed to be obtained from this form.

4.3.4 Accommodations and Interventions (Section D)

This section will provide useful information particularly in regards to information on absence of accommodations and interventions and how this lack influences employment. The question, however, arises again as the knowledge base of the interviewer. This plan is asking the interviewer to have disability, medical and very specific vocational and occupational knowledge. Unless they have this knowledge, interviewing the study participant may not identify that certain accommodations and interventions influence lack of employment. The subject and the interviewer may not be aware of available accommodations or their potential impact. How will this be addressed in this study?

Assistive technology, by itself, is a very broad and complicated field. Many disabled individuals do not have assistive technology due to geographic location, lack of finances to fund the technology, or lack of practitioners who know how to identify the needs and utilization of such technology.

Page

4-15 How reasonable is it for a study participant to be aware of costs of services which they do not use or are not available in the area. Many disabled individuals are unaware of payment sources and assistance which may be available to them, so to plan to obtain this information from the study participant may be unrealistic. Perhaps the goal of this section is to determine the participant's knowledge of costs, payment sources, resources, accessibility etc.

However, the lists and choice of answers on the questionnaire are very good and may compensate for the above described concerns. Going through the questionnaire may be educational and make the subject aware of services, devices and funding sources of which they were not previously aware.

There exists a lack of parameters for some descriptors such as item D41 on page D-10 "very difficult, somewhat difficult, a little difficult, not at all difficult".

The show card for assistive devices is an excellent idea.

This questionnaire is 16 pages long with 59 question sections and 152 individual questions. There are six opportunities to skip to the next question section but there are also several sections where questions must be repeated for each device or service used. This is lengthy but provides an abundance of information and potential education to the subject.

4.4 functional Capacity Measures: Physical Performance Testing

Page 4-15

The plan describes the purpose of tests used in vocational rehabilitation. We know that although various testing procedures may be used they typically are not standardized, may or may not be job specific, and have not been proven to be valid or reliable.

4-16 Westat states they chose tests which were able to be administered by lay interviewer and did not require heavy or expensive equipment, which are reasonable points. The tests which are chosen should give an overall assessment of basic function. However, the issue of accuracy of data collected, especially in the disability population where it requires learned skills to assess function in those with disability, must come to the forefront as problems with this plan.

The upper extremity strength test does not address repetitions or level of lift (waist, floor, above head) all of which are crucial to evaluation of function of ability to lift. Lifting is not a task that is limited to evaluation of the upper extremity but is an evaluation of the lower extremities, coordination of movements, balance and the spine as well. This issue is not addressed with this test.

4-17 The Westat protocol does a good job of summarizing some of the limitations and problems surrounding physical performance testing and predicting ability to work. Under number [3] regarding the difficulty of measuring the extent to which the subject is performing at maximum capacity there have been studies done with the Jamar Dynamometer looking at grip strength and the ability to accurately assess effort from this testing.

4-19 The tests listed under postural testing include one range of motion, two manual muscle tests, both of which are not good functional measures of postural function. Station and gait then are listed but again we believe that Westat needs to define what they want to test under the heading of "posture." The tasks which are listed test knee extension and hip flexor strength which impact ability to kick and to sit from a laying down position but missing are tests of hamstring and gluteal extension for rising from a chair and climbing stairs. These skills are evaluated and perhaps are substituted for manual muscle testing.

Under the task of understanding and memory is given "balance checkbook and pay bill". These are not good tests of understanding and memory. They measure calculation and concentration. Many employed well educated individuals don't balance checkbooks.

4.6 Physical Examination and Medical Testing

4-20 The MES proposes to collect medical information from different sources including 1) medical history from MES questionnaire) medical records 3) physical exams and 4) medical tests in order to evaluate the physical status for respondents as they relate both to the current SSA medical listing of impairments and any alternative disability regulations.

As mentioned before, complete medical records are difficult to obtain (sometimes costly), and can take much time and then not be complete.

The physical exam is stated to consist of a review of eight body systems. There are very few standardized and proven valid and reliable physical exam techniques. The time required for a physician to do a "complete" exam of all eight systems will be significant. Why is the extensive genitourinary physical exam included? It would seem that questions related to bowel and bladder continence and management would be adequate to address impairment listings and any alternative disability regulations without putting the respondent through an embarrassing genitourinary exam looking for genital warts and genital herpes.

How regimented will the data completion on the MES Physical examination Form be and how much is left to the physician to determine appropriate data to collect?

What will be the training/background of the physicians completing the exams?

Plan states that certain specialized noninvasive- tests will be carried out by the survey physician when necessary to help identify a condition and when required by the listings. Will these be done if the tests have already been done but results are not readily available. What endpoint of searching for results will be used?

Plan states that physicians will be asked to provide an overall evaluation of their clinical impressions related to the subject's capacity to carry out certain functions in certain environments, their level of sustained capacity, their ability to travel, and their capacity for self care. How will this be determined? Will the survey physician base this impression on the MES questionnaire? On the respondents self-report? On the physical exam findings? On functional tests done in the physicians office or on functional tests done by a surveyor? How will each be weighted? How will the physician be instructed to weigh each of these data elements in his/her decision making? Will there be instructions as to "weighting of data" or will each physician use a different style, personal style of decision making? How will reliability and validity testing be performed to prove that the assessment completed by the physician, especially regarding the subjects functional capacities, is accurate?

The plan describes how the physical examination components were selected - mainly based on the purpose of detecting the conditions that meet the SSA listing of impairments which are not currently functional/work capacity based and yet a major portion of the physical exam section is for the physician to provide an overall evaluation of the subject's functional capacity. It would seem there is a deficiency in the preparation of this part of the physical examination and physician interpretation related to estimation of functional capacity. The plan states (page 4-21) that " all items included in the physical examination were selected because they are required by the SSA listings". Again, SSA listings are not currently functionally based but instead are based on physical exam, laboratory, radiological and other medical testing documentation of medical illness.

4.6.1 Physical Examination Form

1. General information.

The general demographic information to be collected here are items which could easily be collected by a physician extender including nurse or certified medical assistant and do not require a physician to obtain. There may be some savings on use of physician's time in this area.

2. Musculoskeletal System

The narrative in the plan describing how this exam is to be performed is confusing. It states that the exam is organized according to the order in which the exam will be performed with the joint exam competed first, followed by the range of motion exam , then the strength assessment and finally the identification of soft tissue abnormalities and structural deformities.. Does this mean that each joint will be assessed, then each joint's range of motion assessed and then each joint's strength etc or will all of these tests performed on a single joint before moving on to perform the same testing on the next joint? This is important as to the time it will require. Testing based on the thought process of testing all joint's strengths and then all joint's ROM is very time intensive and requires multiple repositioning of the subject on the exam table. A more functional approach of allowing testing to be done on all accessible joints and musculoskeletal structures based on the position the subject is in (sitting, prone, supine, standing) is much more efficient and comfortable for the subject but requires more concentration by the physician.

From review of exam forms, it is clear that each area (ROM, strength) will be assessed in order of area not by joint or area of body. Following this format will be very time consuming for the physician and require many position changes for the subject.

Page 4-22

The plan lists a description of the joint exams that may be conducted as part of the musculoskeletal system assessment. Interestingly the "face and chest" are listed as joints to be examined. Neither of these are considered independent joints as commonly defined medically. The face has some important joints such as the temporal mandibular joint which can impact talking and eating and can contribute to chronic myofascial pain syndromes. There are a series of tests recommended for examination of this joint such as amount of mouth opening. The chest has multiple joints which could be examined such as the sternoclavicular joint (could be considered part of shoulder exam) the articulations of the ribs with the sternum and the ribs with the spine. The assessment of rib movement, inhalation and exhalation restrictions, and first rib elevations (musculoskeletal problems of the chest which can impact function through pain and limitation of motion contributing to pain and muscle spasm) can be done. However, very few physicians have the training and experience to assess these areas. So again the issues of physician training., physician specialty, and the usefulness of data to be obtained related to potential impact on function are very important and are not adequately addressed here. The physician expertise required to evaluate these "chest joints", identify dysfunction, and to assess the dysfunction's impact on functional capacities is at a very highly specialized level.

The shoulder is a complex "joint" composed of many joints such as the acromioclavicular joint, the glenohumeral joint, the scapular thoracic joint, to name a few.

What specifically will be examined and what criteria will be used? There are different range of motion criteria which are used across the country and can be confusing unless one approach is specified. The issue of shoulder motion and its relationship to cervical and thoracic spine dysfunction is just becoming recognized in the musculoskeletal community as very important to the functional assessment of the shoulder's capacity for motion and function. How will this be incorporated in? Currently the joint exam as described is extremely basic and limited.

The Subtalar joint is listed. How will this be evaluated and measured? It is a very difficult joint to measure and what seems important related to this joint is not specifically motion or strength or deformity but instead the subject's ability to walk, squat, and stand.

The Range of Motion exam lists the cervical spine. How will abnormalities of ROM be identified? It is normal as subjects age to lose some ROM in their cervical spine which may have little impact on function. As it increases and limits ability to turn to see while driving, for example, it becomes important but that is a significant degree change in ROM from the accepted norms. However a few degrees loss of motion in the hands and wrists (fingers are not listed on the list but are very important) can have a major impact on function. So how will the functional impact of loss of ROM and the different impact of degrees lost be assess across all the joints and reported in useful manner? Loss of degree of ROM and loss of functional ROM can be very different depending on the joint evaluated.

It is interesting that the plan differentiates Subtalar joint from feet and ankles yet does not differentiate glenohumeral joint from scapular thoracic in the shoulder. This is inconsistent. They would seem to be comparable parts of a whole and limitations of which would have comparable impact in function. Yet in this plan they are not recognized as such.

4-23 Muscle strength - How will muscle strength be determined for temporomandibular joint? That is an odd measurement requirement and what impact does this have functionally for a subject?

It is also interesting that toe muscle strength is to be assessed while toe ROM is not. Although toe strength is assessed finger strength is not. Hand and wrist strength are listed while a more functional breakdown into pinch and grip strength (for example) are not used.

These areas identified indicate inconsistency in the thought process of those putting together the examination and a significant lack of appreciation for the type of assessment which will lead to the physician being able to give an assessment of functional capacity. The combination of requiring the physician to document physical exam findings which meet the SSA impairment listings while at the same time providing a functional assessment may be very difficult for the physician to do and extremely time consuming and thought intensive - asking the physician to look at the subject from very different view points during the exam.

Amputation and other Structural Deformities

This section lists fractures, soft tissue injuries, amputations, chronic tissue infection or osteomyelitis (these are not synonymous - one is infection of muscle, fascia, or skin and the other is infection of bone), congenital atresia, and other abnormality or injury. The combination of these in one group is interesting and definitely based on an obsolete medical model as opposed to a functional and disability assessment model.

There should be a breakdown with fractures into some type subheadings so that every fracture that a subject has had will not be included. The only fractures that would be important are those currently impacting joint ROM, gait; those not healed or unstable; or those that indicate an ongoing disease process such as osteoporosis where additional fractures would be expected (risk for).

Soft tissue injuries are a major problem in the way they can chronically impact function, be difficult to assess without adequate training and experience, and their inconsistency. Disorders which are characterized by inconsistency and unpredictability have a major negative impact on function and ability to work. Things that may impact soft tissue injuries are posture and ergonomics , fatigue, repetitive use, the subject's ability to self treat or obtain treatment to lessen the impact, stress in workplace; adequate treatment; etc . Data are not collected on these areas on this medical assessment.

How soft tissue injuries are categorized, measured and identified is controversial. Chronic pain associated with soft tissue injuries adds social, emotional, and psychological components. Whether the soft tissue injury is associated with other syndromes often seen with soft tissue injuries is important such as nonrestorative sleep, myofascial syndrome, and fibromyalgia.

Chronic tissue infection and osteomyelitis would seem to be in a completely different category - this is an ongoing process which needs intervention - antimicrobial, surgical.

The assessment of congenital atresia and amputation would center around the subject's adaptation and the appropriate use of prosthetics and orthotics. Most physicians do not have the training to assess the maximization of these aids in people with congenital atresia or amputation. Only Physical Medicine and Rehabilitation and some orthopedic specialists would have the training and experience to evaluate these disorders appropriately and especially from a maximization of function standpoint.

Page 4-24

Head and neck exam - For what purpose is the skull examination? To look for old fracture indicating possible traumatic brain injury, evidence of intraventricular shunt, abnormal size and shape to indicate cognitive dysfunction? Specific expectations/evaluations should be included.

Audiometry and hearing acuity, typically are not done by the physician but by an audiologist for specific testing. How does evaluation of the cervical lymph nodes assess function? The exam of the pharynx, larynx and thyroid gland and cervical lymph nodes is just a general physical exam designed to identify ongoing disease processes - not to assess function.

The respiratory and cardiovascular exams (4 and 5) must be related to the impairment listing because many of the items are not functionally related at all. For example, clubbed fingers gives an indication of respiratory problem but says nothing about functional capacities. Same for vascular system and gastrointestinal system.

7. Genito-urinary System

Page 4-30

Skin and Lymph Nodes - This would seem to be a general physical exam for the purpose of detecting illness and disease processes. What does the breast nipple/areola area exam have to do with impairment listing or functional capacities. Abnormalities here can indicate breast cancer or endocrine dysfunction. Is the purpose of this exam to identify disease processes not previously identified? Is it to take the place of the primary care physician's exam? Will the exam be modified if the subject has a primary care physician they have seen in the last year?

Neurologic - This complete neurologic exam may help to identify criteria meeting impairment listings but is not function related. The plan says an EEG should be included if indicated by exam findings. Unless one is looking to diagnose an undiagnosed seizure disorder or ascertain medication control of seizures an EEG is not very helpful in determining function. Cognitive testing would be more useful.

The muscle exam is repeated/duplicated. The first toe and second fifth toe is added. It is unclear how this contributes to functional assessment or determination of people likely to be on the disability roles.

12. Report of Findings-Overall Evaluation

Page

4-32 The plan states this section asks the physician to use clinical judgment based on exam findings to provide evaluations of functional status in several areas and is intended to be summary of the physician's opinion on the examinee's abilities in these areas. As currently structured, the physical examination will allow the physician to make an accurate assessment of the subject's medical, health, and physical conditions.

However, physicians using this examination form will not be able to accurately evaluate 1) the subject's maximum physical capacity along specified dimensions; 2) the subject's ability to function in specified environments; 3) the subject's sustained capacity to do work; 4) the subject's physical capacity for work 5) the subject's ability to travel independently; and 6) the subject's capacity for selfcare.

The plan states that this will represent only the physician's opinion since these categories of function cannot be objectively measured and will be used only to provide a general evaluation of the subject at the time of the exam. Some of the areas mentioned can be objectively evaluated such as lifting , carrying or pulling strength and can be predicted by a physician with a carefully constructed history and examination format (i.e., PAID project).

In this case (DES) the physician will be very dependent on the subject's self-assessment of function.

Assessment/questionnaire Review

Medical Evaluation Screener (Appendix 3)

In general the wording of the screener should be more in laymen's terms and interpretable at a 4th grade education level. For example, instead of requesting information repetitively on various "conditions" the question could be asked "What other problems keep you from working?" and "What kind of work are your prevented from doing and why". This would seem to get at the data desired more directly. Having knowledge of "conditions" which are diagnoses without the 'how and why' of why they are not working or are limited in performing ADLs limits the usefulness of the data collected. The focus of the screener seems to be slanted to identifying conditions more than identifying functional limitations.

B entire: Listing name of physicians, interpreting the doctors diagnosis, dates of visits and hospitalization, dates of symptoms and names of medications will be full of non-responses or inaccurate information.

B entire: Lots of open-ended questions -- How will this data be computerized?

B entire: This will take several hours minimum. I suggest piloting this history on one relatively healthy intelligent person to get a time estimate ... obviously much longer if disabilities.

Item

B2 This item dealing with how much income the subject receives from a job might be more effective at the end of the screener. Subjects may fear answering correctly related to fear of removal of benefits etc and this may impact the subject's perception and responses to the rest of the screener.

B3 The term "recently" needs to be defined/given parameters.

B18 The term "decrease" needs definition/parameters.

B20 Fatigue and dizziness are ubiquitous, non-specific terms.

B32 "Housework" needs to be defined and broken into types.

B53 This question will be difficult to get reliable and accurate data on because people have different family and self expectations of housework. So the question of have you recently had a change in the kind of housework done needs to be more specific or parameters given.

B84 Headaches are ubiquitous.

Section B Medical History

In general the medical history section is basically a patient/subject self-report being "dictated" to the physician. There are no instructions nor anything else to indicate the physician is to assess the data from the subject in any other fashion that the exact wording of the questions as per the medical history. This is basically a repeat of subject self-report. We found in the PAID study that it was useful for the physician to obtain functionally related information in an indirect fashion without repeating questions which a subject can answer as self-report. For example, instead of asking the subject if they can lift 10 pounds over their head they can be asked if they put up groceries, can they lift a gallon jug of water over their head etc. This type of questioning along with the physical exam allows the physician to give an independent assessment that is less influenced by direct subject self-report. Item B12 There is not enough room to completely list medications. However this may be because the history form is divided by "condition" and the medication question is asked for each condition. Many respondents may not be able to categorize their medications by condition.

B27 Regarding fatigue, the question relates to whether the respondent experienced stressful changes at home or work at about the time they first noticed the fatigue. This is a very poorly worded question and is likely to give inconsistent or inaccurate data. When an individual is experiencing fatigue it is more difficult to meet the routine demands of home and work, therefore the individual may respond "yes" they experienced stress which could lead the data evaluators or physician to believe that stress was the cause of the fatigue as opposed to fatigue causing the stress experienced.

B29 On this item and multiple other items where the subject is questioned as to what the doctor says the condition in question was or rather what diagnosis the physician has attached to the condition being described. There should be a question as to what the subject felt the condition was or was related to. The subject's perception of the condition may be very different from the physician who gave them a diagnosis. Subject's perceptions of illnesses and physical symptoms have been shown to impact the way in which the subject reacts to, and performs with, the condition.

B223 These questions deal with various sensory and communication related impairments. The forms allow for identification if subject has had symptom, when it first began and whether they still have it etc. However it does not allow for data collection on functional impact on the subject or whether the subject is compensating for the symptom. In other sections of the data collection, which appear to be collected by individuals other than the examining physician there may be an addressing of this issue. However, since the functional impact and issue of compensation or substitution of other actions is not addressed in the physician section this will negatively impact the physician's ability to give an accurate functional assessment.

B251,2 One example of many - same questions are asked for each condition) these items are examples of questions related to side effects from medication which the subject may be experiencing. They ask if there are any side effects and what they are. However there is not place for assessment of the impact these side effects may be having on the subject if any. For example, many drugs cause anticholinergic side effects which can range from dry mouth and dry eyes to severe constipation, urinary retention and even respiratory complications in persons with spinal cord injury. Even the most basic symptom of dry mouth can range from just being noticeable to the subject to being very limiting and impacting comfort, speech, and thus work productivity. Urinary retention can be to the extent that Foley catheterization is required to empty the bladder or additional medication is required to compensate for this side effect.

So this is another area where the degree of impact and whether there is an impact on the subject's function is not addressed in the physician's' assessment. This hampers the physician's ability to make functional assessments accurately but also misses an opportunity for the subject to perhaps have a problem which is significantly impacting their lives addressed and resolved.

Section C Functional Assessment

Section C: Many don't know responses ... not forcing definitive response ... same as no response. Stooping, crouching, and kneeling; many may not know definitions. Difference between 10, 20, 50, 100 pounds ... many may not know.

Item C-1 In general the functional assessment as discussed earlier under the medical history section is merely a dictation of subject self-report.

C1 For this item and many others: the "Don't Know" option is a bad option. There may be a multitude of reasons for the subject to respond "Don't know" and without explanation or more specific questioning this is just lost data.

C1 Options "c" through "e" - These questions contain descriptors which are not defined (that this reviewer could determine). The descriptors are "regularly required", "Occasionally required", "less Often than you used to". These are very subjective terms and the subject as well as the examiner must have some guidelines or examples from which to answer within. If these descriptor parameters are not provided for the examiner and the subject this is a significant problem/limitation to the accuracy of this questionnaire.

C4 Options "c" through "e" - These items refer to job requirements related to frequency of stooping, crouching, and kneeling. Again the descriptors of "frequently, occasionally, and less often than you used to" do not have parameters attached. Also, there is no opportunity to identify that the subject has made modifications in their actions to allow them to complete these activities or modifications to complete their job without doing these activities - information which is important in making a functional capacity prediction as the physician is being asked to do.

C5 For this item through C7 - These items deal with lifting of weights varying from 10 to 100 pounds. These questions neglect a major functional determination area which is - from what height to what height is the weight moved? Is it lifted from waist height to waist height or from the ground to above the head? These have very different postural, ergonomic, and physical demands and consequences. So this is a major oversight and limitation in this functional assessment questionnaire. The issue of whether turning or twisting is required or limited is also not addressed. This is considered to be a major error in this functional assessment.

C8 This question combines the activities of pushing, pulling, and the operation of hand or foot controls. These activities should not be combined. They require very different physical requirements. An individual may be able to push and pull with hands but not have the dexterity to operate hand controls. This is considered to be a major error in the functional assessment.

C10 This item deals with whether the subject has difficulty walking for a specific distance. It does not allow for identification of what condition creates the difficulty which will significantly impact the physician's interpretation of functional capacity. The subject may have a medical history of cardiovascular disease as well as spinal stenosis. Both of these conditions could limit the distance walked but they have very different implications. A subject who is unable to walk 2 blocks due to cardiac symptoms is significantly disabled from a variety of activities and should not be pushed to walk over that distance. A subject with spinal stenosis may well be able to walk much farther than 2 blocks if there is the opportunity to sit or flex the spine for a few minutes and the spinal stenosis would not affect use of the upper extremities unless the upper extremity activity placed the subject's spine in an extended position for a prolonged period. A subject with sever enough cardiac symptoms to prevent them from walking 2 blocks would probably be restricted from any upper extremity lifting or overhead activity due to impact on the cardiac system. This is just one example of the separation of good functional questions/history from medical history which makes this questionnaire of questionable utilization for accurate prediction of functional capacity by the physician.

C11 This item and item C12 are examples of questions using descriptors without parameters. The descriptors used are "some difficulty", "a lot of difficulty". Item C12 is example of descriptor without parameter - do you climb a ladder "differently" than you used to.

C15e The term "normal breaks" needs to be defined. What are the parameters? Lack of parameters for descriptors is a major limitation for this assessment.

C20 This is an example of excessive complexity in the functional assessment questions as compared to other questions such as the lifting questions. These questions go into great detail to document exactly what the limitation is related to eating - painful, tiring, amount of help/accommodation needed. So there are inconsistencies in this assessment in different functional areas which may indicate the strengths and weaknesses of the various contributors to the assessment.

3c "Is getting outside by yourself very tiring" ? What is the definition of "getting outside"? Does this refer to work, to school or to home. Does it refer to getting to transportation or getting into a natural environment to enjoy nature?

C24 "Extreme heat" needs to be defined.

C26 Regarding extreme heat and extreme humidity, the questions do not deal with changes in these environmental extremes vs. constancy of the exposure. Constancy vs. frequently changing environments can be very important depending on the illness. In some disorders of temperature control the subject can manage well in changing environments but cannot remain in an extreme environment such as extreme heat for extended periods of time. With some disorders the subject can remain in extreme heat as long as hydration is maintained. With disorders such as multiple sclerosis, extreme heat will worsen/exacerbate neurologic deficits and must be avoided. In spinal cord injuries the level of injury determines whether the individual can tolerate prolonged temperatures in the extremes.

C34 No definition of choices vague.

C35 No definition of choices vague.

C37 "Heavy housework" needs to be defined or examples given.

D1 Definition of brace may not be known.

D12 Question: Difference between some of the time and only occasionally??

D24 Question: Difference between occasionally and seldom??

D34 Question: Difference between sometimes and rarely??

D41 Question: Difference between somewhat and a little??

D59 This section is very good and will assist the physician in determining subject's attitude toward work, perception of illness and impact on work, and motivation.

Physical Assessment

The scripts which are provided for the examiner are good and will help to ensure consistency. However there are no scripts or parameters provided regarding subject's lack of effort, dealing with subject's fatigue; how much encouragement or cheerleading should be given; how should pain or illness behaviors be responded to. All of these things will negatively impact reliability and validity of physical assessment.

There is no script or information provided regarding what the subject will be told is the reason for the assessment, how this data will or will not be used, nor information provided regarding confidentiality of the data obtained.

All tasks should have specific multiple choice responses for reason a task is not attempted or not completed .

Questions: How will the physical assessment information be presented to the physician to review? Will the physician see the data prior to the medical assessment or after? Will the data be in a visually easy to interpret form such as various graphics or will it be presented as raw data in the manner in which it was collected? How much time will the physician spend reviewing this material before the subject is examined? Will time be allowed for this or merely for a quick scan while the subject waits to be evaluated? All of these issues impact the information available for the physician to base assessment upon and will impact the physician's assimilation of the material.

There are no overall cardiovascular/respiratory or other disease related symptom endpoints used. There are no task specific endpoints identified which will negatively impact the reliability and validity of the physical assessment testing. The lack of general and task specific endpoints is considered a major deficiency in the physical performance assessment from an accuracy standpoint and from a medical legal liability standpoint for SSA.

4.4: Appendix 5

In general, these tests seem more applicable to a geriatric population. For many young patients with pain or mental status impairment I would guess these tests have a floor effect. Certainly not appropriate capacity tests for medium or heavy work classifications. I am not sure if all of these individual tests have been validated ... not that familiar with the literature. Medical Evaluation Study Physical Exam Form (Appendix 7)

Other choices may be a problem. Questions: What to fill in if more than one response is present as the case will often be? Many medical terms may not be known to many physicians, i.e. Fremisus? Very tedious and lengthy, several hours minimum, most sections will not be pertinent to a given subject with regards to disability issues.