
Background Information on Selection of the 2-Year Client Survey Sample


Data on participation, degree receipt, job quality, income,
transitional benefits, health care coverage, child care, child outcomes,
and several other measures used in this report come from the Two-Year
Client Survey.  The survey was administered to a subsample of the full
research sample approximately two years after random assignment.

Key analysis samples

The survey eligible sample ("eligibles"). Sample members in the full
research sample who were randomly assigned during months in which the
survey sample was selected and who met the criteria for inclusion.

The fielded sample ("fieldeds").  Members of the eligible sample who
were chosen to be interviewed.

The respondent sample ("respondents"). Members of the eligible sample,
chosen to be interviewed  (i.e., fieldeds),  who were interviewed.

The non-respondent sample ("non-respondents"). Members of the eligible
sample, chosen to be interviewed  (i.e., fieldeds), who were not
interviewed. They could not be located or declined to be interviewed.



I.  Survey Selection and Sampling Ratios

Several of the chapters in this report analyze program impacts
calculated from survey responses as well as impacts calculated from
administrative records for the full sample. It is important to
understand the process by which the survey samples were chosen and
survey responses collected in order to assess the comparability of
these results.

Selecting the Eligible Sample

In all sites, the survey eligible sample includes members of the full
research sample who were randomly assigned during some, but not all,
months of sample intake. (See SAMTBL1.TXT)  Limiting the eligible
sample in this way can introduce "cohort effects,"  impact estimates
that are especially large or small for sample members randomly assigned
during particular months.  A cohort effect may occur because members of
the survey eligible sample differ in measured or unmeasured background
characteristics from persons randomly assigned in other months.
Changes in area labor markets or in program implementation that occur
at some point after the start-up of random assignment may also introduce
cohort effects - e.g., by increasing or decreasing a program's relative
success in moving welfare recipients from welfare to work.  These issues
are most germane to Columbus, Detroit, Portland, and Oklahoma City,
where selection of the survey eligible samples took place over fewer
months than in Atlanta, Grand Rapids, and Riverside.

Further, the research strategy for choosing the survey eligible samples
in Atlanta, Grand Rapids, and Riverside required exclusion of sample
members with certain background characteristics:  teen parents; parents
with children less than 3 years old; men with children ages 3 to 5;
people who did not speak either English or Spanish; and people who did
not provide information on their educational status and children's ages
prior to random assignment.  This selection strategy may affect the
genearalizability of impact results recorded from the survey.

Fortunately, cohort effects were small.  For instance, differences in
2-year earnings gains between the full research samples and the survey
eligible samples varied by less than $100 in 9 of the 11 programs and
by less than $200 in every program.  (Results not shown)


Selecting the Fielded Sample

The percentage of the survey eligibles who were chosen for the fielded
sample is the sampling ratio. Across all sites, sampling ratios ranged
from 15 percent to 100 percent.

In four sites, Atlanta, Grand Rapids, Portland, and  Riverside, the
fielded sample was selected by drawing a stratified random subsample
of the survey eligible sample. In Atlanta, Grand Rapids, and Riverside,
the sampling ratio varied (for research purposes) by research group,
date of random assignment, age of youngest child, and pre-random
assignment educational attainment of the sample member.   In Portland,
sampling ratios varied by research groups  and by date of random
assignment only. Although corrected for, as discussed below, differences
in sampling ratios may also affect survey impact estimates.  For instance,
unless the total sample size is large, different sampling ratios increase
the likelihood that persons chosen in one research group differ (perhaps
in unmeasured characteristics) from persons chosen in another research
group.

In two other sites, Detroit and Oklahoma City, the fielded sample for
program and control group members was selected by drawing a simple random
sample from the eligible sample.  That is, within these sites, a single
sampling ratio was applied to all program and control group members,
irrespective of their background characteristics.  This sampling strategy
was used in Columbus as well, except that the sampling ratio for control
group members was slightly higher than for members of the Integrated and
Traditional groups.


II.  Weighting

To estimate impacts, weights were applied to the survey respondent sample to
correct for differences in sampling ratios between the strata in Atlanta,
Grand Rapids, Portland, and Riverside.  In the unweighted fielded survey
sample in these sites, strata (i.e., sample members who share background
characteristics and have the same sampling ratio) with high sampling
ratios are over-represented and strata with low sampling ratios are
under-represented.  To make the fielded sample more closely replicate
the background characteristics of survey eligibles, weights for each
stratum were set to equal the inverse of the sampling ratio for that
stratum.   For example, a stratum in which 1 in 4 eligible persons were
chosen would receive a weight of 4 (or 4/1) , whereas a stratum in which
every eligible person was chosen would receive a weight of 1 (or 1/1).
THE SAME WEIGHTS ARE USED FOR THE RESPONDENT SAMPLE. Weighting was not
required for sample members in Columbus, Detroit, and Oklahoma City,
because sample members' background characteristics did not affect their
chances of selection.


The variable FIELDWGT stores the weight value for each respondent.
The sampling ratios and weights are as follows:



ABBREVIATIONS


3T5:  Youngest child aged 3 to 5

GE6:  Youngest child aged 1,2 or GE 6

IN NEED: No high school diploma or GED (Atlanta, Grand Rapids)
         In need of basic education (Riverside)

NO NEED: High school diploma or GED (Atlanta, Grand Rapids)
         Not in need of basic education (Riverside)

C1:      Randomly assigned before 7/31/92

C2:      Randomly assigned 8/1/92-11/30/92

C3:      Randomly assigned 12/1/92-1/31/93

C4:      Randomly assigned 2/1/93-6/30/93

C5:      Randomly assigned on or after 7/1/93


C:       Control group member

B:       HCD

J:       LFA

P:       Program group member

                                        PROPORTION
                                        OF STRATUM
                  STRATUM               FIELDED     FIELDWGT

ATLANTA          3T5 IN NEED C1 C       0.69444      1.44
ATLANTA          3T5 IN NEED C1 J       0.84615      1.18
ATLANTA          3T5 IN NEED C1-5 B     0.99537      1.00
ATLANTA          3T5 IN NEED C2-5 C     0.98947      1.01
ATLANTA          3T5 IN NEED C2-5 J     0.88889      1.13
ATLANTA          3T5 NO NEED C1 C       0.48077      2.08
ATLANTA          3T5 NO NEED C1-2 J     0.92800      1.08
ATLANTA          3T5 NO NEED C1-4 B     0.99361      1.01
ATLANTA          3T5 NO NEED C2-5 C     0.99174      1.01
ATLANTA          3T5 NO NEED C3-4 J     0.62963      1.59
ATLANTA          3T5 NO NEED C5 B       0.58120      1.72
ATLANTA          3T5 NO NEED C5 J       0.25581      3.91
ATLANTA          GE6 IN NEED C1 J       0.49020      2.04
ATLANTA          GE6 IN NEED C1-5 B     1.00000      1.00
ATLANTA          GE6 IN NEED C1-5 C     0.99200      1.01
ATLANTA          GE6 IN NEED C2-5 J     0.66409      1.51
ATLANTA          GE6 NO NEED C1 B       0.73077      1.37
ATLANTA          GE6 NO NEED C1-4 C     0.44316      2.26
ATLANTA          GE6 NO NEED C1-4 J     0.48157      2.08
ATLANTA          GE6 NO NEED C2-5 B     0.31448      3.18
ATLANTA          GE6 NO NEED C5 C       0.28477      3.51
ATLANTA          GE6 NO NEED C5 J       0.21678      4.61


COLUMBUS(a)      All Eligibles          0.34891      1.00

DETROIT (a)      All Eligibles          0.23121      1.00

GRAND RAPIDS     3T5 IN NEED C1-5 B     0.93939      1.06
GRAND RAPIDS     3T5 IN NEED C1-5 C     0.93805      1.07
GRAND RAPIDS     3T5 IN NEED C1-5 J     0.89524      1.12
GRAND RAPIDS     3T5 NO NEED C1 J       0.76744      1.30
GRAND RAPIDS     3T5 NO NEED C1-5 B     0.77723      1.29
GRAND RAPIDS     3T5 NO NEED C1-5 C     0.92857      1.08
GRAND RAPIDS     3T5 NO NEED C2-5 J     0.87162      1.15
GRAND RAPIDS     GE6 IN NEED C1 B       1.00000      1.00
GRAND RAPIDS     GE6 IN NEED C1 C       1.00000      1.00
GRAND RAPIDS     GE6 IN NEED C1 J       1.00000      1.00
GRAND RAPIDS     GE6 IN NEED C2-5 B     0.93506      1.07
GRAND RAPIDS     GE6 IN NEED C2-5 C     0.95946      1.04
GRAND RAPIDS     GE6 IN NEED C2-5 J     0.72368      1.38
GRAND RAPIDS     GE6 NO NEED C1 B       1.00000      1.00
GRAND RAPIDS     GE6 NO NEED C1 C       1.00000      1.00
GRAND RAPIDS     GE6 NO NEED C1 J       1.00000      1.00
GRAND RAPIDS     GE6 NO NEED C2-5 B     0.61458      1.63
GRAND RAPIDS     GE6 NO NEED C2-5 C     0.62847      1.59
GRAND RAPIDS     GE6 NO NEED C2-5 J     0.65411      1.53

OKLAHOMA CITY(a) All Eligibles          0.15437      1.00

PORTLAND         RA < 9-1-93 C          0.16067      6.22
PORTLAND         RA < 9-1-93 P          0.16430      6.09
PORTLAND         RA >=9-1-93 C          0.49744      2.01
PORTLAND         RA >=9-1-93 P          0.17167      5.83

RIVERSIDE        3T5 IN NEED C1-3 B     0.51235      1.95
RIVERSIDE        3T5 IN NEED C1-3 C     0.61975      1.61
RIVERSIDE        3T5 IN NEED C1-4 J     0.23762      4.21
RIVERSIDE        3T5 IN NEED C4 B       0.90566      1.10
RIVERSIDE        3T5 IN NEED C4 C       0.96094      1.04
RIVERSIDE        3T5 NO NEED C1-3 C     0.23604      4.24
RIVERSIDE        3T5 NO NEED C1-4 J     0.23511      4.25
RIVERSIDE        3T5 NO NEED C4 C       0.78261      1.28
RIVERSIDE        GE6 IN NEED C1-4 B     0.81507      1.23
RIVERSIDE        GE6 IN NEED C1-4 C     0.92419      1.08
RIVERSIDE        GE6 IN NEED C1-4 J     0.37842      2.64
RIVERSIDE        GE6 NO NEED C1-4 C     0.36047      2.77
RIVERSIDE        GE6 NO NEED C1-4 J     0.32471      3.08


(a) The weight equals 1.00, because the same sampling ratio was applied
to all eligibles.

It should be noted that under some conditions impacts for a weighted
respondent sample may still be different from those for the eligible
sample.  For example, this result could occur if very different
proportions of program and control group fieldeds answered the survey,
or if members of a subgroup within one research group were more likely
to be interviewed than their counterparts in a different research group.


III.  Response Rates

As noted above, sample members who were fielded and interviewed are
survey respondents.  Those chosen to be surveyed but who were not
interviewed are non-respondents. The table below shows the response rate,
the percentage of the fielded sample who responded to the survey, by
program and research group.  As shown, in most programs, response rates
are high enough to suggest that the survey probably represents the eligible
sample.

The goal of the survey effort was to obtain responses from at least 70
percent of  the fielded sample. The 70 percent goal was achieved for all
research groups in all sites; in fact, response rates reached 80 percent
or above for most research groups.  These results inspire particular
confidence in the impacts for respondents.



           National Evaluation of Welfare-to-Work Strategies

            Number of Fielded Survey Sample Members and
                  Two-Year Client Survey Response Rates

                                              Number of
                                              Fielded     Response
Site and Program                              Members     Rate (%)

Atlanta Labor Force Attachment                 908         88.5
Atlanta Human Captial Development             1225         90.9
Atlanta Control                               1200         90.5

Grand Rapids Labor Force Attachment            637         90.1
Grand Rapids Human Capital Development         647         88.7
Grand Rapids Control                           631         92.6

Riverside Labor Force Attachment               740         76.2
Riverside Human Capital Development            819         75.8
Riverside Control                             1396         79.8

Columbus Integrated                            455         81.5
Columbus Traditional                           459         79.7
Columbus Control                               460         77.6

Detroit Program                                261         80.5
Detroit Control                                259         83.4

Oklahoma Program                               356         72.8
Oklahoma Control                               360         70.0

Portland Program                               385         77.1
Portland Control                               377         83.0





