BACKGROUND INFORMATION ON THE 5-YEAR CLIENT- AND CHILD OUTCOMES SURVEY SAMPLES


Data on participation, degree receipt, job quality, income, transitional
benefits, health care coverage, child care, child outcomes, and several other
measures used in the NEWWS Evaluation Final Report come from the 5-Year
Client Survey and 5-Year Child Outcomes Study (COS) Survey.  These surveys were
administered to a subsample of the Full Impact Samples in Atlanta, Grand Rapids,
Portland, and Riverside approximately five years after random assignment.  This
memo documents the sample selection process, weighting strategy, and response
rates for these surveys.


I. Key analysis samples

A. The survey eligible sample ("eligibles"). Sample members in the full research
sample who were randomly assigned during months in which the survey sample was
selected and who met the criteria for inclusion.

   In all sites, the survey eligible sample includes members of the full
   research sample who were randomly assigned during some, but not all, months
   of sample intake. (See SMP5TBL1.TXT)  Limiting the eligible sample in this
   way can introduce "cohort effects,"  impact estimates that are especially
   large or small for sample members randomly assigned during particular months.
   A cohort effect may occur because members of the survey eligible sample
   differ in measured or unmeasured background characteristics from persons
   randomly assigned in other months. Changes in area labor markets or in
   program implementation that occur at some point after the start-up of random
   assignment may also introduce cohort effects - e.g., by increasing or
   decreasing a program's relative success in moving welfare recipients from
   welfare to work. These issues are most germane to Portland, where selection
   of the survey eligible samples took place over fewer months than in Atlanta,
   Grand Rapids, and Riverside.


B. The fielded sample ("fieldeds"). Members of the eligible sample who were
chosen to be interviewed. The sample includes:

  1) People who were chosen to answer only the sections of the survey asked of
  all respondents (The "core" survey).  Most of these sample members had no
  children aged 3 to 5 at the time of random assignment.


  2) People who were chosen to answer both the core survey and the Child
  Outcomes Study (COS) Survey. These sample members had a child aged 3 to 5 at
  the time of random assignment, who was designated as the "focal child" for
  the study. The focal child was the subject of nearly all questions in the COS
  Survey given to his/her mother. The child also completed a Self-Administered
  Questionnaire and took the Woodcock-Johnson Test of Achievement.

C. The respondent sample ("respondents"). Members of the eligible sample, chosen
to be interviewed  (i.e., fieldeds),  who were interviewed.

D. The non-respondent sample ("non-respondents"). Members of the eligible
sample, chosen to be interviewed  (i.e., fieldeds), who were not interviewed.
They could not be located or declined to be interviewed.

The difference between a respondent and non-respondent is straightforward for
the Five-Year Client Survey, but not for the COS. Sample members fielded only
for the Client Survey could be interviewed by phone or in person.  However,
sample members fielded for the COS were supposed to answer the Client Survey
and additional COS Survey questions during an in-person session that also
included observations of interactions between the COS mother and focal child,
an interview with the focal child, and administration of a standard assessment
of the focal child's intellectual development. As shown in the table below, 262
mothers in the COS fielded sample had moved too far away to be visited by
interviewers  or could not conduct a COS in-person interview for other
reasons. They did, however, answer the Client Survey by phone, including the
questions on child care and child outcomes asked of all respondents to the
Client Survey. These sample members are counted as respondents to the Client
Survey, but not as respondents to the COS. Thus, the COS respondent sample is
limited to mothers who participated in the in-person interviews, observations,
and assessments.



       Number of Sample Members Fielded for the 5-Year COS Survey
          Who Only Answered the Five-Year Client Survey


 Quarter of Random Assignment  Total Atlanta  Grand Rapids  Riverside
 ----------------------------  ----- -------  -----------   ---------

 JUL-SEP 91                      5     .            0           5

 OCT-DEC 91                     26     .            0          26

 JAN-MAR 92                     24     3            0          21

 APR-JUN 92                     31     5            4          22

 JUL-SEP 92                     24     8            7           9

 OCT-DEC 92                     37     8            5          24

 JAN-MAR 93                     47    16            7          24

 APR-JUN 93                     48    25            5          18

 JUL-SEP 93                      9     0            9           .

 OCT-DEC 93                      6     0            6           .

 JAN-MAR 94                      5     0            5           .


 ALL                           262    65           48         149



NOTE:  These 262 sample members can be identified using data from the 5-Year
Full Sample Impacts Public Use File (CD #1).  They have the values


COS5FELD =1 AND   /* FIELDED FOR THE 5-YEAR COS */

SRV5RESP =1 AND   /* RESPONDED TO THE 5-YEAR CLIENT SURVEY */

COS5RESP =0       /* DID NOT RESPOND TO THE 5-YEAR COS SURVEY */



II.  Survey Selection and Sampling Ratios

A. Objectives

MDRC had the following goals in mind when choosing the fielded samples
for Atlanta, Grand Rapids, and Riverside:

* Approximately 4,500 respondents among the 3 sites

* Sufficient sample sizes for the Five Year Child Outcome Study (COS) to
  detect program effects of small-to-moderate size.  This meant that more sample
  members were chosen for the COS than for the core.

* As equitable a distribution among sites as possible.

* A 75% completion rate overall.

* Maximize the number of respondents to both the 2-Year and 5-Year Client- and
  COS Surveys

* For Atlanta: To include as many respondents as possible from a special
  "Baseline" survey conducted a few months after random assignment.  (See,
  How Well Are They Faring?  AFDC Families with Preschool-Aged Children in
  Atlanta at the Outset of the JOBS Evaluation.  Prepared by Kristin A. Moore,
  Martha J. Zaslow, Mary Jo Coiro, and Suzanne M. Miller, Child Trends, Inc.,
  and Ellen B. Magenheim, Swarthmore College.  1995.  Washington, D.C.:  U.S.
  Department of Health and Human Services, Office of the Assistant Secretary
  for Planning and Evaluation; and U.S. Department of Education).

* For Riverside and Atlanta:  As much as possible, to replicate the distribution
  of members of the eligible sample on key background characteristics (research
  group, age of youngest child, pre-random assignment educational attainment,
  and date of random assignment)


* For Portland, which did not conduct the COS, MDRC sought to obtain similar
  sample sizes for program and control group members as for the 2-Year Client
  Survey and to maximize the number of respondents to both surveys.


B. Sampling Methods

The fielded samples for 5-Year Client- and COS Surveys were selected from
among the sample members originally fielded for the 2-Year Client Survey and
COS.  For these surveys, the eligible samples in Atlanta, Grand Rapids, and
Riverside had been divided into strata according to sample members' research
group, date of random assignment, age of youngest child, and pre-random
assignment educational attainment.  In Portland, the strata were defined by
sample members' research group and date of random assignment.

A small number of members of the fielded samples from the 2-Year Client- and
COS Surveys were made ineligible for selection to the 5-Year surveys:

* Those who were chosen to be surveyed were subsequently discovered to have
  background characteristics (such as non-proficiency in English or Spanish)
  that made them ineligible.

* In addition, it was decided to drop single fathers from the 5-Year Client
  Survey eligible sample, although some were interviewed at two years.


For research purposes, different sampling ratios, ranging from 16 percent to 100
percent, were used  when selecting members of the fielded sample from within
each stratum. (The sampling ratio is the percentage of eligible sample members
selected.) Sample members were chosen at random within each stratum.  Although
corrected for, as discussed below, differences in sampling ratios may affect
survey impact estimates.  For instance, unless the total sample size is large,
different  sampling ratios increase the likelihood that persons chosen in one
research group differ (perhaps in unmeasured characteristics) from persons
chosen in another research group.


In Portland and Grand Rapids, all sample members fielded at two years (and
not dropped from the sample for reasons listed above) were again fielded at five
years.  In Atlanta and Riverside, funding limitations made it necessary to
select a subsample from the original fielded sample for the 2-Year Client-
and COS Surveys.




III.  Weighting

To estimate impacts, weights were applied to the survey respondent sample to
correct for differences in sampling ratios between the strata in Atlanta,
Grand Rapids, Portland, and Riverside.  In the unweighted fielded survey
sample in these sites, strata (i.e., sample members who share background
characteristics and have the same sampling ratio) with high sampling
ratios are over-represented and strata with low sampling ratios are
under-represented.

Within each site, weights for each stratum were set to equal:


   (N of stratum in eligible sample) / (Total N in eligible sample)
   ---------------------------------------------------------------
   (N of stratum in fielded sample)  / (Total N in fielded sample)


OR

   Proportion of eligible sample in stratum
   ----------------------------------------
   Proportion of fielded sample in stratum


The same weights are used for the respondent sample. The variable FLD5WGT stores
the weight value for each respondent.

IV.  Response Rates

As noted above, sample members who were fielded and interviewed are
survey respondents.  Those chosen to be surveyed but who were not
interviewed are non-respondents. The table below shows the response rate,
the percentage of the fielded sample who responded to the survey, by
program and research group.



 Number of Fielded Survey Sample Members and Client Survey Response Rates


                                                             Five-Year
                                         Five-Year           Child Outcomes
                                         Client Survey       Study Survey

                                       Fielded  Response    Fielded  Response
Site and Program                       Members  Rate (%)    Members  Rate (%)



Atlanta Labor Force Attachment             597    86.9         349    82.8
Atlanta Human Captial Development          717    82.8         473    77.6
Atlanta Control                            649    85.1         389    79.9

Grand Rapids Labor Force Attachment        605    88.4         252    84.5
Grand Rapids Human Capital Development     619    88.4         244    80.3
Grand Rapids Control                       606    92.7         249    85.9

Riverside Labor Force Attachment           680    73.4         294    62.9
Riverside Human Capital Development        509    73.9         309    67.3
Riverside Control                          933    77.2         536    64.9

Portland Program                           364    77.2         n/a    n/a
Portland Control                           289    77.2         n/a    n/a


For the Five-Year Client Survey, response rates exceed 70 percent for all
programs and research groups (and 80 percent in Atlanta and Grand Rapids) and
are high enough to suggest that the survey probably represents the eligible
sample.  These results inspire confidence in the impacts for respondents.

Responses rate also exceed the 70 percent threshold for the COS sample fielded
in Atlanta and Grand Rapids. In Riverside, however, responses rates for the COS
fell below this standard, ranging from about 63 percent (LFA) to 67 percent
(HCD). Therefore, greater caution is required when interpreting results for
the COS sample in Riverside. However, in Riverside, response rates would exceed
75 percent for all research groups if the COS mothers who only answered the
Client Survey were counted as respondents.
