       ESTIMATING PROGRAM IMPACTS WITH 5-YEAR CLIENT SURVEY DATA


As with administrative records data, MDRC used OLS regression (PROC REG in SAS)
to estimate program effects on survey outcomes.  Except for FEMALE (or XFEMALE),
the same covariates were used with each data source to control for differences
among research groups in background characteristics.  FEMALE (or XFEMALE) is
excluded from the regression model used to estimate impacts on outcomes recorded
from survey data, because all 5-Year Client Survey sample members are women.

Before running procedures to estimate impacts, researchers should merge the
survey data BY IDNUMBER with the research group dummy variables and
covariates from the Full Impact Sample file.  Use SRV5RESP to select survey
respondents.

With two exceptions, MDRC used the same procedures for estimating program
impacts with survey data as were used for estimating impacts with
administrative records.  (See IMP_MEMO.TXT).

The exceptions are:

1) In Atlanta, Grand Rapids, Portland, and Riverside, certain subgroups
were oversampled for research purposes when choosing the survey sample.
Therefore, it is necessary to weight the survey sample to make impact estimates
generalizable to the larger "survey eligible" sample from which respondents were
drawn.

The variable called FLD5WGT is used to weight the survey sample. (See
N5PSSAMP.TXT for details).  FLD5WGT should also be used when calculating
adjusted means.  (See IMP_MEMO.TXT for general information on
calculating adjusted means.)


For example, in Atlanta and Grand Rapids, if

WtBJC=All sample members weighted by FLD5WGT

WtB=HCDs weighted by FLD5WGT

WtJ=LFAs weighted by FLD5WGT

a) Adjusted mean for control group=

ADJMEANC=

WtBJC(GDEPVAR) -  WtB(COEFOFB  * MEANOFB) -   WtJ(COEFOFJ  * MEANOFJ)

Weighted          Weighted        Weighted      Weighted      Weighted
site mean         HCD impact=     proportion    LFA impact=   proportion
of dependent      coefficient     of HCDs in    coefficient   of LFAs in
variable          of B from       the sample=   of J from     the sample=
from              regression      Mean of B     regression    Mean of J
MEANS             output          from          output        from
output                            MEANS                       MEANS
                                  output                      output


b) Adjusted mean for HCD group=

ADJMEANB        =       ADJMEANC      +     WtB(COEFOFB)




c) Adjusted mean for LFA group=

ADJMEANJ        =       ADJMEANC      +      WtJ(COEFOFJ)



Note:  Weighting all survey respondents by FLD5WGT produces sample sizes
(by site) close to but not exactly the same as the unweighted sample sizes:


                          Unweighted N   Weighted N
Atlanta           HCD              594      552.136
Atlanta           LFA              519      559.051
Atlanta           CTL              552      552.998
Total                             1665     1664.185

Grand Rapids      HCD              547      551.643
Grand Rapids      LFA              535      550.435
Grand Rapids      CTL              562      539.254
Total                             1644     1641.332

Portland          PROG             281      332.743
Portland          CTL              278      225.330
Total                              559      558.073

Riverside         HCD              376      294.966
Riverside         LFA              499      638.349
Riverside         CTL              720      663.979
Total                             1595     1597.294

When estimating program impacts in SAS (used by MDRC and Child
Trends), the weight variable does not change degrees of freedom or number
of observations in calculations of statistical significance.  However,
researchers using a different software package or running a different procedure
in SAS (such as a crosstabulation with chi square) should first check whether
weighting by FLD5WGT will inappropriately alter the chances of finding
a statistically significant program-control group difference.  If so,
researchers should multiply FLD5WGT by a second weight that returns the sample
sizes to the unweighted number.

IMPORTANT:

1) To calculate impacts for Riverside LFA, weight survey respondents by
FLD5WGT, then follow the procedures for weighting the results again that are
outlined in IMP_MEMO.TXT.  This additional step is needed because FLD5WGT
makes the background characteristics of the survey sample similar to the
characteristics of all members of the Full Impact Sample who were eligible to
be surveyed.  In the Full Impact Sample, however, sample members determined
not to need basic education are overrepresented among LFAs and control group
members.  (See RES_MEMO.TXT).  An additional weight must be applied to make
the results generalizable to the welfare population who were required to
participate in a welfare-to-work program in Riverside during the early to
mid-1990s.


2) Some outcomes on the survey have missing values.  Researchers should be
careful about grouping together outcomes with different sample sizes when
calculating program impacts.  Statistical packages like SAS typically use
LISTWISE deletion as the default for regression or GLM, in which case only
sample members with no missing values on all dependent variables will be
included in the calculations.


3) Researchers will obtain slightly different impact results from those
displayed in report tables. (The tables on this CD were copied from the
Final Report.)  These differences result from small changes to some background
characteristics measures used as covariates in the impact regression model. MDRC
implemented these changes to protect sample members'confidentiality.
