Appendix

Analysis of Best Practices for Take-Up Rates
NORC and Georgetown University
Preliminary Literature Review
January 31, 2008

OVERVIEW

The purpose of the project, “Analysis of Best Practices for Take-Up Rates” is to provide models of best practices that are relevant for States and private health insurance plans as they contemplate how to best tailor outreach, enrollment, and retention for the Administration’s Affordable Choices Initiative as well as existing health insurance programs.

A literature review, the first major task associated with the project, catalogues resources on the basis of populations featured in the research, types of take-up strategies and activities studied, and the study methods used.  The resources gathered were reviewed to determine which should be included in the next phase of the project and what form the next phase should take.  The project is designed to feature a meta-analysis to answer the key research questions:

• What types of outreach, enrollment and retention strategies have proved most effective in ensuring that uninsured individuals have public or private coverage?
• Which outreach, enrollment and retention strategies are particularly cost-effective?
• Which outreach, enrollment and retention strategies are particularly effective for special populations (those living in urban and rural areas, those whose first language is not English, and those whose cultural background or practices are different from the majority of program participants)?
• What lessons are particularly important to consider with regard new coverage expansion initiatives, such as the Affordable Choices Initiative?

An examination of the database constructed during the literature review suggests that the number of methodologically rigorous studies is small and that it may therefore be prudent to consider modifications to the meta-analysis approach.  There are few studies in the database that definitively demonstrate causation between outreach strategies and reported outcomes.  Strong correlations are reported, however, and these findings can help inform responses to the research questions.  This report describes the methods used to construct the database and the contents of the database and discusses how the data can be used to answer the research questions posed for this project.

METHODS

We identified all studies that might be relevant for the meta-analysis.  A thorough review of the articles followed and then we constructed a project database.  These tasks are described in more detail below.

Gathering resources

Health and social science databases including Medline, Healthstar, Social Science Abstracts and CINAHL were searched to identify possible relevant studies or articles published or completed in the last ten years.  Keywords used in searching each database include: take-up, enrollment, outreach, retention and churning.  We also looked for literature on private health insurance (small group, individual market, and employer-sponsored) and public health insurance (Medicaid, SCHIP, Medicare, Part D low-income subsidy, prescription drug cards, and state health insurance programs.)   Several specific peer-reviewed publications were searched for relevant articles as well.  They include Health Affairs, Health Services Research, the Journal of Health Care for the Poor and Uninsured, The Millbank Quarterly, and the Journal of the American Public Health Association. 

The same criteria and keywords were used to search the “gray literature,” including technical reports, working papers, conference proceedings, and newsletters from foundations, government sources, and policy or advocacy organizations.  Some of these resources are available in the files on this topic maintained at Georgetown’s Health Policy Institute.  Other studies were identified by searching websites for relevant organizations and projects.  The technique of snowballing was also used to gather resources; bibliographies were checked for additional sources.  We identified a total of 1,177 articles and created a master spreadsheet of articles.  

Reviewing resources

In gathering the literature we cast a wide net.  In reviewing the resources we looked for very specific literature.  Initially, some 864 articles were classified as “not relevant.”  Based on our knowledge of the literature as well as a review of abstracts and executive summaries, these articles where excluded for reasons such as:

1. They are opinion or editorial pieces or blueprints for change.
2. The focus is on barriers to health care services.
3. They examine “crowd out.”
4. They concern the effect of health care coverage on providers.
5. They examine employer or consumer choice.
6. They are about health status or policies to reduce racial and ethnic disparities.

The majority of articles chosen for an in-depth review were read by a research associate at HPI.  Additional staff at HPI and NORC read the remainder.  Of the 313 articles, 229 more were excluded upon further review because they do not provide information about effects of initiatives to increase take-up rates.  These articles were reviewed by senior staff at HPI before being excluded from the database.  Table 2 summarizes the following general subjects covered by the 229 excluded articles.  These include:

1. Describe barriers to enrollment or retention
2. Present data on patterns of insurance coverage (particularly numbers eligible or enrolled) or describe the characteristics of individuals or groups that do or do not have health insurance
3. Describe the consequences of having or not having health insurance
4. Discuss program policies that affect take-up or retention, rather than initiatives to improve take-up (for example, studies about policies such as: parental expansions, continuous eligibility, the use of premiums, the availability of subsidies were excluded)
5. Concern strategies to increase the use of health care or other services rather than to increase enrollment in health insurance or other benefits
6. Describe a specific process to improve take-up, but do not describe outcomes

The remaining 84 articles comprise the database for this project (see Table 1).

Table 2: Topics of articles excluded from database

THE DATABASE

The project database contains detailed descriptive information about the population groups involved, the type of take-up strategies and activities studied, and the type of study featured in each article.  A large majority of the studies in the database focus on the enrollment and retention of children in public benefits programs.  Four broad types of take-up strategies are featured in the database: strategies to publicize benefits, to provide special assistance for applicants and enrollees, to simplify the application process, and to simplify the renewal process.  Each represents a number of different activities.  A variety of evaluation approaches are used in the research, with case studies being the most popular method among the studies in the database.  The following suumarizes the database in the following sections:

• Groups represented in the database
• Types of takeup strategies and activities studied
• Evaluation approaches
• Quality of the studies in the database

Groups represented in the database

To characterize the studies, we examined the type of coverage studied, the age group targeted, the geographic region represented, and the inclusion of certain groups of participants of interest such as those living in rural areas, those with limited English proficiency, and those representing certain cultural groups.

Type of coverage

Most studies examine public rather than private coverage, though these categories are not exclusive.  Some studies, such as those about high-risk pools – which are generally state sponsored, but provide enrollees with private insurance coverage – would also fall into both categories.  Some states and localities also sponsor their own insurance coverage and outsource their administration to private insurance companies.  As Table 3 indicates, public benefits, primarily insurance programs, are the subject of 72 of the studies in the database.  Six studies concern only private coverage and six others refer to both public and private coverage. 

Within the public insurance category, the focus of the great majority of the studies is on the Medicaid or SCHIP programs, which serve children and some parents.  Public benefit programs for older people or people with disabilities, including the Medicare program, the Medicare Savings Programs (SLMB, QMB, and QI), and Supplemental Security Income (SSI) are also represented but to a much lesser extent.  Private coverage was featured in a large number of the articles collected originally, but most were excluded because they examine the effect of benefit design (subsidies, premiums, other costs) on take-up rather than the effect of specific initiatives to increase take-up. 

Table 3: Articles in the database by type of coverage¹

Includes a small number of articles referring to benefits other than insurance coverage

Age

The populations studied reflect the programs studied.  Therefore, most of the studies in the database concern coverage for children.  A few consider coverage for both children and their parents.  There are relatively few studies focused on the enrollment of older persons or persons with disabilities or on couples or single adults.  The remaining research considers coverage for several age groups or the general population, through a mix of public and private approaches (see Table 4).

Geography
Since the majority of articles concern the Medicaid and SCHIP programs, which are jointly administered and financed by states and the federal government, it is logical that the majority of articles in the database have a state focus.  There is some overlap among geographic category.  For example, some research examines local or regional initiatives to enroll the uninsured into state or national programs.  Likewise, a study detailing Medicaid and SCHIP enrollment in several states could be classified as both state and national.  Studies are classified in Table 5 by the largest geographic area they represent.

Table 5: Articles by largest geographic region represented

Some states are featured more frequently in the identified articles than others.  States represented in ten or more of the articles include some of the most populous states: California, Florida, Georgia, Massachusetts, New York, Ohio, Texas, and Washington.  Activities in California and New York featured in 19 and 22 articles, respectively.  

Populations of particular interest

Certain populations are of particular interest for policymakers considering how to increase insurance coverage among populations that may differ in certain ways from the general population.  Among the articles in the database, 10 provide information about reaching people living in rural areas.  Strategies aimed at reaching individuals who have limited English proficiency are featured in 23 articles.  An additional 18 articles discuss the use of strategies geared to certain cultural groups such as Latinos and Asians.  We also identified one study dealing with outreach to undocumented immigrants, and one that referenced outreach to uninsured adolescents.

Types of take-up strategies and activities studied

Several types of strategies can be used to reach people, inform them about the availability of benefits, and help them enroll in benefit programs.  Studies are broadly categorized in the database according to whether they examine four types of strategies.  A number of studies examine more than one strategy.  Thus, strategies are discussed 155 separate time in the 84 articles.  Table 6 shows the number of studies that discuss each strategy, though not all strategies were evaluated in all studies.  The most popular strategy examined in the articles is the provision of special assistance for applicants or enrollees, which was cited in 57 of the 84 studies.  Efforts to publicize benefits were also the subject of almost half (41 of the 84) studies on take-up.

Table 6: Types of take-up strategies represented in the database

Publicize benefits  

Prior research suggests that many eligible families simply do not know about health insurance programs or are not aware that they may qualify for benefits.  Efforts to raise awareness of programs might include media campaigns, television, radio, or print media, and advertisements on public transit or billboards.  Another outlet is the use of products – objects printed with a program’s information that can be easily distributed, like fans in churches, emery boards at nail salons, and tray liners at fast food restaurants.  Health fairs allow program representatives to speak to potential enrollees in person.  Another tactic, “in-reach,” uses existing organizations such as schools and employers to inform people about the availability of benefits.  As Table 7 shows, media campaigns are the most frequently mentioned activity.  Publicity campaigns may be conducted in languages other than English.  Some 18 of the database articles highlight efforts conducted in other languages other than English to target specific populations.

Table 7: Types of publicity activities represented in the database

Special assistance for applicants and enrollees

Enrollment facilitators guide individuals who need assistance through the process of enrolling in public or private benefit programs or help them maintain their benefits.  Facilitators may be associated with community organizations or other groups.  Community groups provide other types of enrollment assistance as well, such as sponsoring outreach events or making translation services available.  Outreach and enrollment efforts also involve health care providers.  Some providers have an incentive to help uninsured patients sign up for health insurance to reduce the amount of uncompensated care that they provide.  Emergency rooms and community health centers are among the most common sites where individuals receive assistance with benefit applications.  As Table 8 shows, activities involving community workers have been evaluated most frequently. 

Table 8: Types of assistance activities represented in the database

Simplify enrollment
Activities and practices to simplify enrollment may include offering applications and assistance in several languages; eliminating requirements for face-to-face interviews, which spares enrollees the need to take time off from work or travel to an office; creating shorter, simpler applications; allowing applicants to submit paperwork through the mail or online; and allowing individuals to apply for benefits “in the field,” rather than in a Medicaid office.  Programs can further simplify their enrollment by using data already on file from other programs or sources to verify an applicant’s residence, age, or income (see Table 9).

Table 9: Types of enrollment activities represented in the database

Simplify renewal 

Attempts to increase take-up rates cannot succeed if individuals who have coverage lose it.  Most Medicaid and SCHIP programs require enrollees to confirm their eligibility after 6 or 12 months of coverage; for a variety of reasons, many eligible beneficiaries lose coverage. Efforts to simplify the eligibility re-determination process for public programs are consequently needed to help keep people insured.  Some programs or organizations, including managed care organizations, remind enrollees by mail, phone, or an in-person visit that they will need to confirm their eligibility.  They may also follow up with individuals or families that have not responded to notices regarding re-determination.  Another approach is “passive” renewal, in which programs use the data provided in an initial application to fill out a renewal form, which is then sent to enrolled families.  Families are only required to return the form if their circumstances have changed; otherwise, the enrollee is deemed eligible.  Families have even fewer obligations in an “ex parte” renewal process, where their continued eligibility is confirmed based on data from other sources – food stamps, TANF, or wage records.  As Table 10 indicates, the renewal activities most commonly cited in the database are reminders or follow-up with beneficiaries.

Table 10: Types of renewal activities represented in the database

Evaluation approaches

Six broad classifications are used to distinguish the approaches or techniques represented in the studies included in the database.  Table 11 shows the primary method employed. 

Table 11: Types of studies in the database

Case studies

The case study method is the most common.  The category includes research about singular efforts (for example assistance for patients in a hospital emergency department) or about a variety of strategies implemented at several sites.  Some of the case studies use data from a site visit or visits.  The number of enrollees involved in the efforts studied ranges from fewer than 100 to several thousand.  Almost all of the case studies are cross-sectional studies.  All provide descriptive data.  In addition, many include quantitative measures, generally reports of changes in the numbers of program enrollees.  Only a few case studies control for various factors that may affect enrollment.

Focus groups

Only two of the studies represented in the database primarily use focus groups.  One set involved 55 and the other 94 participants.  A number of the articles we collected originally use the focus group technique, but generally they describe barriers to program participation rather than outcomes of an intervention to increase participation. 

Surveys or structured interviews

Responses gathered through surveys or structured interviews of substantial numbers of participants – generally 500 to 1,000 respondents – comprise this category.  Most are cross-sectional though a few follow panels of participants.  The studies differ considerably with regard to how carefully the samples were chosen, how representative they are of the population studied, and whether they are applicable for other populations.  Most are conducted following an initiative, but do make comparisons to circumstances prior to the initiative.

Analysis of existing data sets

Major national or state surveys with data for many thousands of individuals have been examined to glean information about the efficacy of efforts to improve take-up rates.  Similarly, enrollment or other program data have been used to try to make determinations about whether particular interventions have an impact on enrollment.  For the most part, these are longitudinal time-series analyses that attempt to correlate activities with changes in enrollment.  About one-third of these studies try to control for factors other than the intervention that may affect enrollment, while the others typically only look at the change over time without regard to other potential factors. 

Descriptive overviews

In an effort to include as much information as possible about the outcomes of all types of efforts to increase enrollment, these studies also are included in the database.  Generally the articles describe and summarize activities, often from a variety of sites, and attempt to draw conclusions or make recommendations based on observations about current practices.  They have less analytical content than the other articles and therefore do not qualify for categories described above, but some quantitative data regarding particular activities is presented and therefore they are included in the database. 

Research synthesis

Two articles included in the database are of particular significance for this project because they attempt to synthesize research on take-up strategies.  They are discussed in more detail below.

Quality of the studies in the database

One purpose of gathering and reviewing articles for the database is to determine what types of research have been done to examine the efficacy of initiatives to increase take-up rates for public and private insurance coverage.  Each of the 84 articles included in the database presents some good information on this topic.  The studies are very different, however, in terms of how rigorously they are designed and carried out and how methodologically sound they are.  The parameters discussed below are useful in making determinations about the quality of the research represented in the database. 

Quantitative data in the studies

A substantial proportion of the articles in the database – 79 percent, or 66 articles – contain some quantitative data.  As noted above, the types of data and the way they are analyzed and reported differ considerably, however.  In the majority of studies the reported data are not subject to statistical tests.  In a small number of studies, the data are used to demonstrate the relationship between an initiative and the take-up rate.  More often, the percent change in program enrollment is the reported outcome but there are no indications of the extent to which the change might or might not have occurred in the absence of the initiative – an increase in enrollment in the months following a media campaign or efforts to streamline an application process.  Nor do the studies explain whether the change is related to the number of individuals entering or leaving the program.  It is also important to note that some quantitative measures are more meaningful than others.  For example, a number of studies report the number of people reached by a given outreach strategy – the number of people who visit a health fair, for example, or the estimated listening audience for a radio announcement.  These data are of interest, but not sufficient to show whether initiatives are successful at increasing coverage. 

Peer-reviewed studies

Generally, peer-reviewed studies are considered to be more methodologically rigorous than other studies.  As noted above, the database contains articles from peer-reviewed publications and others identified in the “gray literature,” studies sponsored and published by foundations, government agencies, and policy or advocacy organizations.  About one-third of the studies in the database are from peer-reviewed publications.  Table 12 shows the number of peer-reviewed articles by type of study.  Studies involving focus groups and descriptive overviews are least likely to be peer-reviewed.  About half of each of the other types of studies are peer reviewed.

Table 12: Peer-reviewed studies in the database, by type of study

Table 13 indicates whether the studies that discuss each take-up strategy are peer-reviewed. 

Studies about providing special assistance for applicants and enrollees are most likely, relative to other strategies studied, to appear in peer-reviewed publications.  Studies about simplifying the renewal process are least likely to be peer-reviewed.

Table 13: Peer-reviewed studies among articles in the database that discuss each strategy

Study design: examining cause and correlation

Only four of the studies in the database are sufficiently rigorous in design to allow the authors to draw conclusions about the cause of outcomes.  They control for exogenous influences with randomized samples, quantitative models or the use of control and comparison groups.  Three studies in the database are randomized controlled studies: 

• A Randomized, Controlled Trial of the Effectiveness of Community-Based Case Management in Insuring Uninsured Latino Children (Flores et al.) randomly assigned uninsured children to an intervention group with trained case managers or a control group that received traditional outreach.  Interviews with parents provide data for the evaluation. 
• The State Children's Health Insurance Program: A Multicenter Trial of Outreach Through the Emergency Department (Gordon et al.) experimented by giving an intervention group of uninsured children SCHIP applications when they presented at one of five emergency rooms, and not giving applications to a control group.
• Effectiveness of a Targeted Mailing Outreach Program on SSI Applications and Trends (Goulet et al.), examines an initiative by the SSA to send letters to persons who were receiving Retirement, Survivors and Disability Insurance (RSDI) benefits but were not receiving Supplemental Security Income (SSI).  Letters encouraging SSI applications were sent to a random sample of individuals deemed likely to be eligible; those who did not receive a letter served as controls.

The fourth highly-rigorous study, The Effects of County "Outreach Environments" on Family Participation in Medi-Cal and Healthy Families (Kiloche et al.), involves a multivariate analysis of data from two large data sets to examine the impact of outreach funding and activities in different counties. 
Some seven other studies specifically mention that in conducting the study, attempts have been made to control for factors other than the intervention.  Six others note that the study population was randomly selected.  Most of the other studies demonstrate correlations between activities and changes in enrollment.

Measures of cost-effectiveness

Measures of cost-effectiveness are of particular interest for this project.  Thus, we examined articles for any data related to the cost of outreach, enrollment, or retention activities.  Some 27 studies in the database mention cost.  Of these, five provide data on the cost per enrollee or potential enrollee of particular initiatives.  Two calculate the magnitude of expected enrollment increase based on spending for outreach.  The remainder report on other financial aspects of the interventions (see Table 14).  In addition, a few studies that do not mention cost provide enough description to indicate that the initiative is very resource-intensive.

Table 14: Types of information about cost in the studies

DATABASE FINDINGS: IMPLICATIONS FOR THE TAKE-UP PROJECT
The purpose of constructing the database was to collect and review studies that could be used as the basis for a meta-analysis that ideally, would allow us to answer the following questions:
• What types of outreach, enrollment and retention strategies have proved most effective in ensuring that uninsured individuals have public or private coverage?
• Which outreach, enrollment and retention strategies are particularly cost-effective?
• Which outreach, enrollment and retention strategies are particularly effective for special populations (those living in urban and rural areas, those whose first language is not English, and those whose cultural background or practices are different from the majority of program participants)?
• What lessons are particularly important to consider with regard to the Affordable Choices Initiative?

Based on our assessment of the research included in the database, a meta-analysis is not recommended as a means to answer these questions.  The methods used to construct the database ensure that it provides a rich source of reliable information that can be used effectively in other ways to provide answers to the research questions, however.
Considerations regarding the meta-analysis
Although meta-analysis is more commonly used with clinical research studies, it can also be used to examine research related to interventions that occur in non-clinical settings.  Typically, a meta-analysis combines the results of several isolated studies with small sample sizes to create one synthesized result with increased statistical power resulting from the combined sample sizes. 
To be combined in this way, studies included in a meta-analysis are usually all focused on the same research hypothesis.  The literature review indicates that although there is a substantial body of literature, there is not enough uniformity to support a meta-analysis.  Comparison among the studies is complicated by the fact that they do not evaluate a single strategy, but a group of strategies and for each strategy a set of activities.  In addition, it is rare that only one intervention is initiated.  Most often, when there is a policy decision to make an effort to increase take-up, many activities occur.  This reality makes it difficult to conduct definitive outcome-based research and to make comparisons among studies.
Furthermore, the results of a meta-analysis are only as reliable as the studies that feed into the analysis.  The meta-analysis technique takes the relative reliability of the findings from a number of studies into account, and considers the size of the effect reported in synthesizing results.  As the description of the database presented above indicates, however, there are not enough studies of sufficient rigor or quality on which to base a meta-analysis, even on a narrower topic.  For example, of the three controlled experiments described above, one pertains to the strategy of providing assistance and two pertain to the strategy of publicizing the availability of benefits.  And, the two involve different populations – families and the elderly. 
One potential difficulty is that in studies of a non-clinical nature it is more difficult to control for exogenous factors that may affect outcomes.  Many studies did not sufficiently address this concern, making it impossible to determine whether the observed effects were caused by the intervention or by other factors.  We would be reluctant to combine results from such studies with the results of others which did carefully control for exogenous variables.
Lessons from similar research
Past efforts to synthesize research on take-up strategies support the conclusions presented here. In the course of conducting the literature review, we found a publication prepared for the Agency for Healthcare Research and Quality in March 2000, Review of the Literature on Evaluations of Outreach for Public Health Insurance and Selected Other Programs (Laschober et al.).  It describes a literature review and synthesis of the empirically-based evidence on the effectiveness of interventions to increase enrollment of low-income children and families in public health insurance programs.  Methods similar to those developed for this project were used to find and catalogue the literature: nine completed evaluations and 17 in progress at the time.  The study authors concluded,
“that rigorous evaluations…are largely absent from the published literature…no rigorous empirically-based evaluations exist.  Of the few interventions for which there are completed evaluations, we did not identify a body of literature regarding any specific intervention or set of interventions that would help us to conclude with a high degree of confidence that the results are valid.”
The authors also note, “We did not locate any completed cost-effectiveness evaluations.”  While a great deal more research has been conducted since 2000, the topic still has not invited widespread rigorous evaluation. 
A 2003 study, What Other Programs Can Teach Us: Increasing Participation in Health Insurance Programs (Remler and Glied), examines the take-up literature across a variety of programs.  The study authors note, “our approach is akin to that of a meta-analysis, although we cannot do a formal meta-analysis because of insufficient structure and commonality across both programs and estimation strategies.” 
Both reviews reach similar conclusions about the feasibility of conducting a meta-analysis, but it is important to note that both also are able to draw some meaningful conclusions based on a careful systematic review of the available literature.  The Laschober study conducted and reported on a detailed review of nine studies.  Remler and Glied identified 37 studies that contain data on the magnitude of effects (quantitative and non-quantitative) and examined the effects to draw conclusions about methods to increase take-up.

Options for the take-up project
We continue to think that an evaluative rather than a descriptive exercise should be undertaken – one that might serve as the next best alternative to a meta-analysis.  The database can be used to conduct a focused literature review that presents reasonable defensible conclusions based on the best available evidence to help answer the research questions. 
As indicated above, we have constructed a useful database with a very well defined collection of articles that provide a wealth of evidence to examine.  Strong consideration was given in the initial review of the literature to the quality of the research.  The articles included in the database present evidence rather than opinions.  The collection of articles is focused, pertaining just to strategies to promote take-up, rather than to program participation barriers.  It is also limited to interventions, rather than program design or policies that may affect take-up.  Although there is a dearth of causal evidence among the studies, there are some strong correlations reported.  An even more careful, focused review of the data in the database can provide answers to some of the research questions. 
The key in using the database will be to review the data in a careful structured manner.  This can be accomplished with a strategy-specific review that examines the literature pertaining to each of the four strategies, presents findings based on available evidence from the database, explains which findings are strongest and why.  In addition, although this project was not designed originally to develop recommendations about evaluation techniques, we have become quite familiar, in the course of reviewing resources for inclusion in the database, with the strengths and weaknesses of the research that has been done to date on this topic.  Therefore, as part of a “strategy-specific” review, it would also be possible to include a discussion or make recommendations – based on examples from the literature – regarding the type of studies that could be done to improve the availability and quality of future research on the topic of take-up.
 

Where to?

Main Page of Report

Home Pages:
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)