Using Propensity Score Matching to Evaluate Differences in Public Using Propensity Score Matching to Evaluate Differences in Public and Private Students on Self-Control and Private Students on Self-Control

Abstract With the Educational Childhood Longitudinal Kindergarten Cohort class of 2011 (ECLSK-2011) database, I used propensity score matching and a longitudinal multilevel model to evaluate how 4970 kindergarteners in public (assigned public, charter, or magnet) and private (religious, non-religious private) schools, who were matched on a host of covariates, differed on a psychometrically sound self-control construct from fall 2010 to spring 2012. I found no statistically significant difference in self-control scores between private and public-school children. I discuss possible more effective and equitable mechanisms for improving student self-control, as well as suggesting guidance for future research on this topic.


Introduction
Inequities across the school system are still a major issue in the United States.Since the 1970's, the gap between African American and white students on important educational outcomes such as, math, reading, and college completion has yet to go below 25% through 2010 (Gamoran, 2015).Additionally, the gap between college completion rates between African-American and white students as well as between low and high-income students has either increased or remained the same since the 1970's through 2010 (Gamoran, 2015).Other researchers have found similar results demonstrating a pattern of increasing inequity among students of different ethnic and socioeconomic backgrounds (Reardon, 2011;Orfield & Frankenberg, 2014;Kaushal, Magnuson, & Waldfogel, 2011).For example, the office of Civil Rights found that in the 2009-2010 school year, Hispanic students were twice as likely as white students to be held back a grade (Adams, Robelen, & Shah, 2012), while Native American students continue to have the highest high school dropout rates (Faircloth & Tippeconnic, 2010).Unfortunately, when inequity exists in schools, non-white and low incomes students are at higher risks of not graduating, lower rates of physical development, and lower incomes (Orfield, & Lee, 2005).
Given the lack of equity and the consequences associated with it, some policymakers have looked to the controversial policy of school vouchers for students to attend a private school (Altidor, 2005;Department of Education, 2017;Weil, 2002).One idea behind public funding for private schools is that with more choice non-white and or low-income students can choose to attend private schools, which in theory will provide higher levels of education eliminating the achievement gap (Friedman, 1955;Harvey, 2005).Using Friedman and other neo-liberalists market-based theories, the idea is that with vouchers for private schools, low income and minority students whom start at a disadvantage will have the necessary resources to leave a failing school if they choose.
Although, there is a robust yet mixed literature on the effects of voucher programs for private schools (Abdulkadiroglu, Pathak, & Walters, 2015;Austin & Berends, (in press); Epple, Romano & Urquiola, 2015;Figlio, 2009), there is no known research on the effects of private schools' effects on students' social and emotional development constructs, such as self-control.Duckworth, in Duckworth, Gendler, & Gross (2014, p. 200) defined self-control as, "voluntary regulation of attentional, emotional, and behavioral impulses when immediate temptations conflict with more enduringly valued goals".In her review of self-control literature, Duckworth finds two key components of self-control: long-run rewards and a self-initiated component."Long-run rewards" means being able to delay gratification for better future opportunities (e.g., the marshmallow experiments)."Self-initiated" means the construct is initiated by the individual.Although, people can learn techniques to improve their self-control, it is not a construct that others impose upon oneself but, instead, it is an internal process (Duckworth et al., 2014).
An example of what self-control might look in a child, is the marshmallow experiment.The marshmallow experiment usually provides children with some choices: they are often presented with something a child may find appealing to eat, like a marshmallow, but sometimes a cookie, pretzel, etc. may be used.Children are told if they can wait a certain amount of time they will receive an additional marshmallow or whatever food item the experimenter chooses to use.Several researchers have shown that, over time, children who could delay their gratification (i.e., wait for a second marshmallow, cookie, pretzel, etc.) demonstrated higher levels of selfcontrol 30 and 40 years later (Casey et al., 2011).
Such social and emotional development factors such as self-control are often just as or more important to student outcomes (e.g., educational attainment, labor market performance, grades, and health) than traditional cognitive measures (e.g., reading, math, or attendance) (Almlund, Duckworth, Heckman & Kautz, 2011;Heckman & Kautz, 2012;Kern & Friedman, 2008;Lleras, 2008;Moffitt et al., 2011).For example, Moffitt et al. (2011) looked at over 1,000 participants from birth to age 32, finding that self-control statistically significantly improved performance on two dimensions of health (physical health and substance dependence) and six dimensions of wealth (Socio-economic status (SES), income, single-parent child rearing, financial planfulness, financial struggles) and significantly decreased one undesirable dimension of public safety (criminal conviction).Therefore, even if there is mixed evidence for private versus public schools on educational outcomes, it would be helpful to know whether public or private schools are increasing student's self-control.Because of the positive impacts that selfcontrol has on student outcomes, private or public schools could be helping to alleviate the inequity problems the education system faces by increasing student's self-control.
Given the importance of developing social and emotional skills, such as self-control and its potential to improve equity, school practitioners (e.g.school social workers, school counselors, school psychologists) are beginning to invest in programming that has been shown to improve social and emotional skills, such as social and emotional learning programming (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011;Taylor, Oberle, Durlak, & Weissberg, 2017).Durlak et al.'s (2011) meta-analysis included over 270,000 students across 213 school-based Social and Emotional Learning (SEL) programs, finding that SEL programs with adequate implementation found percentile point increases ranging from 11 to 31 points on both social and emotional skills and academic outcomes respectively (Durlak et al., 2011).Additionally, a recent update on the Durlak et al. (2011) analysis that looked at almost 100,000 students across 82 programs finding similar results across the same constructs (Taylor et al., 2017).
Not only are SEL programs effective, but they are cost-effective.The Washington State Institute for Public Policy has conducted cost benefit analyses for several social and emotional learning (SEL) programs.For example, they found that popular programs such as the Good Behavior Game and Positive Alternative Thinking Strategies (PATHS) interventions generate a return on investment of $65 and $20, respectively, for every dollar spent (Washington State Institute for Public Policy, 2017).
Additionally, it is important that students learn social and emotional skills early because they are harder to learn at later ages (Taylor et al., 2017).Some researchers have posited that learning social skills as a young adult is more difficult than earlier childhood because adolescent relationships are more complicated (Gantman, Kapp, Orenski, & Laugeson, 2012).Additionally, studies have shown that children with social and emotional disorders such as Autism get progressively worse as they age making it more important to intervene early (Lange et al., 2015).Therefore, if schools want to provide effective social and emotional skills development, they should not only intervene with effective and cost-efficient programming, but do so as early as possible.
Given the positive impact that increased self-control at an early age can play in increasing students outcomes and thus possibly reducing student inequities and the competing policy options (school vouchers, SEL programming), I compared 4970 public (assigned public, charter, magnet) and private (religious, non-religious private schools) kindergarteners during Fall 2010 to Spring 2012, using propensity score matching, multilevel modeling for longitudinal analysis, and data imputation methods to evaluate differences in how teachers perceive a student's self-control.By using propensity score matching to identify students who are similar on several relevant covariates, I can provide an indication of whether private or public schools produce students with more self-control, contributing the first evidence on how attendance at private versus public schools affects students' self-control.With this evidence, I can help forward the policy debate on whether private schooling can help, hinder, or have no effect on inequities in the school system.

Data
The Institute for Educational Sciences (IES) published the Early Childhood Longitudinal Study Kindergarten Class of 2011 in 2015 (ECLS-K-2011).The ECLS-K-2011 dataset provides information on students' academic, social and emotional development, and demographics from a variety of sources (e.g., students, parents, teachers, principals, and before-and after-school providers).Before matching students on covariates, there was a total of 18,174, students who either began kindergarten for the first time or were repeating kindergarten.After matching, there was a total of 4,970 (2,485 students on each side of the private / public divide) students in the sample.

Outcome (Self Control), Private, and Time Variables
The Institute of Educational Sciences (IES) constructed the outcome variable using items from the Social Skills Rating System (SSRS).IES created what they called self-control (four items) from the SSRS (Elliot & Gresham, 2008).Furthermore, the measure showed some evidence of reliability, with a Cronbach's Alpha of .81(IES, 2015).Self-control is teacherreported on a Likert scale, with options ranging from 1 to 4, with higher scores representing more of the behavior in question.The self-control variable is an average score across the four items.
The variable of interest is the school type variable.School administrators identified whether their school fell into one of the following categories: regular public school, public magnet school, charter school, public school of choice, Catholic school, or other private school and religious affiliation school.The variable of interest is Private, which is a 0 for all publicschool options (regular public school, public magnet school, charter school, and public school of choice) and a 1 for all private school options (Catholic school, or other private and religious affiliation school).Finally, I created a time variable that indicates whether a student's value for the included variables was associated with time 0 (fall 2010), time 1 (spring 2011), time 2 (fall 2011), or time three spring ( 2012).To answer our research question of whether there is a statistically significant difference between matched students in private and public schools on self-control over time, I included an interaction term between Private and Time.

Matching predictor variables
To match students in public and private schools, I included a host of relevant child and parental demographic variables that could be related to a child's self-control, which are presented in Table 2 along with the means and relevant standard deviations for students included in the matching, which are broken down by public and private schools for pretreatment (fall 2010) (Ho, Imai, King, & Stuart, 2011).I also included two cognitive measures; one for math and the other for reading.Each measure was administered using a computer adaptive format where students continued answering questions until a measure of their ability (i.e., a theta value) could be measured with a specific level of precision (IES, 2015).For child demographic variables, I included gender (male, female), body mass index (BMI), and race (white, non-white).For parental demographics, I included the following information on the parent and or guardian who answered the questionnaire, employment status (35 hours or more, less than 35 hours), race (white, non-white), and the following information on the child's household, primary household language (English, Non-English), poverty level (below or at poverty level, above poverty level), and number of persons in the household.Previous researchers have shown each of these variables has some relationship to students' social and emotional development justifying their inclusion (Office of Planning, Evaluation and Policy Development, 2016;Powell, Slater, & Chaloupka, 2004;Richardson, 2008;Wang, 2001;White, 1982).More information about the included is located in the ECLSK-2011 manual (IES, 2015).

Analysis
Because for some variables as much as 28% of the data set is missing, as described in Table 1, I used the Amelia package in R to impute missing values (Honaker, King, & Blackwell, 2011;IES, 2015).Amelia uses an EMB algorithm that has evidence for being just as accurate as other algorithms (IP, EMis) with much greater processing speed (Honaker et al., 2011).Once the missing data were imputed, I created the propensity scores that students in public and private schools would be matched on, using the MatchIT R package.This package uses the matching covariates described in Table 2 to create propensities (i.e., probabilities of being in a public and private school) for both students who are and are not attending a public or private school in the fall of 2010.Those students who have similar propensity scores in private and public schools are matched with one other individual in the opposite group (i.e., each private school student is matched with one public school student with a similar propensity score) using a nearest neighbor algorithm (Ho et al., 2011).I used MatchIT's diagnostics tests to evaluate how well the algorithm matched students.
To account for the nesting of time points within people, I used a multilevel model.Multilevel models can account for possible correlation between time points by allowing each person to have their own intercepts (e.g., random intercepts) and trajectory over time on each parameter value (e.g., random slopes) (Hoffman & Rovine, 2007).I used a model comparison approach, where I compared models, evaluating criteria such as AIC, BIC and likelihood ratios, to sequentially compare the models (Hoffman et al., 2007).I used the nlme package (Nonlinear mixed effects) in R to conduct the multilevel longitudinal modeling with a maximum likelihood (ML) estimator.Equations 1 through 4 display the exact model.Given that we matched participates on baseline time invariant covariates displayed in Finally, I am taking an intent-to-treat (ITT) approach.An ITT approach does not worry about issues such as noncompliance, protocol deviations, or withdrawal from treatment (i.e.private school attendance) (Gupta, 2011).Instead, ITT focuses on the effect of offering a treatment to a population.Given that I have no control over how students were selected into or chose to alter their schooling choices over time, I believe ITT is a more reasonable approach to the analysis.Therefore, I focus on the change in self-control over time of students starting in kindergarten 2010, in either public or private schools, after matching on pretreatment (i.e., fall 2010) covariates and controlling for those same pretreatment covariates over time.

Descriptives
Table 1 contains the means or counts and standard deviations or percentages for pretreatment (i.e. at the first time point), where relevant, for all included variables, which are separated by the variables of interest (i.e., the private and self-control indicators) and the matching variable for both the complete data set and the imputed data set.Additionally, Table 1 includes the means or counts and standard deviations or percentages for the matched groups of the imputed data set.Overall, there was little variation across the complete, imputed, and matched data sets.Given the minimal differences between the data sets, below we describe the matched public and private school's data.
There was a total of 4,970 students (2,485 in public and 2,485 in private) in the final imputed match data set.Both groups have similar reading (public mean = -.316,sd = .842;private mean = -.331,sd = .813)and math scores (public mean = -.182,sd = .843;private mean = -.198,sd = .807).Each group also has similar BMIs (public mean = 16.503,sd = 2.416; private mean = 16.197,sd = 2.160) and a similar number of members in their households (public mean = 4.828, sd = 1.445; private mean = 4.582, sd = 1.283).Parents in both groups are about 36 years old and both groups have about 51% male and 49% female children.Both groups had the responding parent work at least 35 hours (public = 51%, private = 50%) and the same percentage where the primary household language was English (73%).The public group had about 59% white children while the private group had 58% non-white and with 63% the public group's responding parent describing themselves as white and 62% for the private group.

Analysis
I evaluated how well the MatchIT package matched respondents from public and private schools on the observed variables.To evaluate how well these two groups were matched, I can evaluate the jitter plot for the propensity scores of the treatment (i.e., private) and control (i.e., public) groups.Figure 1 contains the jitter plot, which horizontally displays each matched and unmatched respondent.It is clear that the distribution of propensities among the matched private and public students is similar, providing some evidence that matched students are similar on the observed covariates.Additionally, distributions of imputed and actual data for the included variables were evaluated via density plots.I found no noticeable differences in density providing some evidence that the imputed data is at least able to accurately predict the observed data.Density plots are excluded for space concerns; however, are available upon request.
I then evaluated three multilevel models, using a model comparison approach where I sequentially compared a null (i.e., intercept-only) model against a random-intercepts model, with time varying covariates, because pre-treatment (i.e.time invariant) variables are needed after matching (Ho et al., 2011).Then I compared the random-intercepts only model and against the random-intercepts-and-slopes model, which is common for longitudinal data (Singer & Willett, 2003;Hedeker & Gibbons, 2006).For all five imputed datasets, the random-intercepts-andslopes was the best fitting model (Hoffman et al., 2007).However, to avoid the introduction of type-one error, I only present the results for the variable of interest, i.e., the interaction effect between students in private schools over time.Table 2 shows that there is not a statistically significant difference between students who began fall 2010 in private schools, relative to those in public schools, on the self-control construct from fall 2010 to spring 2012 at the .05alpha level.

Discussion
After imputing missing data and matching kindergarteners from private schools with similar kindergartners from public schools, I found no statistically significant relationship between the social and emotional development factor, self-control, from the fall of kindergarten session 2010 to the spring session 2012.Given that I found no evidence that similar students in kindergarten across public and private schools had different trajectories in self-control over time combined with the positive impact that self-control can have on students' outcomes, schools may want to continue exploring options for improving equity via improvements in social and emotional development constructs such as self-control.When combined with the mixed literature on academic outcomes for students in private school, relative to public school students (Abdulkadiroglu et al., 2015;Austin & Berends, (in press); Epple et al., 2015;Figlio, 2009), it appears that the benefits of sending students to private schools are marginal at best.
Given that I have found evidence that starting in a private school as a kindergartner has no effect on student's levels of self-control, it seems as though investments in other programming such as SEL offered to all students at kindergarten has a better chance of producing students with adequate social and emotional skill sets relative to interventions such as school vouchers for private schooling increasing the chances of enhancing equity.Additionally, SEL programming provides a cost-effective mechanism for improving student's social and emotional skills and thus potentially reduce the equity gap, if school practitioners want to improve these important skills, it may make sense to think about investing in SEL programming instead of voucher programs for private schools.
Future researchers have the opportunity for build on this study's limitations.First, because random assignment was not used, I suggest that, when researchers have the opportunity to use either a natural experiment (e.g., a lottery system) or create randomized controlled trials, they should include important social and emotional development factors, such as self-control, to better assess the impact that private schooling can have on these important outcomes.Unfortunately, because I was unable to disentangle regular public schools from magnet or charter schools, it is not possible to say that the lack of differences is due to minimal differences in selfcontrol over time between regular public schools on their own without charter or magnet and private schools.Because this study was forced to combine regular public schools with charter and magnet schools by the data collection agency, future researchers can begin to look beyond the effect of private versus regular public and evaluate regular public or private versus alternatives such as charter and magnet schools.It could be the case different types of charter and or magnet schooling results in higher social and emotional development relative to either regular public or private schools.Given that the items were not made available to the public it is not possible to understand the exact nature of self-control that the assessment is measuring.Therefore, future researchers can evaluate whether other self-control assessments (e.g.Emotionality, Activity, Sociability, Impulsivity, Temperament, Survey III) or behavioral tasks such as delayed gratification are different across time for public and private schools.
Given that this study, to our knowledge, is the first to assess the difference in social and emotional skills for students in private and public schools, it must be seen as a preliminary finding.Policies should be based on a robust set of literature.Therefore, this study's goal is to shed light on a gap in the literature and, thus, encourage future researchers looking at differences between private and public schooling options, such as voucher programs, to include social and emotional measurements, such as self-control, in their studies to better understand the relationship between private / public schooling and social and emotional development.

Figure 1 :
Figure 1: Comparing the distribution of propensity scores for matched treatment (private) and control (public) groups.

Table 1
, only the following time invariant covariates were included in the final model, which are represented by the variable X in the model: reading, math, body mass index (BMI), total persons in household, parent age, and parent employment.