Profs Tapped To Fix School-Choice Lottery

Newly arrived in the city, parents with a 4‑year-old plan to enter the lottery for kindergarten spots at one of New Haven’s public schools next year. They know they’re rolling the dice in a game many consider rigged or broken.

They don’t like their neighborhood school. They keep hearing good reviews of the city’s magnet and charter offerings.

On the lottery application, they rank Davis Street, Mauro-Sheridan, Barnard, and Elm City Montessori as their top four picks.

In the process, they’re playing the longest possible odds.

Their kid could begin the school year seated in one of the most highly coveted spots — or stuck in a less desirable school.

Would knowing the exact odds make any difference in the parents’s choices? If they’d been warned about their slim chances, would they be more likely to swap in a backup? If they still went ahead with a big gamble, should the lottery administrators take that as a signal of how deeply they care about their picks?

The school district has tapped three academics to study those very questions in an effort to make the lottery process more transparent, simple and equitable this year.

At its Monday night meeting on Meadow Street, the Board of Education’s Finance & Operations Committee recommended approving a non-financial agreement with three assistant professors — Princeton’s Adam Kapor and Christopher Neilson and University of Chicago’s Seth Zimmerman — to recommend how the district could do a better job marketing and communicating about a revamped enrollment process.

As part of the agreement, New Haven will change the algorithm that it uses to assign students in 2019 – 20. Modeled on applications in New York, Chicago and Boston, the new model will ideally reward students for listing the schools that they’d like to attend rather than for gaming the system.

“We would be digging into this year’s data to make recommendations for the following year,” said Michele Bonanno, the magnet-school coordinator who’s temporarily overseeing the Choice & Enrollment Office after Sherri Davis-Googe’s resignation. ​“That could potentially include changes to the algorithm, changes to the way that we collect applications and the choices that we allow families, the way we communicate that, how we collect feedback. All of that will be in a set of recommendations to the board to set policy.”

For the following year, in 2020 – 21, the academics will design online tools, like chat bots, that can walk families through the process, introducing them to more schools and showing them how to maximize their odds of a school they like. The contract states that the school district will approve each ​“smart marketing” and ​“application assistance” tool before it goes live on the website.

The researchers have been studying New Haven’s school system for close to a decade, starting as graduate students in economics while at Yale. In an early study, they looked at the effects of New Haven’s $1.4 billion school construction project on enrollment counts, test scores and housing prices. More recently, they turned their attention to the district’s school-choice lottery.

“They’ve been helping us think about our school-choice strategies: the way we deal with the algorithm of the lottery, to help run simulations, to do smarter marketing and to do more with online chat-bots,” Bonanno said. ​“They’ll be helping us communicate through the process, not just advertising before.”

How It Works, Or Doesn’t

In years past, New Haven has asked students to rank their top four choices.

Once all the applications are returned in April, the lottery algorithm begins by clustering every student with the same first-choice school.

Then it splits them up into groups based on whether they qualify for preferential treatment, either because they live in the neighborhood or have a sibling already in the school.

From there, the algorithm randomly assigns an order within each group and starts seating them, in order of priority, until every seat is taken. Theoretically, a class could be filled up entirely with students who have neighborhood and sibling preferences without room for anyone else.

The system then regroups any students who don’t have a spot yet by their second-choice school, on down through all four rankings.

Students who don’t get into any of their choices are assigned to the school closest to their home with available seats, while they see if they can get off the waitlist.

Faulty Assumptions

In 2015 and 2017, Kapor, Neilson and Zimmerman sent out rounds of surveys to 417 households with eighth-graders about this process. They were testing how familiar families were with the sorting mechanism and gauging how students thought about the tradeoffs between their preferences and their chances in getting in.

After matching the questionnaires against the actual lottery submissions, the researchers found that students were indeed strategizing about which schools to prioritize, yet their gaming was often based on faulty assumptions.

Students’ understanding of the algorithm was just as bad as if they’d randomly guessed at answers.

The most common pitfall families run into is how to play their preferential status. Often, they depend on it as a backup to guarantee their lower picks. But if it’s a highly competitive school, all the slots might be taken up before they can even use their special priority.

“In school choice settings, where a group of mostly lower-income households submit application portfolios at most a handful of times in their lives, belief errors are relatively large,” Kapor, Neilson and Zimmerman wrote.

Parents have long been clamoring for statistics around the lottery process, saying for years that they were going into the lottery without knowing their chances of being accepted. In 2013, the district first released full data on the number of students who applied and were accepted to each school, the number of students who had neighborhood or sibling preference for each school, and the number of students accepted to each school that labeled it their first, second, third and fourth choice.

Then, after worrying that parents were picking schools simply because they had low acceptance rates, the district reversed course and stopped releasing full stats. Last year, the magnet-school guide had no information about the chances of getting in to a given school.

In the paper, which was published in the American Economic Review in 2016, Kapor, Neilson and Zimmerman said that there’s currently more uninformed parents making mistakes than sophisticated gamblers making savvy bets. They concluded that ​“offering some means to learn about admissions probabilities for different portfolios” would likely be ​“welfare-improving.”

Equity

Figuring out exactly what that will look like is the subject of Kapor, Neilson and Zimmerman’s next experiment.

As part of the memorandum of understanding that the Finance & Operations Committee reviewed on Monday, the academics agreed to provide a data analyst who will assist the Choice & Enrollment Office in-house at no charge for the next two years. With additional surveys and data analysis, they’ll review how the revamped lottery changes placement outcomes, family satisfaction and student achievement.

“Throughout my transition report, we heard, ​‘What are we doing as it relates to equity in all areas?’ We’re going to look at that in how we place students,” said Superintendent Carol Birks. ​“Right now, not all parents feel like they have a real choice in the decision. That’s why we’re looking at this early and having a data analyst to see what policies might be.”