Rep. Steve Scalise is dropping out of the speaker’s race after House Republicans failed to coalesce behind him in the aftermath of Kevin McCarthy’s historic ouster.

House Republicans met behind closed-doors for more than two hours Thursday afternoon, where the Majority leader urged his detractors to explain their opposition to him in front of the conference. After the meeting ended, Scalise huddled with those opposed to him in his office. And Republicans scheduled a second members-only conference meeting for Thursday evening.

But the opposition to Scalise as the next speaker only grew Thursday, with roughly 20 Republicans publicly opposing him. Scalise needs a majority of the House to be elected speaker, meaning he can only afford to lose four votes.