Four House Republicans said anonymously that Paul Ryan might step down as House Speaker after the completion of the presidential election.
"Speculation is growing that Paul will not return," a senior Republican lawmaker close to Ryan told the Hill.
Though Ryan is still relatively new to the job, his popularity has dropped following his frequent criticism of GOP nominee Donald Trump, particularly after he said in October that he could no longer campaign with or defend Trump following leaked audio tapes in which Trump admitted to sexually harassing women.
"I'm still solidly in the camp that thinks Ryan will not run," said another legislator.
If the GOP majority drops, the 2017 floor vote to keep Ryan on as Speaker would be a much more narrow one.
"Why would he put himself in a situation where as few as 10 dissident members or one ill-timed quote from Trump could put his future in jeopardy?" the first source added.
And if Trump loses to Democratic nominee Hillary Clinton in a close race on Nov. 8, there is a strong chance that Ryan will take much of the blame for it, notes the source.
"Those who talk to Paul say he is all in to stay Speaker," said the senior lawmaker. "But if you talk to members from the South, many will struggle to vote for him -- even though they like him -- because their constituents are furious" over his tepid treatment toward Trump.
However, some sources say that Ryan's recent announcement that he has already cast his ballot could win him over more support.
"I stand where I've stood all fall and all summer," Ryan told "Fox and Friends" on Nov. 1, according to CNN. "In fact I already voted… for our nominee last week in early voting. We need to support our entire Republican ticket."
One conservative Southern lawmaker who supports Ryan told the Hill he believes Ryan will run again but will face difficulties if Clinton becomes president.
"If Trump wins, Ryan wins," said the source. "I have been hit hard in my district by folks who know Paul and I are friends. He is probably more unfavorable in my district than [former Speaker] John Boehner was."
Ryan's aides said that he is not stepping down.
"He is running," said Ryan spokeswoman AshLee Strong. "The Speaker's only focus until Election Day is defeating Democrats and protecting our majority, and nothing else."