The 85-year-old told Larry King Thursday that he believes Ripa took it to heart when he decided to retire in 2011.
“She thought I was leaving because of her,” Philbin said. "I was leaving because I was getting older and that wasn't right for me anymore."
King pressed: “You mean she took it personal?”
“Yeah, I think so,” the former morning show host said.
He added that he hasn't kept in touch with Ripa since his departure, and has never received an invitation to return to the show.
"Never once did they ask me to go back," he told King.
But that does not appear to be the case, as Philbin, whose image appears on a plaque outside the show's New York City studio, did return to Live! for its Halloween episode in 2015.
The ABC network also took issue with the fact that he has never been invited back.
"It was wonderful to have Regis on the show for our 2015 Halloween special," ABC told Inside Edition in a statement. "He's also been invited back several times as a guest, and in fact was confirmed for a date, but was not able to make it at the time."
Philbin co-hosted Live! for 28 years and Ripa joined in 2000 following the departure of his original co-host, Kathie Lee Gifford.