"I would never have touched my son," says Larry James.
"I'm a scared, abused kid," Jesse told the interviewer.
Larry says he was heartbroken when he heard Jesse claim that he was responsible for breaking Jesse's arm when he was 7, the same age as Jesse's daughter Sunny.
"Sunny's the same age I was when my dad broke my arm," sniffled Jesse during his Nightline interview.
Larry says Jesse did break his arm, but it wasn't abuse. According to him, Jesse was going after a ball that had been kicked into a neighbor's yard and he fell.
"He dove over the fence landed on the ground and broke his arm," says Larry.
Larry says he took good care of his son after he was injured: "He neglected to say that I was right there at the fence, picked him up, lifted him over the fence and I didn't let go of him until his arm was in a cast."
Jesse also claimed on Nightline his dad gave him a black eye.
"It wasn't so much getting kicked or getting punched...it was the in between time the fear of that happening again, I was a terrorized kid," he said.
But Larry says that was from an accident on a dirt bike. "He banged himself up pretty good, got a black eye and had a couple of cuts on his arm but it was nothing serious. For sure I didn't blacken his eye."
Jesse's allegations of child abuse are now dividing his family.
"He was very aggressive with Jesse and I. He did abuse Jesse," she was quoted as saying.
But their stepmother, Janina said: "Larry was the most caring father. He loved Jesse to death."
"He was the apple of my eye. He was my life, and everything I did was for him," says Larry. "It's a tough thing for somebody you've been so close to for almost his whole life, for him to talk about me the way he does and say the horrible things about how violent and mean I was.
Larry has a tearful moment of his own, just as his famous son did on Nightline.
"I wasn't that way at all. I could never hurt my son. I don't even remember ever spanking him. My heart cries out to him and I'm so sorry for what's happening in his life."
Jesse is known as a "bad boy," with his tattoos and motorcycle outlaw persona. But his dad says he actually had a Norman Rockwell-style childhood.
"He played football, he raced bikes, we fished, we did everything together," Larry says.
Now Larry is making a poignant offer of reconciliation to the troubled son he's been on the outs with for six years.
"I'm 70 now, and I'm not going to be around a lot longer, I've got a bad heart and I'm hoping sometime before I'm gone that my son will find it in his heart to call me...I don't want an apology and I'm not going to apologize to him, I just want us to be father and son like we always were," he says.