READ: David Letterman's Most Important Moments Didn't Bring Laughter
Colbert said, "The only way I can think to properly honor Dave is that starting in September we'll do the best show we possibly can and occasionally make the network very angry at us."
Speaking of Letterman, we're counting down to his retirement on May 20th. He spoke about Julia Roberts who popped by to give him a big kiss goodbye.
READ: President Obama Jokes with David Letterman One Last Time
He said, "She's one of the biggest movie stars on the planet so you're saying to yourself, 'What the hell is she doing on your show?'"
Watch Below: Tina Fey Strips Down to Her Spanx For David Letterman