"Where I was a year ago crosses my mind a bit. People remind me of it and I really have to sit down and think of what I've achieved and how fortunate I am to be in this position in my career."
Grace had only his second bogey of the week on the 478 yard ninth and his lead was down to three.
That was because Olesen hit back from a dropped shot at the fifth with birdies at the next two, while Anton Haig made a brilliant start to the back nine at Carnoustie to join him on 15 under.
The former Johnnie Walker Classic winner, who has battled back from a back injury that even led to him announcing his retirement from the sport last November, eagled the 12th and birdied the tenth and 14th.
Going even better at Kingsbarns was Swede Alex Noren, who had an eagle and four birdies in the first seven to move into fourth spot at 14 under.
Grace was back in firm control after he birdied the long 12th and 176 yard next.
At 20 under his advantage was back up to five over Olesen and Noren, with Haig and Thailand's Thongchai Jaidee one further back.
Grace suddenly gave the chasing pack hope when he double-bogeyed the 15th, and with Olesen grabbing birdies at the 14th and 16th, the gap was suddenly down to two.