The Norwegian was two over par after three holes at Oak Hill but responded with three straight birdies and ended the day two under par.
That was three shots behind surprise overnight leader Eric Cole, who dropped back into a share of second at three under after the delayed first round concluded on Friday.
That meant Bryson DeChambeau led the second men's Major Championship of the season after his four-under-par 66 on Thursday.
"Very pleased with today, especially after that start being two over through three. Getting it to three under, making five birdies, that was really, really cool," Hovland said.
"I felt like I didn't really hit any bad shots the first three holes, but I was two over par.
"Obviously being able to bounce back with three birdies in a row, that kind of got me into a good position and then kept going there for a while as well."
Hovland, starting on the back nine, bogeyed the 10th and 12th holes but got a shot back at the next before a good bunker shot at the 14th and a wonderful tee-shot at the par-three 15th put him in position for three birdies in a row.
Two more at the second and fourth boosted his challenge before he dropped a shot at the difficult sixth.
"Obviously not the best drive on six to make a bogey there, but to play that stretch - six, seven, eight, nine - one over par is not a bad score" he added. "I'm very pleased with two under today."
Adam Scott rolled back the years to put himself in contention even as a closing double-bogey took some of the shine off a fine round.
The 2013 Masters Champion carded five birdies in a 68 to share sixth place with Hovland, Ryan Fox and Keegan Bradley when play was suspended for bad light, after morning frost had forced a delayed start.
Scott sent his tee-shot on the third right at the pin and holed the seven-foot putt, then put another beautiful approach to within five feet at the sixth to set up another birdie.
He rolled in a nerveless 14-footer at the eighth and though he gave a shot back after finding sand at the next, he went very close to an eagle at the tenth before settling for a fourth birdie of the day.
He looked like making pars the rest of the way in but in quickly fading light, an approach into a bunker to the left of the 18th green led to a double-bogey six.
Fox turned in 33 after birdies at the second, fourth and ninth, adding another at the 14th before a second bogey of the day at the 17th.
Cole made three birdies in four holes from the 12th - albeit interrupted by a bogey six at the 13th - and another three in succession from the second.
He was five under through 14 holes before hitting a tee shot safely into the sixth fairway and opting to call it a night.
On the resumption, he found the water with his opening shot of the day as he made a double bogey before he closed with three consecutive pars.
He was alongside fellow American and World Number Two Scottie Scheffler at three under after his first ever bogey-free round in a Major Championship.
Scheffler started on the back nine and birdied the 14th and fifth, both after 14-foot putts, and the eighth after a superb approach to just over three feet.
Corey Conners and Dustin Johnson were alongside him after rounds of 67.