Mick Dunlop’s early opener for Rovers was cancelled out by Jamie Hamill before Ryan Wallace was sent off with three minutes left.
But the Cliftonhill outfit kept pushing with Ferguson hitting the late reward for his plucky team-mates.
And he beamed: “It’s just so emotional. When the goal went in, before I knew it I was in the middle of the park.
“Thirty seconds before the winning goal we were shouting at the linesman asking for time up and then we end up on the counter-attack.
“We limited them in terms of clear-cut chances and, fortunately, we took the chances when they fell to us.
“I’m delighted for the club and the fans. Drawing Celtic is a great tie for the players. The effort they put in was unbelievable.
“I could not ask for any more from them. Even when we went down to 10 men we were still pushing on.
“It’s a great opportunity for us now and massive for the club financially. Hopefully, the telly covers it.
Rovers, who reached the quarter-finals of the Cup three seasons ago, took the lead in the 16th minute when Dunlop met Scott McBride’s corner with a bullet header.
The Dumfries outfit believed they had forced a replay at Palmerston when substitute Hamill turned Stephen Dobbie’s corner into the net in the 76th minute.
Wallace then saw red for two bookings in quick succession only for Rovers to seal their place in the next round thanks to Ferguson’s close-range finish.
Queens caretaker boss Jim Thomson said: “This is the worst night of my career. The mood in the dressing room is terrible. We’re gutted.