Scott Sinclair curled a shot against the post for the visitors before Lars Stindl volleyed Gladbach ahead.
Andre Hahn fired off the crossbar as the hosts pressed for a second and Celtic’s Moussa Dembele drew a save from goalkeeper Yann Sommer.
Dembele was then fouled by Julian Korb, who was sent off, and the striker converted the resulting penalty.
Brendan Rodgers’ side, who face Barcelona in Glasgow on 23 November, move on to two points at the bottom of Group C, trailing Gladbach by two points and final group opponents Manchester City by five.
Though Celtic’s Champions League fate is all but sealed, Dembele’s cool penalty ensures they can still overtake Borussia in third and secure a Europa League berth.
Having missed a spot-kick in Camp Nou at a crucial stage of the group opening 7-0 defeat by Barca, the French striker showed great confidence to step up in another intimidating atmosphere and earn his side a potentially crucial point.
It was his 16th goal of the season and third in the Champions League group stage.
He has been an outstanding acquisition by Rodgers and at the age of 20 already looks an accomplished performer at the highest level.
His evening could have been even better had Sommer not kept out an earlier effort on a night when Celtic could conceivably have nicked a victory.