The Spanish great, who was being outplayed by his younger opponent, was a set and a break down when he appeared to agitate a muscle on his left hip when reaching for a ball.
Nadal played the third set practically unable to get to any ball that was directly at him as McDonald triumphed 6-4, 6-4, 7-5 in two hours 17 minutes.
It is the biggest win of the 27-year-olds career and he moves on to face either Dalibor Svrcina or Yoshihito Nishioka in round three.
The match was played indoors under the roof in Melbourne and the conditions seemed to favour World No.65 McDonald as the US star broke the Nadal serve in the very first game.
Nadal was unsettled and meekly gave up his serve again shortly after whilst 30-0 up to trail 4-1.
The Mallorca man was clearly irritated and took that frustration out on the ball in the next game as he pulled one of the breaks back.
McDonald now stood with the ball in hand and, serving for the first set at 5-4, a game that felt match-defining, Nadal forced a look at a break point after a passing forehand winner down the line at 30 all.
But McDonald – who promised pre-match to make the occasion memorable – showed he is made of string stuff, mixing big serve and solid groundstroke to take a shock lead.
The pair exchanged breaks at the start of the second set but it was McDonald who made the breakthrough in game seven, forcing Nadal wide to lead 4-3.
It was in the next game, with McDonald dominating, that Nadal pulled up and, after a medical timeout, many feared it was an early end to his Australian Open defence.
But Nadal has only retired from open play 10 times in his career and he battled on, holding serve to force the American to serve for a two-set lead.
It was worrying times for Nadal though, as three relatively routine forehand were put into the net and McDonald was two up.
Nadal was barely making short balls and only some wayward McDonald backhands kept him in the third set, getting it to 5-5 without facing much trouble.
But McDonald dominated the next Nadal service game, setting up three break points and taking it on the second one.
The man from California duly converted the next game to stun those watching in the Rod Laver Arena and progress to the next round.