Answer: When your spine has compressed to the point that you're a foot shorter than when you started. :-) Seriously, that's a personal / philosophical question. The best answer (in my opinion) is:
If you're constantly in pain, never able to enjoy your time on the trail, always anxious for the next rest stop, then your pack is too heavy for you regardless. Well, unless you're into that whole "hurt me, beat me, make me watch 'A Very Brady Christmas'" kind of pain thing.... *8-)
Note that, depending on your personal goals in the backcountry, being ABLE to carry the weight does NOT necessarily mean it isn't too heavy for you. Remember the words, "...detracts from the experience YOU seek". Perhaps you want to cover 10, 15, or even 20+ miles a day (whether fast-packing or wandering slow and easy) and still be wearing a smile at day's end, or simply want to travel your usual 5-10 miles and be able to forget that you're even carrying a pack!
If those are your goals, odds are a 50+lb pack just ain't gonna make it happen.
On the other hand, if you want to hike for half the day, set up a luxurious basecamp and eat like royalty, 50 lbs of gear may well make you very happy. Nothing wrong with that, if it's what you like doing!
The reason I'm starting this site, however, is to help folks realize just how light you (yes, YOU) can really travel and still be comfortable, safe and happy. And even if you never plan to pare down to a 15-20 lb load, nearly anyone lugging that 50 lb pack around can easily shave 10 lbs off in the blink of an eye, without noticing any loss of luxury. The principles are the same, and you can begin to explore the possibilities. Why carry 50 lbs if some of us can help you carry the exact same "comfort level" of gear that's 10 lbs lighter? More on that later....
As I write this, I'm about to take a weekend trip with some friends. Their packs will (as usual) weigh in at around 40 lbs each, mine will be around 15 pounds (including food and water). All of us will be happy, and none of us thinks our personal load is too heavy. Just very different styles of doing things. We all end up having fun. Simply seek your own level of pleasure from the experience, whatever it may be.
Rule #1 - Carry what YOU need to keep you safe and happy.
Rule #2 - Never stop investigating what you can do to lighten your load, without violating Rule #1.
So go forth, pare down as you see fit, and HAVE FUN!
(And beware the carnivorous marmots, ice weasels and snow snakes....)