You're right. But beinging courteous is a very good rule of thumb.
Example

s)
:If you catch up to a slow group on any given tee box, politely ask if you can play through. If it looks like their uncomfortable with your request, just add that you've got: a wedding to attend; an angry wife you've got to appease if I want to ever play again; a meeting at __; be creative and keep it light!
: discreetly talk to a marshall out of ear shot of group
: discreetly call the clubhouse
: work on your game a little (especially if it's not the group directly ahead of you)
: leave and complain to clubhouse on your way home (they know when you teed off and they may even give you a complementary round)
BE NICE...Try to enjoy your day.
p.s.
If a group does let you play through, MOVE IT. Nobody likes to watch someone go through their entire "routine"(s) and be put in another groups slow play books. Hit your ball go after it and pick it up if you duff it (or don't bother look for it if it's out of bounds) Remember, you want them to let a group through any other round. If they experience an inconsiderate group of golfers, they'll never do it again.