Probably on the other end, actually. The joke is that, a lot of times, there are some very annoying people who go out of their way to make a spectacle of something they enjoy, but which would be offensive (or at least annoying) to other people. However, when someone calls them on it and tells them to knock it off, they respond with "Don't like it? Quit looking!" - as if they weren't being so obnoxiously public about it that even actively trying to avoid them is a futile act.
Most commonly, this is used to mock the in-your-face flamboyant members of the homosexual community, such as those that dress up in loud, outlandish costumes and ride around on the floats at gay pride parades. It's become something of a stereotype for such people to say "Well, if you don't like it, why do you keep looking?" (insert overblown gay lisp here) and the like.
