All heroes are a pain to fight in the right hands. However, some seem to surpass the normal levels of annoyance. Personally I feel that Mei is annoying no matter who is playing her. She has too much CC, survivability, and damage... even without her OP ultimate that builds up fast, huge area of effect that slows/stuns and damages all opponents.
However, I feel like as a "team" (aka, most people I play with) have the hardest time fighting 2 particular heroes.
Pharah and Genji. A good Genji is a pain because he is always a presence, very similar to tracer except I am personally much better at killing Tracer than I am Genji. Mostly because Genji can climb and double jump and Tracer can't. That added vertical mobility is really difficult to deal with. It means at almost anytime the Genji can get away. Similar issue with Pharah honestly.
Both Pharah and Genji have vertical superiority over other heros. This makes them difficult to hit and finish off for a majority of heros. Both characters can safely harass from range and therefore cannot be fully ignored, but also can't be dealt with by most heroes, so kind of have to be ignored.
Widowmaker got nerfed so it is no surprise that Pharah's popularity has increased. Genji has always had dedicated people playing him due to his unique high mobility assassin style.
I personally need to remember to switch to Winston as soon as these heroes start becoming a problem. Winston is fantastic vs Genji and honestly probably my best hero vs pharah as long as I attack her during her "descent" (plus the barrier is helpful for protecting team).