"Yet those who take in their stride the most abstruse complexities of Beethoven, the subtlest nuances of Mozart, and the most complex effects of Wagner or Mahler, will stalk angrily out of the hall when presented with, say, the enchanting simplicities of Alban Berg’s post-card lieder...It is paradoxically not what is actually to be heard that makes music difficult, but what cannot be heard because it is not there. It is the lack of something which the listener expects to hear but which is refused him that makes his blood boil, that brings the aged Philharmonic subscriber to the verge of apoplexy. Every original work represents an omission, even a deliberate erasure of what was previously indispensable to art, as well as a new ordering and new elements. The real irritant for the listener is that what he has so far considered as essential to a work of music he now cannot perceive. The composer has left it out. The appreciation of a new style is as much an effort of renunciation as of acceptance."
-from an article discussing Elliot Carter's Double Concerto (read the rest here). Probably the best explanation of the difficulty listeners have in accepting/understanding modern works as will ever be said by anyone. He passed away yesterday.