Maybe its just me not being an experienced enough listener to appreciate those things.
I would say this is a very normal and appropriate reaction.
I think people get nervous when confronted by music (or other art forms) that they don't immediately understand.
And with good reason.
For several hundred years the 'listening of non-musicians' has been way behind the creative output of the very best music has to offer.
It took Felix Mendelssohn to revive and bring to light the power, grace and intelligence of J.S. Bachs music which had, for nearly 100 years, been brushed aside as mere musical didacticism.
Berlioz' 'Benvenuto Cellini' was deemed unlistenable and the audience almost rioted.
Musicians claimed the music was unplayable.
Stavinsky saw the audience stamp their feet, boo, hiss and eventually fight to leave the hall when he premiered his ballet The Right of Spring.
The music of John Coltrane and Ornette Coleman was termed anti-jazz and some people came to their concerts specifically to ridicule and/or disrupt things.
Messiaen, Xenakis, Varese, Ravel, Bartok all had music that was initially received as utter crap.
All these people were eventually vindicated; there music being accepted and integrated with musical culture.
They are all now revered and acknowledged as musical masters.
The same stories occur in the history of art again and again, so I would be wary of dismissing something without putting some energy into it, living with it for a while and trying to understand it.
Of course you wouldn't waste your time with all that if it were just plain crap. And we all know what that sounds like, right? :)
Too many jazz musicians turn me off when they launch into endless meandering displays of their technical abilities.
I made a decent living out of this for a while.
I would speak to people in the audience sometimes and the majority had no idea what was going on musically, but were appreciating something about the overall sound or simply the excitement of live (improvised) music.
Most people can't follow a simple 32 bar song form unless the melody is being explicitly stated. So it goes without saying that taking a solo over 12 choruses of All the Things You Are is going to leave you cold.
Shame though. You might be missing out on some amazing music making.
As a (jazz) musician it's an interesting line to walk; you keep things spare and simple and some people will refer to your stuff as elevator music or claim you can't really play that well.
Tear it up and you are thought of as... well "endless meandering displays of their technical abilities" will do nicely.
End rant.