Modal jazz sounds pretty horrible to the untrained ear.
Do not worry, my ear is untrained as well.
There are some advantages to this interesting way of playing jazz though.
The most common modes that must be studied are:
Dorian is useful for minor 7 chords.
Mixolydian is useful for dominant chords. Also you can squeeze in altered chords somehow.
Lydian is useful for #11 chords. Also you can squeeze in the blues scale somehow.
Ionian is useful for major 7 chords.
Stacking fourths is the cheat code of Jazz piano. This is because when when you stack fourths on a chord tone, you get some neat effects.
Basically you hit some combination of chord tone and chord extensions for free! With quartal chords, it always fits. Some scale degrees work better than others, but not always.