There’s a technical reason for these differences in perception between speakers and phones. Now in theory, a stereo recording is made up of a left channel, intended for the left ear, and a right channel, intended for that right ear. But when listening over monitor speakers in a room, both ears hear the sound from both speakers - it’s unavoidable. This mix of channels - the left ear also hearing the right channel, and the right ear also hearing the left - is called Inter-Aural Crosstalk. It’s why the stereo soundfield from speakers is limited to the space between the speakers, rather than extending out to the sides beyond them, in a wider arc. It also contributes to a slight loss of detail (along with other room phenomena like reflections and comb-filtering).
But in headphones, there is no Inter-Aural Crosstalk: thanks to the proximity of each earcup to the corresponding ear, the stereo signal is heard as intended - left->left ear only, right->right ear only. That’s why headphones often reveal those more subtle aspects of the mix - like background parts and subliminal effects - that are much less clear in speakers.
While this is a plus for the headphone listener, it should give pause to the mixer - if you do a mix in phones, balancing all the elements just so, it’ll sound fine for other headphone users, but when that mix is heard over speakers, a degree of necessary detail may be lost, and the mix may sound overly diffuse and muddy.
This is one of the reasons mixers mix primarily in speakers - the mix will sound proper over other speaker systems, and rather than suffering a loss of detail and clarity, it may gain some detail and clarity when heard in headphones. All the mixer needs to do is check that any differences in perception in phones are not detrimental, and make a few subtle tweaks for compatibility in both listening environments.