I tried the --joystick=-1 switch and it made no difference for me. I've tried it with the ultima6 and ultima6_enh versions and both have the same problem.

I'm using the GOG version of Ultima 6 and the 9f97501 version of the master branch.