I have the same problem with my 1981. I've replaced the joystick. The mirrors worked 100% for a day, then the driver's mirror would only work up/down. The next day the passenger mirror was back to only up/down, too. Occasionally, the passenger mirror works 100% but is unpredictable. Sometimes, the driver's mirror will be turned all the way inward by itself. I can manually adjust it out, but after driving it for a while it will be back in. I don't think it is mechanically loose and vibrating inward. I'm wondering if there is a loose relay that is activating the driver's mirror in and blocking the passenger mirror from operating.
The previous owner told me that after he got it back from a detailing shop it had several electrical issues -- inop mirrors, dead radio, inop windows, inop rw defrost. After several hours & $'s searching, they found a dime in the cigarette lighter. Apparently the short blew several things. I've got all but the rw defrost and mirror issues resolved. Any suggestions? My next step will be to pull the door panels and tap directly into the mirror wiring to see that the motors are OK.