Yiou must remove the speaker, too.
I had to rebuild the window mechanism in the right door of my '84 and getting the thing out of the door after it was unbolted was the hardest part. I ended up removing the speaker so I could swing the bottom trackway upwards at the front, lift the edge of the back over the bottom of the hole in the inner panel, then swing that part downward outside of the door until the two upper mounting bolts cleared the upper edge. Of course, that's after the glass is off and everything else is unbolted and removed, including guides, stops and wiring. And don't make the same mistake I did ... replace the motor AND the nylon track ribbon. My new window motor had enough power to snap the old ribbon after just a few runs up and down, so I had to do it all over again after waiting a week for the new ribbon to arrive.