My A/C was doing the same thing, it would always blow air to defroster and some to the floor. As noted in above link I pulled the unit out thats located above the gas pedal, what a pain. Next time seat will come out also.
If I had to do it again I would add onto the vacuum hoses, especially the tan one so I could get to the unit better. This control unit houses the relays that control the vent flaps when you select them from the master panel.
I noticed when I removed the front side vacuum hose plate that there was still vacuum present which told me that the unit was not leaking and at least I was getting vacuum to this location.
The problem I would bet is the rubber tips that are attached to the housing are partly closed from years of use and not allowing the vacuum to get to the manifold that is connected inside the control unit, this would account for the delay some of you'll have experienced. These tips slide into the plate/block you will have to pry off. I used a paper clip and opened up the rubber tips, reassembled it all using a 8-32 machine nut (7mm socket) to hold the front side vacuum hose assembly on. When I started it up, presto I had air out the dash vents for first time in 2 years.
I hope this is the end to it but suspect the rubber tips/ends will close back up again, but next time I will take a razor blade and slice off the very top of each one and I'm sure this will cure the problem.
Bill