By the description of your problem I think you answered your own question. The stock side yoke ends do wear down over time and this is normal. I removed mine years ago to inspect and replace the side yoke seals because one was seeping fluid. Not that bad of a job once you get the exhaust, rear spring and half shafts out of the way. When I was in there I replaced the side yoke bearings and snap rings with new, also new cover gasket, fluid and additive. You will have to have snap ring pliers with 90 degree tips that fit your new snap rings properly. This may also be a good time to replace your u-joints. IMO, It may not be a safety issue now, but remember that depositing damaging metal into the rear end from side yoke wear could cause other issues down the road.
Brian