Yeah, looks like a problem!
Well, it sounds as if your fluid has broken down and it is simply not providing what you need to "do the braking" -

The heat that builds up in your drums and rotors (in certain cases the surface temperatures of which can exceed 1333 degrees fahrenheit) also causes the brake fluid to heat up. This is why DOT #3 and DOT #4 brake fluids usually have a boiling point which surpasses 401 degrees fahrenheit. DOT #3 and DOT #4 brake fluids are made from Polyalkylene Glycol Ether which, by nature, has a very strong attraction to water (brake fluid is hygroscopic). Water contaminating brake fluid makes its boiling point drop which causes bubbles to form in the brake fluid (imagine sprinkling water into a frying pan full of hot oil). Bubbles and water are more compressible than brake fluid which makes it more difficult for the master cylinder to compress the brake fluid and in turn, compress the calipers and brake drums. A sign of this may be a low, soft, or spongy brake pedal feel (caused by the the contaminating bubbles and water compressing before the brake fluid does).
Liekwise, DOT #5 brake fluid, on the other hand, is formulated from silicone. While silicone is not hygroscopic and has a higher boiling point than DOT#3 and DOT #4 brake fluid, any water that gets into the brake system will sink to the lowest point of that system (water being heavier than silicone) and may potentially find its way to the calipers, boil at 212 degrees fahrenheit and cause a vapor lock which could cause brake failure.
You need to determine, what color or how you are determining the "dirty fluid"

--- The color of the brake fluid itself, offers a clue. New, clean brake fluid usually has a golden color (unless it is DOT #5 which may be purple or yellow), while dirty old brake fluid is brown or if heavily contaminated, black. A brake fluid moisture tester can also be used to gauge the purity of the fluid in your brake system. So for the record, draining and replacing your brake fluid helps to remove contaminants (such as water and rust) and to lubricate the metal and rubber parts of your brake system (like the master cylinder, brake lines and seals, valves, wheel cylinders and caliper pistons) which may help them last longer.
HTH,
Andy
-=The Fiddler=-