1# bleed your brakes
2# you need to have a 1 way valve between the booster, and the plenum. You either don't have one, or it's stuffed (pretty commen), one way valve will fix your problems. But remeber, the booster can only hold so much vacume, so depending on the lenth of time your on the brakes, your pedal will get harder.
3# could be your rack, and/or pump.