Synchro's gone. Maybe able to decrease the grind with a new/better oil.
Another thing to try if that doesn't work, and it's a little unorthodox, is;
hold steady revs on flat ground with nothing in front of you @ around 2000rpm. With your foot off the clutch (out) move the gear lever toward fourth gear slot (don't worry, it won't jump in!) and gradually increase the pressure with which you try to pull it into 4th. You'll notice the engine will actually load a little. Do it for up to 20 seconds, in increments of 5 seconds. It has the effect of "re-bedding" the synchro. Trust me, it works, bought me another 20,000k's in my GTR.
If all else fails, fix it, or put up with double clutching on the way up! {'cause we all heel and toe on the way down don't we people? I hope so! }