I have the same thing (at least from what you've described)
In my case at least, you will find that the transmission - from clutch to diff - is not absulutely tight. There is an amount of free play in the drive shaft. When the wheels are being driven (there is load on the wheels from the engine turning them) the free play is being taken up in one direction. When you depress the clutch, the engine stops driving the wheels, causing the wheels to "drive" the gearbox, in turn taking up the free play in the other direction. This change can cause a small clunk sound from the diff, which is where the "looseness" more than likely lies.
The sound will be much louder depending on how loose the diff/gearbox is. If you can get under the car, turn the drive shaft with you hands backward and forward, you will see what I mean.
Not saying this is your problem, but it's something to consider