Id pay the money for the test drive.but only after i have been for a lap with the owner first would i then have my go.Just call it a security thing on both peoples parts.
If i am serious about buing a car i would most definently put a deposit down.
That said,If you dont lilke the car then get your money back off him.And if you do like it then the $500 can be taken off the total price at the completion of the purchase.