If you can read a map, read numbers and count you are all good.
I came to Japan two years ago for a holiday and knew nothing (hell I still can't even make a sentance) but made it around, saw stuff, got drunk, caught the bullet train and even got laid. All without any functional Jap (except a phrase book).
So you will be fine, living here on the other hand, well there are lots of things I would really have struggled with were it not for the assistance of my friend's Japanese wife.