initially i'd say its the 20's. but you also have wheel alignment, tyre pressures and worn bushes/shocks/springs to consider.
low profile tryes (40% or less) will tend to wander as it drives over inconsistent surfaces, resulting in lots of steering correction as you go along.
if you put 17's or 18's back on (pumped to 32psi) and went down the same road, it should drive alot straighter. if that makes no difference, id be checking mechanical components
the worst ride ive ever experienced was on 285-25-20's which had lots of centre wear from underinflation. the back of the car handled alot like a car with solid rear axle and a broken watts linkage ( scary feeling!)