.17 all the way for your small game!!! That is a MEAN little rascal! Fast, flat, and deadly!
Like Cheez said, Deer hunting is open to many different calbers. Most common choice of folks would be a .270 or 30-06 as a universal gun for anything we can kill east of the Mississippi. And a .308 is an excellent caliber with plenty of pop. And if you want something that has plenty of knock down power, but not a whole lot of recoil, look at a 7mm-08. Great shooting round and will kill a deer no problem, with hardly any recoil compared to a 30-06 or .308.
And don't overlook the .280...doesnt get a lot of attention, but always a great caliber for whitetails. The .280 was a popular round years and years ago, but it kind of phased out for a while. But now it is starting to make a comeback as a popular round. My Dad used to be a gunsmith way back when he was in his 20s, so I have about one of every caliber you can imagine laying around....and my 2 favorites are my Browning A-Bolt .270 and my Winchester .308, with my Remington model 70 7mm-08 following closely behind in 3rd...all with Leupold optics