Change of Heart? Help?

So I've been a pre-med student for my whole undergraduate career, and am due to graduate at the end of this semester. For the past year, however, I've fallen in love with physics. Although Physics I was kind of "meh" Physics II and onward has been fantastic. I've been reading a lot of Brian Greene, Carl Sagan, and Stephen Hawking. It makes me want to become a physicist, because I feel like it's my passion. I feel as if my proverbial bed has been made. All my efforts have been towards medicine and I am going to be going through the application process this summer. I was wondering if it sounds possible to self study my way to theoretical physics post medicine. I know I can't just change my whole career path in one fell swoop, but I'd love to pursue physics.