Control problems in skyrim
As others said in the comments, it is more than likely your joystick. I actually had a similar problem happen with one of my Xbox 360 controllers. The problem is simply from usage. I am unsure what you could do to stop the problem, other than buy a new controller.
One other thing (if your controller has sentimental value :P) is to take it to a tech repair shop and have it fixed. Game Stop might fix it, but more and likely just replace it and say they fixed it.