When using graphics mods such as OptiFine, do I need to patch server or only client?

Solution 1:

You only need to patch the client, as all this does is managing how things are load and displayed.

For communication it just requests more chunks if you want to display more, there is nothing that needs to be changed server side to accomplish that. So, there is no need to patch the server.

Solution 2:

No, as long it is not a mod that affects gameplay, only the graphics. It will not affect your world, nor does the server need to be patched.

I am not certain about OptiFine as I never tried it, but it should work fine when only modded client-side.

Solution 3:

Optifine is a client only mod. Actually adding in Optifine to a server (with Forge) will throw a ClassNotFound exception.

(I did this to my pack a few days ago, wondered why it wouldn't load!)