"Do not ... and ..." grammar confusion

I believe the intent is (1) and (3). If (1) and (2) were intended, the more normal way to express it would be with or, not and.

Do not declare this callback function with a void return type or cast the function pointer to LPTHREAD_START_ROUTINE when creating the thread.

I don't know if there's a logical reason for it, but not usually distributes over an or conjunction, but not over and and conjunction (except in cases where the conjunction refers to a well known combination, e.g. don't drink and drive).