carry/overflow & subtraction in x86
Solution 1:
Here's a reference table that might help. This shows an example of every possible combination of the 4 arithmetic flags that can result from the ADD and SUB instructions on x86. 'h' 'ud' and 'd' stand for hex, unsigned decimal and signed decimal representations of each value. For example, the first row for SUB says 0xFF - 0xFE = 0x1 with no flags set.
But, I think the short story is that Alex's answer is correct.
ADD
A B A + B Flags
--------------- ---------------- --------------- -----------------
h | ud | d | h | ud | d | h | ud | d | OF | SF | ZF | CF
---+------+-------+----+------+-------+----+------+-------+----+----+----+---
7F | 127 | 127 | 0 | 0 | 0 | 7F | 127 | 127 | 0 | 0 | 0 | 0
FF | 255 | -1 | 7F | 127 | 127 | 7E | 126 | 126 | 0 | 0 | 0 | 1
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0
FF | 255 | -1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1
FF | 255 | -1 | 0 | 0 | 0 | FF | 255 | -1 | 0 | 1 | 0 | 0
FF | 255 | -1 | FF | 255 | -1 | FE | 254 | -2 | 0 | 1 | 0 | 1
FF | 255 | -1 | 80 | 128 | -128 | 7F | 127 | 127 | 1 | 0 | 0 | 1
80 | 128 | -128 | 80 | 128 | -128 | 0 | 0 | 0 | 1 | 0 | 1 | 1
7F | 127 | 127 | 7F | 127 | 127 | FE | 254 | -2 | 1 | 1 | 0 | 0
SUB
A B A - B Flags
--------------- ---------------- --------------- -----------------
h | ud | d | h | ud | d | h | ud | d || OF | SF | ZF | CF
----+------+-------+----+------+-------+----+------+-------++----+----+----+----
FF | 255 | -1 | FE | 254 | -2 | 1 | 1 | 1 || 0 | 0 | 0 | 0
7E | 126 | 126 | FF | 255 | -1 | 7F | 127 | 127 || 0 | 0 | 0 | 1
FF | 255 | -1 | FF | 255 | -1 | 0 | 0 | 0 || 0 | 0 | 1 | 0
FF | 255 | -1 | 7F | 127 | 127 | 80 | 128 | -128 || 0 | 1 | 0 | 0
FE | 254 | -2 | FF | 255 | -1 | FF | 255 | -1 || 0 | 1 | 0 | 1
FE | 254 | -2 | 7F | 127 | 127 | 7F | 127 | 127 || 1 | 0 | 0 | 0
7F | 127 | 127 | FF | 255 | -1 | 80 | 128 | -128 || 1 | 1 | 0 | 1
Solution 2:
All 4 combinations of the carry and overflow values are possible when adding or subtracting. You can see more examples in this answer.
This answer contains a proof of the fact that the carry that you get from A-B is the inverse of the carry you get from A+(-B). The code by the first link exploits this property to turn ADC into SBB.
The signed overflow flag value, however, must be the same for both A-B and A+(-B) because it depends on whether or not the result has the correct sign bit and in both cases the sign bit will be the same.