[PATCH] nice: fix overflow checking in int_add_no_wrap()

[Xi Wang]

In C, signed integer overflow is undefined behavior.  Many compilers
optimize away checks like `a + b < a'.

Use safe precondition testing instead.

Signed-off-by: Xi Wang <xi at mit.edu>
---

Try the simplified code below.

#include <stdlib.h>
void foo(int a, int b)
{
	int s = a + b;
	if (b >= 0) {
		if (s < a)
			__builtin_trap();
        }
}
int main(int argc, char **argv)
{
	int a = atoi(argv[1]);
	int b = atoi(argv[2]);
	foo(a, b);
}

The behavior of the resulting binary varies depending on the compiler,
since signed integer overflow is undefined.

$ gcc t.c -O2
$ ./a.out 2147483647 1
Illegal instruction (core dumped)

$ icc t.c -O2
$ ./a.out 2147483647 1

$ clang t.c -O2
$ ./a.out 2147483647 1

---
 libc/sysdeps/linux/common/nice.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/libc/sysdeps/linux/common/nice.c b/libc/sysdeps/linux/common/nice.c
index 3694db8..ed39946 100644
--- a/libc/sysdeps/linux/common/nice.c
+++ b/libc/sysdeps/linux/common/nice.c
@@ -25,15 +25,15 @@ static __inline__ _syscall1(int, __syscall_nice, int, incr)
 
 static __inline__ int int_add_no_wrap(int a, int b)
 {
-	int s = a + b;
-
 	if (b < 0) {
-		if (s > a) s = INT_MIN;
+		if (a < INT_MIN - b)
+			return INT_MIN;
 	} else {
-		if (s < a) s = INT_MAX;
+		if (a > INT_MAX - b)
+			return INT_MAX;
 	}
 
-	return s;
+	return a + b;
 }
 
 static __inline__ int __syscall_nice(int incr)
-- 
1.7.10.4