3845060
                                                                                                                                                                                                                                                               
3845060
Delivered-To: jwboyer@gmail.com
3845060
Received: by 10.220.45.11 with SMTP id c11cs62970vcf;
3845060
        Mon, 31 Oct 2011 08:56:49 -0700 (PDT)
3845060
Received: by 10.101.15.19 with SMTP id s19mr2706064ani.103.1320076596057;
3845060
        Mon, 31 Oct 2011 08:56:36 -0700 (PDT)
3845060
Return-Path: <linux-kernel-owner@vger.kernel.org>
3845060
Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67])
3845060
        by mx.google.com with ESMTP id x8si7676575ani.27.2011.10.31.08.56.32;
3845060
        Mon, 31 Oct 2011 08:56:36 -0700 (PDT)
3845060
Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
3845060
Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mail=linux-kernel-owner@vger.kernel.org
3845060
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
3845060
	id S934545Ab1JaP4X (ORCPT <rfc822;mel.lkml@gmail.com> + 99 others);
3845060
	Mon, 31 Oct 2011 11:56:23 -0400
3845060
Received: from mx1.redhat.com ([209.132.183.28]:23653 "EHLO mx1.redhat.com"
3845060
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
3845060
	id S934538Ab1JaP4X (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
3845060
	Mon, 31 Oct 2011 11:56:23 -0400
3845060
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
3845060
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p9VFuHOO027543
3845060
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
3845060
	Mon, 31 Oct 2011 11:56:18 -0400
3845060
Received: from dhcp-26-164.brq.redhat.com (dhcp-26-164.brq.redhat.com [10.34.26.164])
3845060
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id p9VFuEK3018476;
3845060
	Mon, 31 Oct 2011 11:56:15 -0400
3845060
From:	Frantisek Hrbata <fhrbata@redhat.com>
3845060
To:	rientjes@google.com
3845060
Cc:	linux-mm@kvack.org, linux-kernel@vger.kernel.org,
3845060
	akpm@linux-foundation.org, kosaki.motohiro@jp.fujitsu.com,
3845060
	oleg@redhat.com, minchan.kim@gmail.com, stable@kernel.org,
3845060
	eteo@redhat.com, pmatouse@redhat.com
3845060
Subject: [PATCH v2] oom: fix integer overflow of points in oom_badness
3845060
Date:	Mon, 31 Oct 2011 16:56:09 +0100
3845060
Message-Id: <1320076569-23872-1-git-send-email-fhrbata@redhat.com>
3845060
In-Reply-To: <1320048865-13175-1-git-send-email-fhrbata@redhat.com>
3845060
References: <1320048865-13175-1-git-send-email-fhrbata@redhat.com>
3845060
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
3845060
Sender:	linux-kernel-owner@vger.kernel.org
3845060
Precedence: bulk
3845060
List-ID: <linux-kernel.vger.kernel.org>
3845060
X-Mailing-List:	linux-kernel@vger.kernel.org
3845060
3845060
An integer overflow will happen on 64bit archs if task's sum of rss, swapents
3845060
and nr_ptes exceeds (2^31)/1000 value. This was introduced by commit
3845060
3845060
f755a04 oom: use pte pages in OOM score
3845060
3845060
where the oom score computation was divided into several steps and it's no
3845060
longer computed as one expression in unsigned long(rss, swapents, nr_pte are
3845060
unsigned long), where the result value assigned to points(int) is in
3845060
range(1..1000). So there could be an int overflow while computing
3845060
3845060
176          points *= 1000;
3845060
3845060
and points may have negative value. Meaning the oom score for a mem hog task
3845060
will be one.
3845060
3845060
196          if (points <= 0)
3845060
197                  return 1;
3845060
3845060
For example:
3845060
[ 3366]     0  3366 35390480 24303939   5       0             0 oom01
3845060
Out of memory: Kill process 3366 (oom01) score 1 or sacrifice child
3845060
3845060
Here the oom1 process consumes more than 24303939(rss)*4096~=92GB physical
3845060
memory, but it's oom score is one.
3845060
3845060
In this situation the mem hog task is skipped and oom killer kills another and
3845060
most probably innocent task with oom score greater than one.
3845060
3845060
The points variable should be of type long instead of int to prevent the int
3845060
overflow.
3845060
3845060
Signed-off-by: Frantisek Hrbata <fhrbata@redhat.com>
3845060
---
3845060
 mm/oom_kill.c |    2 +-
3845060
 1 files changed, 1 insertions(+), 1 deletions(-)
3845060
3845060
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
3845060
index 626303b..e9a1785 100644
3845060
--- a/mm/oom_kill.c
3845060
+++ b/mm/oom_kill.c
3845060
@@ -162,7 +162,7 @@ static bool oom_unkillable_task(struct task_struct *p,
3845060
 unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *mem,
3845060
 		      const nodemask_t *nodemask, unsigned long totalpages)
3845060
 {
3845060
-	int points;
3845060
+	long points;
3845060
 
3845060
 	if (oom_unkillable_task(p, mem, nodemask))
3845060
 		return 0;
3845060
-- 
3845060
1.7.6.4
3845060
3845060
--
3845060
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
3845060
the body of a message to majordomo@vger.kernel.org
3845060
More majordomo info at  http://vger.kernel.org/majordomo-info.html
3845060
Please read the FAQ at  http://www.tux.org/lkml/