Josh Boyer 36ae5a6
                                                                                                                                                                                                                                                               
Josh Boyer 36ae5a6
Delivered-To: jwboyer@gmail.com
Josh Boyer 36ae5a6
Received: by 10.220.45.11 with SMTP id c11cs62970vcf;
Josh Boyer 36ae5a6
        Mon, 31 Oct 2011 08:56:49 -0700 (PDT)
Josh Boyer 36ae5a6
Received: by 10.101.15.19 with SMTP id s19mr2706064ani.103.1320076596057;
Josh Boyer 36ae5a6
        Mon, 31 Oct 2011 08:56:36 -0700 (PDT)
Josh Boyer 36ae5a6
Return-Path: <linux-kernel-owner@vger.kernel.org>
Josh Boyer 36ae5a6
Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67])
Josh Boyer 36ae5a6
        by mx.google.com with ESMTP id x8si7676575ani.27.2011.10.31.08.56.32;
Josh Boyer 36ae5a6
        Mon, 31 Oct 2011 08:56:36 -0700 (PDT)
Josh Boyer 36ae5a6
Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Josh Boyer 36ae5a6
Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mail=linux-kernel-owner@vger.kernel.org
Josh Boyer 36ae5a6
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
Josh Boyer 36ae5a6
	id S934545Ab1JaP4X (ORCPT <rfc822;mel.lkml@gmail.com> + 99 others);
Josh Boyer 36ae5a6
	Mon, 31 Oct 2011 11:56:23 -0400
Josh Boyer 36ae5a6
Received: from mx1.redhat.com ([209.132.183.28]:23653 "EHLO mx1.redhat.com"
Josh Boyer 36ae5a6
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
Josh Boyer 36ae5a6
	id S934538Ab1JaP4X (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
Josh Boyer 36ae5a6
	Mon, 31 Oct 2011 11:56:23 -0400
Josh Boyer 36ae5a6
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
Josh Boyer 36ae5a6
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p9VFuHOO027543
Josh Boyer 36ae5a6
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
Josh Boyer 36ae5a6
	Mon, 31 Oct 2011 11:56:18 -0400
Josh Boyer 36ae5a6
Received: from dhcp-26-164.brq.redhat.com (dhcp-26-164.brq.redhat.com [10.34.26.164])
Josh Boyer 36ae5a6
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id p9VFuEK3018476;
Josh Boyer 36ae5a6
	Mon, 31 Oct 2011 11:56:15 -0400
Josh Boyer 36ae5a6
From:	Frantisek Hrbata <fhrbata@redhat.com>
Josh Boyer 36ae5a6
To:	rientjes@google.com
Josh Boyer 36ae5a6
Cc:	linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Josh Boyer 36ae5a6
	akpm@linux-foundation.org, kosaki.motohiro@jp.fujitsu.com,
Josh Boyer 36ae5a6
	oleg@redhat.com, minchan.kim@gmail.com, stable@kernel.org,
Josh Boyer 36ae5a6
	eteo@redhat.com, pmatouse@redhat.com
Josh Boyer 36ae5a6
Subject: [PATCH v2] oom: fix integer overflow of points in oom_badness
Josh Boyer 36ae5a6
Date:	Mon, 31 Oct 2011 16:56:09 +0100
Josh Boyer 36ae5a6
Message-Id: <1320076569-23872-1-git-send-email-fhrbata@redhat.com>
Josh Boyer 36ae5a6
In-Reply-To: <1320048865-13175-1-git-send-email-fhrbata@redhat.com>
Josh Boyer 36ae5a6
References: <1320048865-13175-1-git-send-email-fhrbata@redhat.com>
Josh Boyer 36ae5a6
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
Josh Boyer 36ae5a6
Sender:	linux-kernel-owner@vger.kernel.org
Josh Boyer 36ae5a6
Precedence: bulk
Josh Boyer 36ae5a6
List-ID: <linux-kernel.vger.kernel.org>
Josh Boyer 36ae5a6
X-Mailing-List:	linux-kernel@vger.kernel.org
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
An integer overflow will happen on 64bit archs if task's sum of rss, swapents
Josh Boyer 36ae5a6
and nr_ptes exceeds (2^31)/1000 value. This was introduced by commit
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
f755a04 oom: use pte pages in OOM score
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
where the oom score computation was divided into several steps and it's no
Josh Boyer 36ae5a6
longer computed as one expression in unsigned long(rss, swapents, nr_pte are
Josh Boyer 36ae5a6
unsigned long), where the result value assigned to points(int) is in
Josh Boyer 36ae5a6
range(1..1000). So there could be an int overflow while computing
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
176          points *= 1000;
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
and points may have negative value. Meaning the oom score for a mem hog task
Josh Boyer 36ae5a6
will be one.
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
196          if (points <= 0)
Josh Boyer 36ae5a6
197                  return 1;
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
For example:
Josh Boyer 36ae5a6
[ 3366]     0  3366 35390480 24303939   5       0             0 oom01
Josh Boyer 36ae5a6
Out of memory: Kill process 3366 (oom01) score 1 or sacrifice child
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
Here the oom1 process consumes more than 24303939(rss)*4096~=92GB physical
Josh Boyer 36ae5a6
memory, but it's oom score is one.
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
In this situation the mem hog task is skipped and oom killer kills another and
Josh Boyer 36ae5a6
most probably innocent task with oom score greater than one.
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
The points variable should be of type long instead of int to prevent the int
Josh Boyer 36ae5a6
overflow.
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
Signed-off-by: Frantisek Hrbata <fhrbata@redhat.com>
Josh Boyer 36ae5a6
---
Josh Boyer 36ae5a6
 mm/oom_kill.c |    2 +-
Josh Boyer 36ae5a6
 1 files changed, 1 insertions(+), 1 deletions(-)
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
Josh Boyer 36ae5a6
index 626303b..e9a1785 100644
Josh Boyer 36ae5a6
--- a/mm/oom_kill.c
Josh Boyer 36ae5a6
+++ b/mm/oom_kill.c
Josh Boyer 36ae5a6
@@ -162,7 +162,7 @@ static bool oom_unkillable_task(struct task_struct *p,
Josh Boyer 36ae5a6
 unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *mem,
Josh Boyer 36ae5a6
 		      const nodemask_t *nodemask, unsigned long totalpages)
Josh Boyer 36ae5a6
 {
Josh Boyer 36ae5a6
-	int points;
Josh Boyer 36ae5a6
+	long points;
Josh Boyer 36ae5a6
 
Josh Boyer 36ae5a6
 	if (oom_unkillable_task(p, mem, nodemask))
Josh Boyer 36ae5a6
 		return 0;
Josh Boyer 36ae5a6
-- 
Josh Boyer 36ae5a6
1.7.6.4
Josh Boyer 36ae5a6
Josh Boyer 36ae5a6
--
Josh Boyer 36ae5a6
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Josh Boyer 36ae5a6
the body of a message to majordomo@vger.kernel.org
Josh Boyer 36ae5a6
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Josh Boyer 36ae5a6
Please read the FAQ at  http://www.tux.org/lkml/