2:55 a.m.
The "runq -g" option may fail on some vmcores from customers, and report
the following error:
crash> runq -g
...
malloc_bp[1998]: 11592c20
malloc_bp[1999]: 11662490
...
average size: 11922
runq: cannot allocate any more memory!
This is because the maximum number of malloc() was reached through
GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
Furthermore, the error messages is not very clear.
Given that, let's expand the limitation of MAX_MALLOC_BUFS and make the
error message clear and concise.
With the patch:
crash> runq -g
...
CPU 95
CURRENT: PID: 64281 TASK: ffff9f541b064000 COMMAND: "xxx_64281_sv"
ROOT_TASK_GROUP: ffffffffa64ff940 RT_RQ: ffff9f86bfdf3a80
[no tasks queued]
ROOT_TASK_GROUP: ffffffffa64ff940 CFS_RQ: ffff9f86bfdf38c0
[120] PID: 64281 TASK: ffff9f541b064000 COMMAND: "xxx_64281_sv"
[CURRENT]
TASK_GROUP: ffff9f47cb3b9180 CFS_RQ: ffff9f67c0417a00 <user.slice>
[120] PID: 65275 TASK: ffff9f6820208000 COMMAND: "server"
TASK_GROUP: ffff9f67f9ac2300 CFS_RQ: ffff9f6803662000 <oratfagroup>
[120] PID: 1209636 TASK: ffff9f582f25c000 COMMAND: "crsctl"
Reported-by: Buland Kumar Singh <bsingh(a)redhat.com>
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
tools.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools.c b/tools.c
index a9ad18d520d9..6676881c182a 100644
--- a/tools.c
+++ b/tools.c
@@ -5698,7 +5698,7 @@ ll_power(long long base, long long exp)
#define B32K (4)
#define SHARED_BUF_SIZES (B32K+1)
-#define MAX_MALLOC_BUFS (2000)
+#define MAX_MALLOC_BUFS (3072)
#define MAX_CACHE_SIZE (KILOBYTES(32))
struct shared_bufs {
@@ -6130,7 +6130,7 @@ getbuf(long reqsize)
dump_shared_bufs();
return ((char *)(long)
- error(FATAL, "cannot allocate any more memory!\n"));
+ error(FATAL, "cannot allocate any more memory, reached to max numbers of malloc()
via GETBUF()!\n"));
}
/*
--
2.51.1
10:50 p.m.
On 2025/11/18 17:55, Lianbo Jiang wrote:
The "runq -g" option may fail on some vmcores from
customers, and report
the following error:
crash> runq -g
...
malloc_bp[1998]: 11592c20
malloc_bp[1999]: 11662490
...
average size: 11922
runq: cannot allocate any more memory!
This is because the maximum number of malloc() was reached through
GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
Furthermore, the error messages is not very clear.
Given that, let's expand the limitation of MAX_MALLOC_BUFS and make the
error message clear and concise.
Hi Lianbo,
out of curiosity, does this mean that the cause is clear and there
is no other way to fix the issue? IOW, is there no buffer leak,
wasteful GETBUF or etc?
I'm sorry if you have already investigated them.
Generally, relaxing a limitation is the last resort, I think,
because limitations are kind of safety mechanism. Also, relaxing
the limitation may be a stopgap solution for the vmcore. If you
get another vmcore hitting this again, do you relax it again?
Thanks,
Kazu
>
> With the patch:
> crash> runq -g
> ...
> CPU 95
> CURRENT: PID: 64281 TASK: ffff9f541b064000 COMMAND: "xxx_64281_sv"
> ROOT_TASK_GROUP: ffffffffa64ff940 RT_RQ: ffff9f86bfdf3a80
> [no tasks queued]
> ROOT_TASK_GROUP: ffffffffa64ff940 CFS_RQ: ffff9f86bfdf38c0
> [120] PID: 64281 TASK: ffff9f541b064000 COMMAND: "xxx_64281_sv"
[CURRENT]
> TASK_GROUP: ffff9f47cb3b9180 CFS_RQ: ffff9f67c0417a00 <user.slice>
> [120] PID: 65275 TASK: ffff9f6820208000 COMMAND: "server"
> TASK_GROUP: ffff9f67f9ac2300 CFS_RQ: ffff9f6803662000 <oratfagroup>
> [120] PID: 1209636 TASK: ffff9f582f25c000 COMMAND: "crsctl"
>
> Reported-by: Buland Kumar Singh <bsingh(a)redhat.com>
> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
> ---
> tools.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tools.c b/tools.c
> index a9ad18d520d9..6676881c182a 100644
> --- a/tools.c
> +++ b/tools.c
> @@ -5698,7 +5698,7 @@ ll_power(long long base, long long exp)
> #define B32K (4)
>
> #define SHARED_BUF_SIZES (B32K+1)
> -#define MAX_MALLOC_BUFS (2000)
> +#define MAX_MALLOC_BUFS (3072)
> #define MAX_CACHE_SIZE (KILOBYTES(32))
>
> struct shared_bufs {
> @@ -6130,7 +6130,7 @@ getbuf(long reqsize)
> dump_shared_bufs();
>
> return ((char *)(long)
> - error(FATAL, "cannot allocate any more memory!\n"));
> + error(FATAL, "cannot allocate any more memory, reached to max numbers of
malloc() via GETBUF()!\n"));
> }
>
> /*
1:39 a.m.
On Wed, Nov 19, 2025 at 12:50 PM HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com>
wrote:
On 2025/11/18 17:55, Lianbo Jiang wrote:
> The "runq -g" option may fail on some vmcores from customers, and report
> the following error:
>
> crash> runq -g
> ...
> malloc_bp[1998]: 11592c20
> malloc_bp[1999]: 11662490
> ...
> average size: 11922
> runq: cannot allocate any more memory!
>
> This is because the maximum number of malloc() was reached through
> GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
> Furthermore, the error messages is not very clear.
>
> Given that, let's expand the limitation of MAX_MALLOC_BUFS and make the
> error message clear and concise.
Hi Lianbo,
out of curiosity, does this mean that the cause is clear and there
is no other way to fix the issue? IOW, is there no buffer leak,
wasteful GETBUF or etc?
I'm sorry if you have already investigated them.
Good questions, Kazu.
So far I haven't got the better way to fix it, the malloc_bp will be
exhausted when running the runq -g, and
I did not see the buffer leak(malloc_bp) on the specific code path(if
anybody finds it, please let me know).
Generally, relaxing a limitation is the last resort, I think,
because limitations are kind of safety mechanism. Also, relaxing
the limitation may be a stopgap solution for the vmcore. If you
Agree with you.
get another vmcore hitting this again, do you relax it again?
That needs to be considered according to the actual situation, against the
current case, if the limitation is not expanded, probably we have to tell
customers that the "runq -g" can not work because of the max limitation of
MAX_MALLOC_BUFS(2000).
BTW: for some large-scale servers equipped with multi-core(even hundreds of
cpus) running thousands of tasks, and utilizing the task group, the max
value of 2000 is really too small, therefore, it could be good to increase
it appropriately.
Thanks
Lianbo
Thanks,
Kazu
>
> With the patch:
> crash> runq -g
> ...
> CPU 95
> CURRENT: PID: 64281 TASK: ffff9f541b064000 COMMAND: "xxx_64281_sv"
> ROOT_TASK_GROUP: ffffffffa64ff940 RT_RQ: ffff9f86bfdf3a80
> [no tasks queued]
> ROOT_TASK_GROUP: ffffffffa64ff940 CFS_RQ: ffff9f86bfdf38c0
> [120] PID: 64281 TASK: ffff9f541b064000 COMMAND:
"xxx_64281_sv" [CURRENT]
> TASK_GROUP: ffff9f47cb3b9180 CFS_RQ: ffff9f67c0417a00
<user.slice>
> [120] PID: 65275 TASK: ffff9f6820208000 COMMAND: "server"
> TASK_GROUP: ffff9f67f9ac2300 CFS_RQ: ffff9f6803662000
<oratfagroup>
> [120] PID: 1209636 TASK: ffff9f582f25c000 COMMAND: "crsctl"
>
> Reported-by: Buland Kumar Singh <bsingh(a)redhat.com>
> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
> ---
> tools.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tools.c b/tools.c
> index a9ad18d520d9..6676881c182a 100644
> --- a/tools.c
> +++ b/tools.c
> @@ -5698,7 +5698,7 @@ ll_power(long long base, long long exp)
> #define B32K (4)
>
> #define SHARED_BUF_SIZES (B32K+1)
> -#define MAX_MALLOC_BUFS (2000)
> +#define MAX_MALLOC_BUFS (3072)
> #define MAX_CACHE_SIZE (KILOBYTES(32))
>
> struct shared_bufs {
> @@ -6130,7 +6130,7 @@ getbuf(long reqsize)
> dump_shared_bufs();
>
> return ((char *)(long)
> - error(FATAL, "cannot allocate any more memory!\n"));
> + error(FATAL, "cannot allocate any more memory, reached to
max numbers of malloc() via GETBUF()!\n"));
> }
>
> /*
4:26 a.m.
On 11/19/25 1:09 PM, lijiang wrote:
On Wed, Nov 19, 2025 at 12:50 PM HAGIO KAZUHITO(萩尾 一仁)
<k-hagio-ab(a)nec.com>
wrote:
> On 2025/11/18 17:55, Lianbo Jiang wrote:
>> The "runq -g" option may fail on some vmcores from customers, and
report
>> the following error:
>>
>> crash> runq -g
>> ...
>> malloc_bp[1998]: 11592c20
>> malloc_bp[1999]: 11662490
>> ...
>> average size: 11922
>> runq: cannot allocate any more memory!
>>
>> This is because the maximum number of malloc() was reached through
>> GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
>> Furthermore, the error messages is not very clear.
>>
>> Given that, let's expand the limitation of MAX_MALLOC_BUFS and make the
>> error message clear and concise.
Hi Lijiang,
I would like to propose another method to go about this problem.
What if we keep the default value as 2000, and introduce a command line
option
to `recommend` user to increase the max limit.
Regards.
~Shivang.
>
>
> --
> Crash-utility mailing list -- devel(a)lists.crash-utility.osci.io
> To unsubscribe send an email to devel-leave(a)lists.crash-utility.osci.io
> https://${domain_name}/admin/lists/devel.lists.crash-utility.osci.io/
> Contribution Guidelines: https://github.com/crash-utility/crash/wiki
4:53 a.m.
New subject: [PATCH] make the MAX_MALLOC_BUFS customizable
default value is kept to 2000, but can be changed with
command line with flag `--max-malloc-bufs`.
Signed-off-by: Shivang Upadhyay <shivangu(a)linux.ibm.com>
---
defs.h | 1 +
main.c | 4 ++++
tools.c | 9 ++++++---
3 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/defs.h b/defs.h
index ab4aee8..863baf9 100644
--- a/defs.h
+++ b/defs.h
@@ -5608,6 +5608,7 @@ void exec_args_input_file(struct command_table_entry *, struct
args_input_file *
/*
* tools.c
*/
+extern int MAX_MALLOC_BUFS;
FILE *set_error(char *);
int __error(int, char *, ...);
#define error __error /* avoid conflict with gdb error() */
diff --git a/main.c b/main.c
index 71bcc15..e986792 100644
--- a/main.c
+++ b/main.c
@@ -46,6 +46,7 @@ static struct option long_options[] = {
{"version", 0, 0, 0},
{"buildinfo", 0, 0, 0},
{"cpus", required_argument, 0, 0},
+ {"max-malloc-bufs", required_argument, 0, 0},
{"no_ikconfig", 0, 0, 0},
{"hyper", 0, 0, 0},
{"p2m_mfn", required_argument, 0, 0},
@@ -163,6 +164,9 @@ main(int argc, char **argv)
else if (STREQ(long_options[option_index].name, "cpus"))
kt->cpus_override = optarg;
+ else if (STREQ(long_options[option_index].name, "max-malloc-bufs"))
+ MAX_MALLOC_BUFS = atoi(optarg);
+
else if (STREQ(long_options[option_index].name, "hyper"))
pc->flags |= XEN_HYPER;
diff --git a/tools.c b/tools.c
index a9ad18d..c9867e1 100644
--- a/tools.c
+++ b/tools.c
@@ -5698,7 +5698,7 @@ ll_power(long long base, long long exp)
#define B32K (4)
#define SHARED_BUF_SIZES (B32K+1)
-#define MAX_MALLOC_BUFS (2000)
+int MAX_MALLOC_BUFS = 2000; /* can be changed from command line args */
#define MAX_CACHE_SIZE (KILOBYTES(32))
struct shared_bufs {
@@ -5723,7 +5723,7 @@ struct shared_bufs {
long buf_8K_ovf;
long buf_32K_ovf;
int buf_inuse[SHARED_BUF_SIZES];
- char *malloc_bp[MAX_MALLOC_BUFS];
+ char **malloc_bp;
long smallest;
long largest;
long embedded;
@@ -5744,6 +5744,7 @@ buf_init(void)
bp->smallest = 0x7fffffff;
bp->total = 0.0;
+ bp->malloc_bp = (char**) calloc(MAX_MALLOC_BUFS * sizeof(char*), 1);
#ifdef VALGRIND
VALGRIND_MAKE_MEM_NOACCESS(&bp->buf_1K, sizeof(bp->buf_1K));
@@ -6130,7 +6131,9 @@ getbuf(long reqsize)
dump_shared_bufs();
return ((char *)(long)
- error(FATAL, "cannot allocate any more memory!\n"));
+ error(FATAL, "cannot allocate any more memory!\n"
+ "try increasing --max-malloc-bufs (current value : %d)\n",
+ MAX_MALLOC_BUFS));
}
/*
--
2.51.0
3:08 a.m.
On Wed, Nov 19, 2025 at 6:27 PM shivang upadhyay <shivangu(a)linux.ibm.com>
wrote:
On 11/19/25 1:09 PM, lijiang wrote:
> On Wed, Nov 19, 2025 at 12:50 PM HAGIO KAZUHITO(萩尾 一仁) <
k-hagio-ab(a)nec.com>
> wrote:
>
>> On 2025/11/18 17:55, Lianbo Jiang wrote:
>>> The "runq -g" option may fail on some vmcores from customers, and
report
>>> the following error:
>>>
>>> crash> runq -g
>>> ...
>>> malloc_bp[1998]: 11592c20
>>> malloc_bp[1999]: 11662490
>>> ...
>>> average size: 11922
>>> runq: cannot allocate any more memory!
>>>
>>> This is because the maximum number of malloc() was reached through
>>> GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
>>> Furthermore, the error messages is not very clear.
>>>
>>> Given that, let's expand the limitation of MAX_MALLOC_BUFS and make the
>>> error message clear and concise.
Hi Lijiang,
I would like to propose another method to go about this problem.
What if we keep the default value as 2000, and introduce a command line
option
to `recommend` user to increase the max limit.
Thanks for sharing your idea, Shivang.
More code changes are required, and probably many users do not know
what values are appropriate, they have to try it, again and again. E.g:
need to exit, and then loading crash with a parameter...
Lianbo
Regards.
~Shivang.
7:05 p.m.
On 2025/11/19 16:39, lijiang wrote:
On Wed, Nov 19, 2025 at 12:50 PM HAGIO KAZUHITO(萩尾 一仁)
<k-hagio-ab(a)nec.com <mailto:k-hagio-ab@nec.com>> wrote:
On 2025/11/18 17:55, Lianbo Jiang wrote:
> The "runq -g" option may fail on some vmcores from customers, and
report
> the following error:
>
> crash> runq -g
> ...
> malloc_bp[1998]: 11592c20
> malloc_bp[1999]: 11662490
> ...
> average size: 11922
> runq: cannot allocate any more memory!
>
> This is because the maximum number of malloc() was reached through
> GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
> Furthermore, the error messages is not very clear.
>
> Given that, let's expand the limitation of MAX_MALLOC_BUFS and make the
> error message clear and concise.
Hi Lianbo,
out of curiosity, does this mean that the cause is clear and there
is no other way to fix the issue? IOW, is there no buffer leak,
wasteful GETBUF or etc?
I'm sorry if you have already investigated them.
Good questions, Kazu.
So far I haven't got the better way to fix it, the malloc_bp will be exhausted when
running the runq -g, and
I did not see the buffer leak(malloc_bp) on the specific code path(if anybody finds it,
please let me know).
Generally, relaxing a limitation is the last resort, I think,
because limitations are kind of safety mechanism. Also, relaxing
the limitation may be a stopgap solution for the vmcore. If you
Agree with you.
get another vmcore hitting this again, do you relax it again?
That needs to be considered according to the actual situation, against the current case,
if the limitation is not expanded, probably we have to tell customers that the "runq
-g" can not work because of the max limitation of MAX_MALLOC_BUFS(2000).
BTW: for some large-scale servers equipped with multi-core(even hundreds of cpus) running
thousands of tasks, and utilizing the task group, the max value of 2000 is really too
small, therefore, it could be good to increase it appropriately.
Thank you for the reply, Lianbo.
Sure, if there is no better way, we need to expand the limitation.
My question was, if so, what does the number of GETBUFs grow in
proportion to in the "runq -g" option?
Also, it looks like the "runq -g" has recursive calls, I thought that
there might be GETBUFs that can be reduced.
I'm not sure which GETBUF causes the issue and this is just an example,
I found a buf which goes into a recursive call. If recursive calls with
the buf causes the issue, maybe we can reduce them.
(but this may have a trade-off between memory and speed, there is need
to check whether we can accept it, though.)
--- a/task.c
+++ b/task.c
@@ -10086,9 +10086,6 @@ dump_tasks_in_task_group_rt_rq(int depth, ulong rt_rq, int cpu)
char *rt_rq_buf, *u_prio_array;
k_prio_array = rt_rq + OFFSET(rt_rq_active);
- rt_rq_buf = GETBUF(SIZE(rt_rq));
- readmem(rt_rq, KVADDR, rt_rq_buf, SIZE(rt_rq), "rt_rq",
FAULT_ON_ERROR);
- u_prio_array = &rt_rq_buf[OFFSET(rt_rq_active)];
if (depth) {
readmem(rt_rq + OFFSET(rt_rq_tg), KVADDR,
@@ -10111,8 +10108,8 @@ dump_tasks_in_task_group_rt_rq(int depth, ulong rt_rq, int cpu)
for (i = tot = 0; i < qheads; i++) {
offset = OFFSET(rt_prio_array_queue) + (i * SIZE(list_head));
kvaddr = k_prio_array + offset;
- uvaddr = (ulong)u_prio_array + offset;
- BCOPY((char *)uvaddr, (char *)&list_head[0], sizeof(ulong)*2);
+ readmem(rt_rq + OFFSET(rt_rq_active) + offset, KVADDR, &list_head,
+ sizeof(ulong)*2, "rt_prio_array queue[]",
FAULT_ON_ERROR);
if (CRASHDEBUG(1))
fprintf(fp, "rt_prio_array[%d] @ %lx =>
%lx/%lx\n",
@@ -10169,7 +10166,6 @@ is_task:
INDENT(5 + 6 * depth);
fprintf(fp, "[no tasks queued]\n");
}
- FREEBUF(rt_rq_buf);
}
static char *
Like this, if the number of GETBUFs grow depending on some data/code
structures, there might be a way to avoid it by code work.
The crash-utility handles various vmcores, it may have a broken or
unexpected structure. The limitation can avoid a lot of malloc calls
for such unexpected data. so if a lot of GETBUFs are required, we
should check whether the code is reasonable enough first, imho.
But yes, if it's hard to change the code, it's good to change the
limitation.
Thanks,
Kazu
>
> Thanks
> Lianbo
>
>
> Thanks,
> Kazu
>
> >
> > With the patch:
> > crash> runq -g
> > ...
> > CPU 95
> > CURRENT: PID: 64281 TASK: ffff9f541b064000 COMMAND:
"xxx_64281_sv"
> > ROOT_TASK_GROUP: ffffffffa64ff940 RT_RQ: ffff9f86bfdf3a80
> > [no tasks queued]
> > ROOT_TASK_GROUP: ffffffffa64ff940 CFS_RQ: ffff9f86bfdf38c0
> > [120] PID: 64281 TASK: ffff9f541b064000 COMMAND:
"xxx_64281_sv" [CURRENT]
> > TASK_GROUP: ffff9f47cb3b9180 CFS_RQ: ffff9f67c0417a00
<user.slice>
> > [120] PID: 65275 TASK: ffff9f6820208000 COMMAND:
"server"
> > TASK_GROUP: ffff9f67f9ac2300 CFS_RQ: ffff9f6803662000
<oratfagroup>
> > [120] PID: 1209636 TASK: ffff9f582f25c000 COMMAND:
"crsctl"
> >
> > Reported-by: Buland Kumar Singh <bsingh(a)redhat.com
<mailto:bsingh@redhat.com>>
> > Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com
<mailto:lijiang@redhat.com>>
> > ---
> > tools.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/tools.c b/tools.c
> > index a9ad18d520d9..6676881c182a 100644
> > --- a/tools.c
> > +++ b/tools.c
> > @@ -5698,7 +5698,7 @@ ll_power(long long base, long long exp)
> > #define B32K (4)
> >
> > #define SHARED_BUF_SIZES (B32K+1)
> > -#define MAX_MALLOC_BUFS (2000)
> > +#define MAX_MALLOC_BUFS (3072)
> > #define MAX_CACHE_SIZE (KILOBYTES(32))
> >
> > struct shared_bufs {
> > @@ -6130,7 +6130,7 @@ getbuf(long reqsize)
> > dump_shared_bufs();
> >
> > return ((char *)(long)
> > - error(FATAL, "cannot allocate any more
memory!\n"));
> > + error(FATAL, "cannot allocate any more memory, reached
to max numbers of malloc() via GETBUF()!\n"));
> > }
> >
> > /*
>
1:44 a.m.
On Thu, Nov 20, 2025 at 9:05 AM HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com wrote:
I'm not sure which GETBUF causes the issue and this is just
an example,
> I found a buf which goes into a recursive call. If recursive calls with
> the buf causes the issue, maybe we can reduce them.
Thanks for sharing your thoughts. I will look at it later.
> (but this may have a trade-off between memory and speed, there is need
> to check whether we can accept it, though.)
> --- a/task.c
> +++ b/task.c
> @@ -10086,9 +10086,6 @@ dump_tasks_in_task_group_rt_rq(int depth, ulong
> rt_rq, int cpu)
> char *rt_rq_buf, *u_prio_array;
> k_prio_array = rt_rq + OFFSET(rt_rq_active);
> - rt_rq_buf = GETBUF(SIZE(rt_rq));
> - readmem(rt_rq, KVADDR, rt_rq_buf, SIZE(rt_rq), "rt_rq",
> FAULT_ON_ERROR);
> - u_prio_array = &rt_rq_buf[OFFSET(rt_rq_active)];
> if (depth) {
> readmem(rt_rq + OFFSET(rt_rq_tg), KVADDR,
> @@ -10111,8 +10108,8 @@ dump_tasks_in_task_group_rt_rq(int depth, ulong
> rt_rq, int cpu)
> for (i = tot = 0; i < qheads; i++) {
> offset = OFFSET(rt_prio_array_queue) + (i *
> SIZE(list_head));
> kvaddr = k_prio_array + offset;
> - uvaddr = (ulong)u_prio_array + offset;
> - BCOPY((char *)uvaddr, (char *)&list_head[0],
> sizeof(ulong)*2);
> + readmem(rt_rq + OFFSET(rt_rq_active) + offset, KVADDR,
> &list_head,
> + sizeof(ulong)*2, "rt_prio_array queue[]",
> FAULT_ON_ERROR);
> if (CRASHDEBUG(1))
> fprintf(fp, "rt_prio_array[%d] @ %lx = > %lx/%lx\n",
> @@ -10169,7 +10166,6 @@ is_task:
> INDENT(5 + 6 * depth);
> fprintf(fp, "[no tasks queued]\n");
> }
> - FREEBUF(rt_rq_buf);
> }
> static char *
> Like this, if the number of
GETBUFs grow depending on some data/code
> structures, there might be a way to avoid it by code work.
> The crash-utility handles various vmcores, it may have a
broken or
> unexpected structure. The limitation can avoid a lot of malloc calls
> for such unexpected data. so if a lot of GETBUFs are required, we
> should check whether the code is reasonable enough first, imho.
> But yes, if it's hard to change the code, it's good to change the
> limitation.
Thanks,
> Kazu
> > > Thanks
> > Lianbo
> > > > Thanks,
> > Kazu
> > > > >
> With the patch:
> > > crash> runq -g
> > > ...
> > > CPU 95
> > > CURRENT: PID: 64281 TASK: ffff9f541b064000 COMMAND:
> "xxx_64281_sv"
> > > ROOT_TASK_GROUP: ffffffffa64ff940 RT_RQ: ffff9f86bfdf3a80
> > > [no tasks queued]
> > > ROOT_TASK_GROUP: ffffffffa64ff940 CFS_RQ: ffff9f86bfdf38c0
> > > [120] PID: 64281 TASK: ffff9f541b064000 COMMAND:
> "xxx_64281_sv" [CURRENT]
> > > TASK_GROUP: ffff9f47cb3b9180 CFS_RQ: ffff9f67c0417a00
> <user.slice > > > [120]
PID: 65275 TASK: ffff9f6820208000 COMMAND:
> "server"
> > > TASK_GROUP: ffff9f67f9ac2300 CFS_RQ: ffff9f6803662000
> <oratfagroup > > > [120]
PID: 1209636 TASK: ffff9f582f25c000 COMMAND:
> "crsctl"
> > > > > Reported-by:
Buland Kumar Singh <bsingh(a)redhat.com <mailto:
> bsingh(a)redhat.com> > > > Signed-off-by:
Lianbo Jiang <lijiang(a)redhat.com <mailto:
> lijiang(a)redhat.com> > > > ---
> > > tools.c | 4 ++--
> > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > > > > diff --git
a/tools.c b/tools.c
> > > index a9ad18d520d9..6676881c182a 100644
> > > --- a/tools.c
> > > +++ b/tools.c
> > > @@ -5698,7 +5698,7 @@ ll_power(long long base, long long exp)
> > > #define B32K (4)
> > > > > #define
SHARED_BUF_SIZES (B32K+1)
> > > -#define MAX_MALLOC_BUFS (2000)
> > > +#define MAX_MALLOC_BUFS (3072)
> > > #define MAX_CACHE_SIZE (KILOBYTES(32))
> > > > > struct
shared_bufs {
> > > @@ -6130,7 +6130,7 @@ getbuf(long reqsize)
> > > dump_shared_bufs();
> > > > > return
((char *)(long)
> > > - error(FATAL, "cannot allocate any more
memory!\n"));
> > > + error(FATAL, "cannot allocate any more memory,
> reached to max numbers of malloc() via GETBUF()!\n"));
> > > }
> > > > > /*
>
On 2025/11/19 16:39, lijiang wrote:
> On Wed, Nov 19, 2025 at 12:50 PM HAGIO KAZUHITO(萩尾 一仁) <
k-hagio-ab(a)nec.com <mailto:k-hagio-ab@nec.com>> wrote:
> On 2025/11/18 17:55, Lianbo Jiang wrote:
> > The "runq -g" option may fail on some vmcores from customers,
and
report
> > the following error:
> > > crash> runq
-g
> > ...
> > malloc_bp[1998]: 11592c20
> > malloc_bp[1999]: 11662490
> > ...
> > average size: 11922
> > runq: cannot allocate any more memory!
> > > This is because the
maximum number of malloc() was reached through
> > GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
> > Furthermore, the error messages is not very clear.
> > > Given that,
let's expand the limitation of MAX_MALLOC_BUFS and
make the
> > error message clear and concise.
> Hi Lianbo,
> out of curiosity, does this mean that the cause is
clear and there
> is no other way to fix the issue? IOW, is there no buffer leak,
> wasteful GETBUF or etc?
> I'm sorry if you have already investigated them.
> Good questions, Kazu.
> So far I haven't got the better way to fix it, the malloc_bp will be
exhausted when running the runq -g, and
> I did not see the buffer leak(malloc_bp) on the specific code path(if
anybody finds it, please let me know).
> Generally, relaxing a
limitation is the last resort, I think,
> because limitations are kind of safety mechanism. Also, relaxing
> the limitation may be a stopgap solution for the vmcore. If you
> Agree with you.
> get another vmcore hitting this again, do you relax
it again?
> That needs to be considered according to the actual
situation, against
the current case, if the limitation is not expanded, probably we have to
tell customers that the "runq -g" can not work because of the max
limitation of MAX_MALLOC_BUFS(2000).
> BTW: for some large-scale servers equipped with
multi-core(even hundreds
of cpus) running thousands of tasks, and utilizing the task group, the max
value of 2000 is really too small, therefore, it could be good to increase
it appropriately.
Thank you for the reply, Lianbo.
Sure, if there is no better way, we need to expand the limitation.
My question was, if so, what does the number of GETBUFs grow in
proportion to in the "runq -g" option?
I did not make an accurate count of that, but roughly it should be related
to the number of runq and *tasks* in the task group.
Also, it looks like the "runq -g" has recursive calls, I
thought that
You are right, Kazu. There are several recursive calls in the
dump_tasks_by_task_group().
there might be GETBUFs that can be reduced.
2:44 a.m.
On Fri, Nov 21, 2025 at 3:44 PM lijiang <lijiang(a)redhat.com> wrote:
On Thu, Nov 20, 2025 at 9:05 AM HAGIO KAZUHITO(萩尾 一仁)
<k-hagio-ab(a)nec.com>
wrote:
> On 2025/11/19 16:39, lijiang wrote:
> > On Wed, Nov 19, 2025 at 12:50 PM HAGIO KAZUHITO(萩尾 一仁) <
> k-hagio-ab(a)nec.com <mailto:k-hagio-ab@nec.com>> wrote:
> >
> > On 2025/11/18 17:55, Lianbo Jiang wrote:
> > > The "runq -g" option may fail on some vmcores from
customers,
> and report
> > > the following error:
> > >
> > > crash> runq -g
> > > ...
> > > malloc_bp[1998]: 11592c20
> > > malloc_bp[1999]: 11662490
> > > ...
> > > average size: 11922
> > > runq: cannot allocate any more memory!
> > >
> > > This is because the maximum number of malloc() was reached
> through
> > > GETBUF(), currently which is limited to MAX_MALLOC_BUFS(2000).
> > > Furthermore, the error messages is not very clear.
> > >
> > > Given that, let's expand the limitation of MAX_MALLOC_BUFS and
> make the
> > > error message clear and concise.
> >
> > Hi Lianbo,
> >
> > out of curiosity, does this mean that the cause is clear and there
> > is no other way to fix the issue? IOW, is there no buffer leak,
> > wasteful GETBUF or etc?
> > I'm sorry if you have already investigated them.
> >
> >
> > Good questions, Kazu.
> > So far I haven't got the better way to fix it, the malloc_bp will be
> exhausted when running the runq -g, and
> > I did not see the buffer leak(malloc_bp) on the specific code path(if
> anybody finds it, please let me know).
> >
> >
> > Generally, relaxing a limitation is the last resort, I think,
> > because limitations are kind of safety mechanism. Also, relaxing
> > the limitation may be a stopgap solution for the vmcore. If you
> >
> >
> > Agree with you.
> >
> > get another vmcore hitting this again, do you relax it again?
> >
> > That needs to be considered according to the actual situation, against
> the current case, if the limitation is not expanded, probably we have to
> tell customers that the "runq -g" can not work because of the max
> limitation of MAX_MALLOC_BUFS(2000).
> >
> > BTW: for some large-scale servers equipped with multi-core(even
> hundreds of cpus) running thousands of tasks, and utilizing the task group,
> the max value of 2000 is really too small, therefore, it could be good to
> increase it appropriately.
> >
>
> Thank you for the reply, Lianbo.
>
> Sure, if there is no better way, we need to expand the limitation.
> My question was, if so, what does the number of GETBUFs grow in
> proportion to in the "runq -g" option?
>
>
I did not make an accurate count of that, but roughly it should be
related to the number of runq and *tasks* in the task group.
> Also, it looks like the "runq -g" has recursive calls, I thought that
>
You are right, Kazu. There are several recursive calls in the
dump_tasks_by_task_group().
there might be GETBUFs that can be reduced.
>
I'm not sure which GETBUF causes the issue and this is just an example,
>
In the current case, it's the fill_task_group_info_array(), this is also a
recursive call, and there are a lot of memory allocation, E.g: when
the tgi_array reaches to max group num, it will expand the max group num
according to the actual task groups.
Given that, the current changes seem to be the simplest. Or is there a
better solution to fix it, any thoughts?
Thanks
Lianbo
I found a buf which goes into a recursive call. If recursive calls
with
> the buf causes the issue, maybe we can reduce them.
>
Thanks for sharing your thoughts. I will look at it later.
> (but this may have a trade-off between memory and speed, there is need
> to check whether we can accept it, though.)
>
> --- a/task.c
> +++ b/task.c
> @@ -10086,9 +10086,6 @@ dump_tasks_in_task_group_rt_rq(int depth, ulong
> rt_rq, int cpu)
> char *rt_rq_buf, *u_prio_array;
>
> k_prio_array = rt_rq + OFFSET(rt_rq_active);
> - rt_rq_buf = GETBUF(SIZE(rt_rq));
> - readmem(rt_rq, KVADDR, rt_rq_buf, SIZE(rt_rq), "rt_rq",
> FAULT_ON_ERROR);
> - u_prio_array = &rt_rq_buf[OFFSET(rt_rq_active)];
>
> if (depth) {
> readmem(rt_rq + OFFSET(rt_rq_tg), KVADDR,
> @@ -10111,8 +10108,8 @@ dump_tasks_in_task_group_rt_rq(int depth, ulong
> rt_rq, int cpu)
> for (i = tot = 0; i < qheads; i++) {
> offset = OFFSET(rt_prio_array_queue) + (i *
> SIZE(list_head));
> kvaddr = k_prio_array + offset;
> - uvaddr = (ulong)u_prio_array + offset;
> - BCOPY((char *)uvaddr, (char *)&list_head[0],
> sizeof(ulong)*2);
> + readmem(rt_rq + OFFSET(rt_rq_active) + offset, KVADDR,
> &list_head,
> + sizeof(ulong)*2, "rt_prio_array queue[]",
> FAULT_ON_ERROR);
>
> if (CRASHDEBUG(1))
> fprintf(fp, "rt_prio_array[%d] @ %lx =>
> %lx/%lx\n",
> @@ -10169,7 +10166,6 @@ is_task:
> INDENT(5 + 6 * depth);
> fprintf(fp, "[no tasks queued]\n");
> }
> - FREEBUF(rt_rq_buf);
> }
>
> static char *
>
>
> Like this, if the number of GETBUFs grow depending on some data/code
> structures, there might be a way to avoid it by code work.
>
> The crash-utility handles various vmcores, it may have a broken or
> unexpected structure. The limitation can avoid a lot of malloc calls
> for such unexpected data. so if a lot of GETBUFs are required, we
> should check whether the code is reasonable enough first, imho.
> But yes, if it's hard to change the code, it's good to change the
> limitation.
>
Thanks,
> Kazu
>
> >
> > Thanks
> > Lianbo
> >
> >
> > Thanks,
> > Kazu
> >
> > >
> > > With the patch:
> > > crash> runq -g
> > > ...
> > > CPU 95
> > > CURRENT: PID: 64281 TASK: ffff9f541b064000 COMMAND:
> "xxx_64281_sv"
> > > ROOT_TASK_GROUP: ffffffffa64ff940 RT_RQ: ffff9f86bfdf3a80
> > > [no tasks queued]
> > > ROOT_TASK_GROUP: ffffffffa64ff940 CFS_RQ: ffff9f86bfdf38c0
> > > [120] PID: 64281 TASK: ffff9f541b064000 COMMAND:
> "xxx_64281_sv" [CURRENT]
> > > TASK_GROUP: ffff9f47cb3b9180 CFS_RQ: ffff9f67c0417a00
> <user.slice>
> > > [120] PID: 65275 TASK: ffff9f6820208000 COMMAND:
> "server"
> > > TASK_GROUP: ffff9f67f9ac2300 CFS_RQ: ffff9f6803662000
> <oratfagroup>
> > > [120] PID: 1209636 TASK: ffff9f582f25c000 COMMAND:
> "crsctl"
> > >
> > > Reported-by: Buland Kumar Singh <bsingh(a)redhat.com <mailto:
> bsingh(a)redhat.com>>
> > > Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com <mailto:
> lijiang(a)redhat.com>>
> > > ---
> > > tools.c | 4 ++--
> > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/tools.c b/tools.c
> > > index a9ad18d520d9..6676881c182a 100644
> > > --- a/tools.c
> > > +++ b/tools.c
> > > @@ -5698,7 +5698,7 @@ ll_power(long long base, long long exp)
> > > #define B32K (4)
> > >
> > > #define SHARED_BUF_SIZES (B32K+1)
> > > -#define MAX_MALLOC_BUFS (2000)
> > > +#define MAX_MALLOC_BUFS (3072)
> > > #define MAX_CACHE_SIZE (KILOBYTES(32))
> > >
> > > struct shared_bufs {
> > > @@ -6130,7 +6130,7 @@ getbuf(long reqsize)
> > > dump_shared_bufs();
> > >
> > > return ((char *)(long)
> > > - error(FATAL, "cannot allocate any more
> memory!\n"));
> > > + error(FATAL, "cannot allocate any more memory,
> reached to max numbers of malloc() via GETBUF()!\n"));
> > > }
> > >
> > > /*
> >
41
days inactive
49
days old
devel@lists.crash-utility.osci.io
0 comments
1 participants
participants (1)
-
HAGIO KAZUHITO(萩尾 一仁) -
Lianbo Jiang -
lijiang
-
shivang upadhyay -
Shivang Upadhyay