Question on the current behaviour of malloc () on uClinux

[Jie Zhang]

Hi,

Currently uClinux will call oom-killer if some process fails to
allocate a block of memory by malloc (), which usually will kill that
process. This behaviour can be observed by running a small test, which
repeatedly call malloc (), on uClinux. This is not expected according
to SuSv3. SuSv3 says:

  If the space cannot be allocated, a null pointer shall be returned.

So the application should be notified the failure of malloc () rather
than being killed.

I traced this issue to __alloc_pages (). If there is not enough free
pages for allocation, it will call try_to_free_pages () to reclaim
page frames. If try_to_free_pages () cannot help, which usually occurs
when run a fail-to-malloc application the second time, __alloc_pages
() will call out_of_memory (), which in turn will kill something.


Can we make __alloc_pages () not call out_of_memory () if the
allocations request comes from malloc () and just return NULL?

Thanks,
Jie