[olug] understanding Linux memory usage

[Christopher Cashell]

On Thu, Feb 21, 2008 at 10:23 AM, Ryan Stille <ryan at cfwebtools.com> wrote:
> Christopher Cashell wrote:
>  > Once the application actually makes use of the memory, it will start
>  > to show up in top and ps as being taken by the application.
>
>  Ok, so doesn't this make the output of "free" and the free value shown
>  in the top command *worthless*?  It may say you have 2GB free, so you
>  think that server is running fine and could handle an additional task or
>  something, but really that 2GB might be allocated to other processes
>  that just haven't used the memory yet?

Not exactly.  The output of free is telling you how much memory is
actually being used right now.  Remember, because of how memory
allocation works, and some of the quirks mentioned, a lot of that
"reserved" memory may never get used.  Not only that, but a lot of
applications have a tendency to reserve a lot more memory than they
actually need (be it for libraries that won't get used, or because
some application decides to reserve a lot, even though it may never
need it).

Also, the kernel will not allow applications to reserve more memory
(depending on overcommit settings) than is available from the whole
Virtual Memory system (real + swap).  If a free says you have 2GB
free, then you have that much real memory not currently being
utilized.  It doesn't know how much actual memory an application will
use in an hour or a week, so it can only tell you how much it's using
right now.  If you have 90% of your memory allocated though, even if
2GB of it isn't being used right now, you won't be able to run a new
program that needs 2GB (assuming there is 4GB total on the machine).

>  If this is the case, are there any better tools for watching memory
>  usage on a Linux box?  On windows, I just right click on my task bar and
>  select Task Manger, and it tells me everything I could want to know.

There are some additional tools, but nothing is going to make it
really simple, because there's no great way to deal with some of the
complexities.  For example, if your program is linked to libfoo, and
libfoo uses 1MB of RAM, you would want to have that 1MB show up in the
amount of memory your program is using.  But, now if someone else
starts up another program that uses libfoo, it will share that 1MB of
memory, because libfoo is already in memory for the system.  However,
you still need to show that second program as using that 1MB for
libfoo, because if the first program exits, the second will still be
using that memory.  libfoo shows up twice, as using 1MB for two
different programs, even though only 1MB of actual real memory is
being used.  Multiply that by dozens of libraries, and add in the
shared memory of forked/cloned processes (where the memory they use is
shared and identical until they write something to memory, at which
point Linux does a Copy On Write (COW) operation to just differentiate
that page of memory), and things just flat out get complicated.

One of the most useful things you can do is get some real data over
time (trending data).  There's lot of tools out there to pull that
kind of information and graph it for you.  Personally, I'm a big fan
of Munin (http://munin.projects.linpro.no/) for being very
lightweight, trivial to setup and install, and for pulling a pretty
good amount of information with the default plugin set.  There are
lots of options out there, though.  The more information you have
about the system resources being used by your application over time,
the better you can understand it's needs (now and in the future).

>  The sole purpose of this this server is run this java app, so I want it
>  to be able to get all the memory it can.  Also I thought there was some
>  kind of performance increase my having it grab all the memory at once,
>  instead of having it work its way up to the max amount as it needs it.

If the sole purpose for it is to run that java app, then chances are
there is very little in the way of other programs running on the box.
If that's the case, as long as the JVM's max memory is set high
enough, it will use as much memory as it needs, and is available.

The performance increase to grabbing all memory up front is hugely
overrated in my experience.  The *only* time I think it's a good idea
is if you know with absolute 100% certainty that your application is
going to immediately *use* all of the memory you're requesting.
Otherwise, memory allocation is a reasonably fast operation,
especially if it's going to be spread over anything more than a few
minutes of time.

And again, pre-allocating too much memory loses you a lot of
flexibility, and can lead to Out Of Memory situations on the box, even
though there is lots of memory that isn't being used.

> -Ryan

-- 
Christopher