You didn't say what other tools you're using so fwiw ...
ASE uses shared memory, so to see how much shared memory the OS says ASE is using ... run 'ipcs -a' and add up all the chunks owned by your ASE instance.
Within the dataserver run 'sp_configure memory' and at the bottom of the output is the amount of free/available memory (from 'max memory') you can still allocate. Under normal circumstances 'max memory' minus the free/available memory should be pretty close to the total you got from 'ipcs -a'.
As for the 'vfork failed' ... *shrug* ... do you know what process is complaining/failing? Is the ASE generating messages about 'vfork failed'?
Because of your 'allocate max shared memory' and 'dynamic allocation on demand' settings I wouldn't be surprised if you have a lot of entries in the 'ipcs -a' output, ie, shared memory is fragmented. While you may be running out of memory (do the OS utilities show a lot of swapping?) you may also have an issue with too much fragmentation in shared memory (either the chunks left are too small to allocate to a process or you've run out of the number of shared memory segments allowed in the system). If possible you may want to consider setting 'allocate max shared memory = 1' and then bouncing the dataserver; this should reduce the number of fragments and possibly the 'vfork failed' messages. If you continue to have memory problems you may want to dial back 'max memory' some (eg, reduce by 1GB) and bounce the dataserver again. [NOTE: if 'sp_configure memory' shows less than 1GB of free/available memory then you may need to reduce some memory config settings before trying to bounce the dataserver with less memory, ymmv.]