After a lot of monitoring and reading I think we've found the issue (or at the very least we've found some solutions/workarounds).
Here are some useful links
https://scn.sap.com/thread/3746247
http://www.ndm.net/bi/pdf/Managing-DBMS-Workloads-v1.0-WP.pdf
http://wiki.scn.sap.com/wiki/display/SYBASE/Spinlocks+and+CPU+usage+in+SAP+ASE
http://scn.sap.com/thread/3658742
http://mattzhang-tech.blogspot.co.uk/2013/01/stored-procedure-processing-in-sybase.html
I agree with Simon's comment that the ELC was a "little rushed" and doesn't help in every case.
We've decided on these action ... if anyone thinks this is wrong let me know - thanks.
Assuming the monProcedureCacheMemoryUsage and monProcedureCacheModuleUsage are reporting the correct values, then we're not running short of procedure cache, but we are potentially ending up with so much fragmentation that requests for 4k or more procedure cache space are having to go to the global procedure and when many processes are doing this the spinlock contention is huge. In our case we end up with over 90% fragmentation.
So far we've worked around this by clearing out the procedure cache (dbcc proc(free_unused)) - but this seems heavy handed to me and is surely only a workaround.
We also tried dbcc proc(flush_elc) and this improved responsiveness of a server, but we don't know why this would be the case. We really need to understand in detail
how the ELC and procedure cache interact.
We're going to increase the amount of memory to the procedure cache to allow for fragmentation - although 16Gb does seeem rather high
(years ago a server of 32Gb RAM was considered large)
Additionially, we're going to increase the 'engine local cache percent' to 75% to give more allocation to the ELC which we hope will also reduce contention.
According to the manual the ELC should be allocated using the calc : proc cache size * 'engine local cache percent' / engine.
That should give us 150,00 pages but the log says "Proc header memory allocated 18891 pages for engine 1 local cache".
Haven't heard back why these figures are inconsistent ?
I think a better solution is using traceflag 753, which (if I've read this correctly) will only allocate 2k pages. This will reduce the fragmentation and means a page will always be available. If 16k is required, then 8 requests will be needed so will in theory be slower. In effect we're accepting lower average performance for much less contention (and since our machine has been un-responsive at times thats ok with us)
We've also looking at traceflag 757 which might be more aggressive in its procedure cache eviction - but that seems over the top when only half the cache is full.
Thanks again Simon.