Yes - my default data cache is 6GB.....but not on Win2012...which is why I think it is OS related. I am not as familiar with Windows as Unix/Linux - but in Unix/Linux, there are kernel configs that control the size of the shared memory segment as well as the number of shared memory segments. When ASE starts, it attempts to grab everything in ONE shared memory segment of X size (whatever total memory is set to). Soooo...normally, when ASE is running, in Unix, you do an ipcs -a and you will see the ASE user (Sybase typically) with 1 shared memory segment. If you sp_configure 'total memory' higher while ASE is running, it then grabs another segment - so, it would have two segments. However, when you shutdown and restart, it will attempt to grab a single segment at the now higher size. If you were on Unix, that would explain why you could increase it slightly while running, but it wouldn't restart at the higher setting.
However, on Windows, ASE will grab more than one segment....if you look in your errorlog at boot time, you should see lines similar to:
10:49:31.10 kernel Allocating a shared memory segment of size 1075904512 bytes.
10:49:31.36 kernel Allocating a shared memory segment of size 191758336 bytes.
10:49:31.37 kernel Allocating a shared memory segment of size 86245376 bytes.
10:49:31.37 kernel Allocating a shared memory segment of size 78446592 bytes.
10:49:31.37 kernel Allocating a shared memory segment of size 35586048 bytes.
10:49:31.37 kernel Allocating a shared memory segment of size 119603200 bytes.
10:49:31.38 kernel Allocating a shared memory segment of size 4295032832 bytes.
10:49:31.41 kernel Allocating a shared memory segment of size 7002324992 bytes.
10:49:31.45 kernel Kernel memory at 0x0000000020000000, 1075904512 bytes
10:49:31.45 kernel Kernel memory at 0x0000000020000000, 3536896 bytes
10:49:31.46 kernel Server part of first shared memory region at 0x000000002035F800, 1072367616 bytes.
10:49:31.46 kernel Failed to log the current message in the Windows NT event log
10:49:31.46 kernel Server region 0 at 0x0000000060220000, 191758336 bytes
10:49:31.46 kernel Server region 1 at 0x000000006B9D0000, 86245376 bytes
10:49:31.46 kernel Server region 2 at 0x0000000070CF0000, 78446592 bytes
10:49:31.46 kernel Server region 3 at 0x00000000757E0000, 35586048 bytes
10:49:31.47 kernel Server region 4 at 0x0000000077DD0000, 119603200 bytes
10:49:31.47 kernel Server region 5 at 0x000000007FFF0000, 4295032832 bytes
10:49:31.47 kernel Server region 6 at 0x0000000180030000, 7002324992 bytes
10:49:31.47 kernel Highest valid address is 0x0000000321620000
10:49:31.47 kernel Adaptive Server is using the threaded kernel mode.
...as you can see, I grabbed 8 shared memory segments with sizes from 35MB to ~7GB....likely due to fact that other apps were using memory and since shared memory is normally pinned (non-swapped), Windows had to scrounge multiple non-contiguous blocks of memory.
Windows did some funky stuff with shared memory and permissions (which is why I suggested running as Administrator)....some of which I suspect was to do with same sort of things that Linux tried adding some security around by randomizing virtual address space, etc. I am not sure if that is related to your problem except only indirectly perhaps.....
....what it looks like to me is that your OS installation (and not sure who/how it was done) is capping your shared memory segments at 4GB...or you are being capped at a set number of shared memory segments. The latter in Unix (for example) is set system wide, so if you set the OS to limit only 32 shared memory segments, then all processes on the host combined can only have 32 shared memory segments. This can get fun if running SAP installations as the NW worker processes also use shared memory....since this post is in the Custom vs. SAP area, I will assume you are not running SAP - but it is possible that other programs such as MSSQL, Oracle, etc. all will need shared memory.....
It would be interesting to see the corresponding entries in your errorlog similar to the above when ASE booted fine and also when it failed....might point us to whether it is a total MB of shared memory as the issue or the number of segments.....