"allocate max shared memory" improved it slightly. I am now able to increase default data cache
to 3800MB (instead of 2300MB before). But still anything beyond that doesn't allow me to restart
ASE.
max memory: 9242880
total logical memory is now: 56000364
free memory: 3642504 KB
At least, this might indicate that it's related to the OS. Did you use 4+GB data cache in your environment before ?