With helps from Sybase TS Team (Many thanks to Mike Meek!), this problem is resolved. The root cause of the problem was, the 32bit ASE server running on SUN Solaris was configured with 4GB of RAM for [max memory] (worked fine for several years!), but the ASE can actually only handle 3.78GB of RAM. When running a very big transaction, we hit the ceiling of configured memory and got this problem. The server's error log message seems misleading. After changes the memory configuration and RE-START the database server, the problem is gone and the DB server works fine! Cheers!
(This thread "answered")
Wenge