Hi Christine,
Thank you again for the detail info. I had increased "cpu grace time" from 500 to 2000, but the problem still happens, and I can reproduce the problem on two ASE server (on Solaris with same configuration) by just running a very big transaction statement. I got the pstack data when it happened as below:
feac5798 pause ()
feee7c60 _aiopanic (feef876c, 1, 103f0, feee75dc, feef8000, feef8680) + 5c
feee38ec _aio_req_add (23076d8, feef8880, 1, d0, 2450598, 2450588) + 250
feee2088 _aiorw (10a, 1ae12000, feef8880, 1, 1055b590, 1) + 378
feee1d00 aiowrite64 (10a, 1ae12000, 12, 0, 8b899000, 1055b590) + 28
010624e4 basis_daiostart (10a, 1055b590, 223a800, 117132, fffffffe, 1055b590) + 7c
01062ea8 basis_dstart (223d400, 1055b590, 223aa04, 10a, 2fc0062, 223a800) + ec
01064b00 dstartio (1055b590, fffffffd, 438d0c0, 1055b5e4, 223d400, 105137e0) + 7c
0105dcac udstartio (1055b590, 1, 20002, fffffffd, 8891010, 1055b590) + b8
009817c8 bufawrite (90a43320, ffffffff, 1aa66dc, 7400, 220505c, ea9b880) + 230
0099b444 cm__wash_mass_lru (fd, 90a43320, 438d0c0, 16, 91bb49a0, 18e8400) + 148
00998af4 cm_cache_wash (1317d400, 1317f418, 2, 16, 0, 0) + 1a4
0097e44c bufread (1e, fd8d078, 926e4770, fd849b8, 926e4770, 0) + 844
00781360 getpage_with_validation (fd8d078, fd849b8, 4, 0, 0, 0) + 6e8
0077a7e0 apl_getnext (fd849b8, 1e0ad000, 113a58, fd8d148, fd8d078, fd8d078) + 3b0
00779ed4 getnext (fd8d078, fd849b8, 4000, 0, 77a430, 3c9400) + bc
00fadcc0 ups_deadlock_getnext (fd8d078, 0, 7000, 18f6cd8, ee4f6698, fd849b8) + c
00faf05c stat_ins_worktable (fd8d078, 1175a060, 1, a0000, a, 10b34030) + 114
00faf578 update_statistics_col (0, dd0ae20, fd8d078, 6f446d4, 6f42a14, d7fb104) + 1b8
00dd1050 _ex_cleanup (2022834, 5ecc, 14, 5c00, 7104, fd8d078)
00dd1050 _ex_cleanup (fd8cc38, 10b3328c, ffffffff, dd0ae20, 600000, 6f4a164)
00faad60 update_statistics (0, fd8cc38, 6f4a164, 600000, 0, ffffffff) + 1d8
00dd1050 _ex_cleanup (1000, 1983a54, 600000, 11bf3e08, 11bf3eb8, 10b32830)
00dd1050 _ex_cleanup (fd849b8, 0, 1, 1, 146, 1)
00d726b8 sequencer (1000, 65b48d0, 438d0c8, 4849400, 9a, 0) + 2cc
00dd1050 _ex_cleanup (8, 73f0, 6398, 81b8, 438d0c0, 7428)
00dd1050 _ex_cleanup (e, 361ff4, 0, 0, 0, 0)
Reboot the machine or re-start ASE, everything work fine. (I am running the 32bit ASE on 64bit Solaris 10). Thanks again for any idea or suggestion.
-Wenge