I suspect the decrease in performance is due to the way ASE allocates new extents to objects when doing fast bcp or creating indexes.
The allocation process uses the Global Allocation Map (GAM) structure that is stored in the sysgams system table. Sysgams isn’t a normal table. It is essentially a large bitmap with one bit per allocation unit (block of 256 pages, further broken down into 32 extents). If the bit is on, all 32 extents on the allocation unit are allocated, if the bit is off, at least one extent is free.
There are two types of extents, I call them “short” and “long”. Each allocation unit contains 32 extents. The first page of each allocation unit contains the allocation page itself, and holds information on which objects each of the 32 extents on the allocation unit are allocated too, which pages are actually in use, and which of OAM page is tracking each extent.
Because the first page is used for the allocation page, the first extent is “short”, containing only 7 pages for table data. The remaining 31 extents are all “long”, containing 8 pages for table data.
Because of internal optimizations, “Fast” bcp and Create Index can only allocate long extents. In a database that has recently experienced a lot of BCP and/or Index creation, the result can be a large number of allocation units that have all 31 long extents in use but the short extent free.
As you bcp into ASE and fill the current extent, ASE goes looking for a new free extent. It looks first on the same allocation unit the current extent was on. Then it scans the OAM looking for an allocation unit with a free extent (an OAM hint causes the search here to start where the previous search ended). If the OAM search comes up empty, the GAM is used to find an allocation unit with a free extent. In this state there are many allocation units that the GAM shows as having at least one free extent, however upon reading in the allocation page ASE finds it has only the unusable free extent free, so the next allocation unit that isn’t full is read in and the process is repeated until it finally reaches an allocation unit that has a free long extent. The GAM search always starts at the beginning of the database, and big bcp jobs need to allocate a lot of new extents, so the same all-but-full allocation pages keep getting read over and over as more extents are needed.
Typically this problem is not seen because other types of DML activity (normal inserts, expanding updates, etc.) are perfectly happy to use the short extent and as soon as they do these all-but-full allocation units become full, the GAM bit is cleared, and allocation doesn’t look at that allocation unit again (until something deallocates an extent). But the problem is seen at times when there are heavy consumers of the long extents.
A workaround for this issue is to temporarily drop the default segment from the device fragments early in the database that have already become full – the GAM search is limited to allocation units that are part of the object’s segment.
Alternatively, create a new segment on an unused device, use sp_placeobject to cause new allocations for that object to be made on the new segment. However, this is more likely to run out of space then the previous method of dropping the default segment on full devices.
You can test for this being the issue by issuing a number of DBCC STACKTRACE commands on the spids doing the bcp in, if most of the stacks show functions like pggam_getfree_lowhigh() and pg__allocate_extents() this is probably the issue.
-bret