Like *all* DBMS's, ASE attempts to translate literals into datatypes - e.g. you couldn't even begin the above in Java without specifying types. ASE attempts based on the value - and for most numeric values without a decimal, it treats as an 'int' unless >2B. So in your case you have:
int * int /int
....but the int*int overflows int.
Simple fix
select @totalsize=(894720.0*4096/1048576)
Now, because of the .0, ASE treats the first value as a float - and following standard ANSI rules float * int = float ... /int = float...