ASE 12.5.4 is EOL. There is really no need to install it on Solaris 11 as others have pointed out. And it was never certified (Solaris 11.x did not exist/ASE 12.x was EOL by the time Solaris 11 was available).
If this is an application that requires 12.5.4, then you should contact the vendor to upgrade to a supported ASE release.
If this is for you to learn ASE, there are several editions available of current ASE. If your edition is fully licensed and supported, then you should be able to migrate to current. There are many resources out there on how to migrate applications to 15.x.
That being said - the issue you appear to be experiencing is most likely not related to ASE 12.5.4, but the installer. The installer is Java-based and provides its own JVM. As far as I know, the JVM the installer is using is a 32-bit JVM and uses a GUI by default. The ASE could be 32-bit or 64-bit (not mentioned) but that does not matter right now.
Check your Solaris installation and make sure it has any 32-bit support needed by a 32-bit JVM. You can also try running the installer in console mode (./setup.bin -i console) to see if that works.
As the supplied JVM is old, there is also a possibility that current Solaris 11 security is disallowing an old JVM from running.
Chris