re: estimated execution time
The short answer is the optimizer looks at table/index/column stats, optimization goal/options, and query complexity.
While you could try tweaking your system to get a more realistic estimated execution time, I'd opt to spend my time trying to speed up the actual compilation phase.
re: allrows_oltp == few steps backwards
Keep in mind that allrows_mix/allrows_dss enable additional features in the optimizer which in turn means longer compilation times and, unfortunately, more opportunities for the optimizer to pick a 'bad' query plan.
In OLTP enviroments I find it more effective to set allrows_oltp at the server level and then enable allrows_mix/allrows_dss (@ session- or query-level) on a case by case basis (as opposed to setting allrows_mix/allrows_dss at the server level and then having to 'tune' an excessive number of queries to use nested loop joins), ymmv.
If your environment can really benefit from (sort-)merge and/or hash joins, then by all means enable allrows_mix/allrows_dss.
I don't consider any of the allrows_* optgoals as better or worse than any other ... they all have their pros and cons ... and it's up to the DBA to determine which is 'better' for a given environment.
NOTE: I really like hash joins, as opposed to (sort-)merge joins, but still find (in OLTP environments) that nested loop joins often provide the best performance.