I was under the impression there are situations where SQL Server will simplify a query not look at certain underlying tables if they are not useful for the query. i.e. If I LEFT OUTER JOIN to a table but then don't include any columns in the results, or where clause, or any further joins, SQL Server can choose to exclude that table from the query.
However, if you've got CASE around each column, I'm guessing it's not able to simplify in this way.