An effective cloud computing environment requires both good performance and high efficiency of computing resources. Through extensive experiments using a representative n-tier benchmark application (Rice University Bulletin Board System (RUBBoS)), we show that the soft resource allocation (e.g., thread pool size and database connection pool size) in component servers has a significant impact on the overall system performance, especially at high system utilization scenarios. Concretely, the same software resource allocation can be a good setting in one hardware configuration and then becomes an either under- or over-allocation in a slightly different hardware configuration, causing a significant performance drop. We have also observed some interesting phenomena that were caused by the non-trivial dependencies between the soft resources of servers in different tiers. For instance, the thread pool size in an Apache web server can limit the total number of concurrent requests to the downstream servers, which surprisingly decreases the Central Processing Unit (CPU) utilization of the Clustered Java Database Connectivity (C-JDBC) clustering middleware as the workload increases. To provide a globally optimal (or near-optimal) soft resource allocation of each tier in the system, we propose a practical iterative solution approach by combining a soft resource aware queuing network model and the fine-grained measurement data of every component server. Our results show that to truly scale complex distributed systems such as n-tier web applications with expected performance in the cloud, we need to carefully manage soft resource allocation in the system.