Troubleshooting a single applications is an art. Troubleshooting a single application with a database requires yet more skill.

When you get in a multi-tiered service oriented architecture you may have:

  • Multiple applications including web services,
  • A database server,
  • Multiple servers,
  • Third party or open source components,
  • Networking,
  • Firewalls,
  • Load balancing,
  • Web servers,
  • Operating systems,
  • Hardware, and
  • The .NET Framework

You need a very broad set of skills for this (or a team), access to a search engine and some luck.

If you are finding that your web application is calling external web services and seems to be under-performing, be sure to check your Machine.config default settings. Out of the box, IIS and .NET have settings set conservatively (presumably so your applications don't overload the hardware).

In one  recent case, after tracing through the application stack and doing numerous sub-system tests, we realized that the second tier web service calling the third tier web service was being limited to two outbound network connections.

Tuning maxconnections (and some related settings) for the hardware increased performance dramatically.

See the following two Microsoft articles for details:

Note, for many of these settings you can set overrides in the application.config file as well.