As you already know my team works on ESRI products, which are off-the-shelf products, we have daily battles of understanding and solving performance issues. the main challenge is to understand where in our branched architecture the problem.
I already wrote about this ESRI processes and the "Process Explorer" I also called this Debugging the unknown.

I always said that we all good programmers, we all know to write (more or less) readable code, but most of us lack the understanding of how things are working at the background. The difference between a good programmer to an expert is that the experts truly know what is going on behind the scenes.

Last week Liran, talented boy from our Unix team, shows me this magic: Cacti.
Cacti is an RRDTool which can help us gather important details about the OS functionality and its background operations.
I am still not sure what we wrote their in his scripts, but I saw that it was short&easy to write new scripts.
The real cool thing is that creates an intranet site, which makes this easier to connect, it runs in front of our UNIX\Linux servers. I know how to run some cool scripts in UNIX, I also wrote some scripts using shell\bash\kshell, but this is better, you can get everything using graphs, and do not need to remember even you DB server name.
We are using Cacti to monitor our SDE server (Linux) and our DB Server(Oracle 10.2 on AIX5.3).

Two days later, we return our investment, when we had two crashes because of Virtual memory, you can easily see that in those graphs, which saves the history in MySQL DB.

Again, It won't solve all your problems, but you must understand that without such tools you will spend hours on hours when your DB fall again and again.
With such tool you can also predict errors when you are doing you exams at the development phase.

I would like to hear about more tools you know and use.