What was that app server doing?
The real reason why I wrote this tool was that once too often involved in the following discussion:
| User: | The app is slow |
| Developer: |
It runs fine on my one machine test install |
| System admin: |
My servers are idling the living daylights out of themselves, no way they're slowing this down |
| DBA: |
Database has indexes all over the place and we didn't catch a slow query in years |
| ALL: |
So it must be the network! |
And off course I turned out to be the network administrator.
Further investigation turns out to be necessary. As far as I know, no real open source killer app is around to make this case. I don't want to claim that conchart is exactly that, but at least it makes stating your case easier than having to make enormous documents with lengthy tcpdumps and explanations that nobody will ever read. Never underestimate the power of an image ;-)
Slapping the DB around
More than once I ended up in the discussion as stated above, and basically everybody was speaking the truth...
See here we have an example of the app server needing 3 seconds to answer the clients request. The reason is that it uses 8 concurrent connection to the database and issues a few 100 queries.
In once sentence: This app is broken by design!
A few factors can create or worsen this situation:
- When during low traffic hours the avg response time for a db query is 2ms, the situation might still be acceptable for the user. However when load increases slightly and the avg query response time climb to a still reasonable 6~8ms the situation might turn out unacceptable for the user.
- Our programmers didn't consider that the history list they're showing will be growing indefinitely, it used to run fine when it was deployed, but a few years later...
Slapping the LDAP around
A bigger picture of a similar crappy program problem, however this time LDAP is the victim:
The app servers listen for incomming HTTP requests on 39.19.144.173 and uses 39.19.143.199 as a DB and 39.7.2.153 as an LDAP. It shows that some HTTP request take a few seconds to finish. Without knowing a single thing on the internals of this app it is clear that the LDAP is heavily overqueried by the app, and that caching the LDAP queries will probably solve the performance problems reported by the users.
