Performance of TicketOverview

[sms]

Hi,

after a lot of searching, testing and debugging we are running out of ideas...

the problem:

it takes about 15s to display a list of about 100 tickets (around 8s for 50 tickets). We just switched to OTRS and get a lot of complaints from people (agents). Actually some people want to display all of their several hundred tickets on a single page.

the system:

OTRS 3.0.11 (apache 2.2.3) in a RH5.7 VM (vsphere5), using six 2.4 GHz cores and 8GB of RAM
mysql 5.5-18 in a separate VM on the same machine with four cores and 80GB of RAM

addons: KIX 4.0.1 and ITSM 3.0.5

Apache::DBI is enabled
we use mod_perl 2.0.4
we use StaticDB an run otrs.RebuildTicketIndex.pl every night

currently we have about 220k tickets (188k archived, 3200 open/new) with 600k articles
450 agents (only about 60-100 working at the same time)
at the moment we get only around 100 new tickets/day on average

what we know:

we do not have network issues between client and web server or between OTRS and mysql server

the DB responds quickly and and the query does not contribute significantly to the overall response time

the httpd process that answers the request for the ticket list uses about 100% CPU for most of the 15 seconds of the total response time

we traced the httpd and found that it opens and reads (and closes) the AgentTicketOverviewMedium.dtl 846 times when displaying a list of ~95 tickets (~10s between first an last call). It also stats every possible place where a AgentTicketOverviewMedium.dtl could be, e.g. in /opt/otrs/KIXCore/Kernel/Output/HTML, /usr/Custom/Kernel/Output/HTML, /opt/otrs/Kernel/cpan-lib/Kernel/Output/HTML and many more - every time it looks into AgentTicketOverviewMedium.dtl. It also sends more than 400 SELECTs to the DB with the last one sent ~11s after the first request.

Is there a way to reduce the number of system calls per ticket in a TicketOverview or otherwise optimize the process of getting the list assembled?
do we have to put /opt/otrs on a faster disk/SSD?
any other suggestions?

we would really like to see a response time of not more than 1s for a list of 50-100 tickets.

[jojo]

Can you check the speed of the disk of the VM? OTRS will built a lot of caches.

So you can have /opt/otrs on a fast disk, or built a large RAMDISK for /opt/otrs/var/tmp

Please also check why the apache is taking so much load, is it waiting IO?


Is the machine sometimes swapping? 8GB won't fit for 100 concurrent users

[sms]

disk speed seems to be ok. hdparm consistently reports more than 200 MB/s on a mostly idle system as well as during loading ticket lists in several browser tabs at the same time. Loading of the ticket lists does not seem to slow down significantly while running hdparm in a loop.

as far as I can tell the apache is not in IO wait while producing ticket lists. We will dig deeper into where httpd is spending time and I'll report back

with 40 concurrent users we have >5GB of memory free, the system is not swapping