High Load and hung httpd processes

[Concept]

Hi All,

System Info
CPU - 2 x Intel Xeon Quad-Core 2.3GHz
RAM - 8GB
OS - Redhat 5.5 ES
OTRS - 2.4.5
Apache / mod_perl


Been running OTRS on a RH5 system for many months without issue, until very recently. We have 2 particular tickets (so far) that cause this problem, but I cannot see any difference between these two tickets and the other thousands of tickets on the system. The issue arises when a user attempts to access one of the specific tickets in question. The initial template of the ticket view is shown, with the header history of the ticket, but the main body of the page is not displayed. As soon as this ticket is accessed, an httpd process is started on the server that instantly hits 100% CPU usage of one core. The page is never displayed on the client, and eventually produces an Internal Server Error. However, the httpd process remains for many, many hours, taking full usage of one core. Obviously with a fairly large number of users on the system, it does not take long for multiple attempts at accessing these 'problem tickets', with each access (or even a refresh of the page) spawning another httpd process that 'hangs' using as much CPU time as it can get, never producing the page at the client side, and taking many hours to die.

With an 8-core server running the system, and 8GB of RAM, it is surprisingly easy to create 100%CPU usage on all cores, and also maxing out memory usage. I have been digging at this for quite some time now, and cannot seem to fathom what is causing the issue, or why the two tickets in question are seemingly the only ones affected (that I've found so far). I fail to see why it would be an Apache configuration issue, as there are other websites (non-otrs) that function correctly, as does the majority of the OTRS system itself.

Any ideas at all? I can provide any further information that may aid in pinning down what the issue is.

Thanks!

[Mike_B]

Hi Concept,

It seems like you have some sort of corruption in your database? I noticed you did not mention what database you are using. If you can reproduce the issue by opening one specific ticket, please try to backup and restore your database on a separate host and see if you can replicate the issue there. If you can't, then please try if a full backup and then restore on your production server will alleviate the issue there. Of course, even if there would be a corruption in the database, OTRS should handle this more gracefully.

If you can reproduce this in some sort, please open a bug report over at http://bugs.otrs.org to see if something can be improved.

--
Mike

[Concept]

Hi Mike, Thanks for the response. I've finally had some time to test this out on our test environment OTRS System.

MySQL Database is being used.

I exported the DB from our Live system, and then imported to a blank database on our test system. The same issues are present with the same tickets. I updated OTRS to 2.4.7, just in case it made a difference, but still the issue remains. Thankfully, there have not been any reports of further tickets causing the same issue, but I do still need to get to the bottom of this, as the two (current) tickets that are affected are essentially untouchable.

I shall endeavor to gather as much information about this as I can to submit to the bug report.

[crythias]

I'm going out on a big limb here and provide a thought (short: delete the ticket record)

BACKUP DATABASE BEFORE DOING THIS (you could accidentally delete your db, so be prepared).
Seriously, back up your database. I don't want to be blamed if this doesn't work.

(don't schedule this)

From a Generic Agent:
Admin, Generic Agent
add name: delete [Add]
Ticket# (bad ticket number)
Delete -> yes
Save

If you're prepared to do it, run now. Then edit this job for the other ticket number and run it. then delete this Generic Agent.


---
If you want to see the ticket before doing this, you may want to browse otrs2.ticket from mysql command line or phpmyadmin.

[tux-]

Just in case some experiences the same problem: try disabling GnuTLS. I was experiencing exactly the same problems, which were solved by disabling GnuTLS.

[Wolfgangf]

Another reason could be: there was a bug in /opt/otrs/Kernel/System/HTMLUtils.pm
which caused the same symptoms
the solution was to download and install a new HTMLUtils.pm for your version from the source
or upgrade to the last version of this branch