OTRS Crashing

[retro2707]

Hi all,

We are having an issue over the last month where the web access will grind to a halt every week or so. When it happens, CPU and Memory usage spike with the /opt/otrs/bin/c processes each taking up 5% or so of the CPU and remote access to the server becomes very slow.

The fix for this is to restart the HTTPD service on the CentOS server. Prior to this the server has been stable for 3+ months without issue.

Once HTTPD is restarted, everything is fine for another week.

I updated to the latest OTRS/Kix 3.x release around a month ago so I dont know if its due to that.

See below for the error generated in the httpd error log. I have thousands of these in the logs when the issue occurs.

Code: Select all

[Tue Mar 10 10:07:21 2015] -e: DBD::mysql::st execute failed: MySQL server has gone away at /opt/otrs/KIX4OTRS/Kernel/System/DB.pm line 650.
[Tue Mar 10 10:07:21 2015] -e: Version string '-1, set by base.pm' contains invalid data; ignoring: '-1, set by base.pm' at /opt/otrs//Kernel/System/Log.pm line 203.
ERROR: OTRS-CGI-01 Perl: 5.10.1 OS: linux Time: Tue Mar 10 10:07:21 2015

 Message: MySQL server has gone away, SQL: 'SELECT id, value_text, value_date, value_int, field_id
            FROM dynamic_field_value
            WHERE object_id = ?
            ORDER BY id'

 RemoteAddress: 10.36.165.10
 RequestURI: /otrs/index.pl?Action=AgentTicketEscalationView

 Traceback (29813): 
   Module: Kernel::System::DynamicFieldValue::ValueGet (OTRS 3.3.11) Line: 275
   Module: Kernel::System::DynamicField::Driver::BaseSelect::ValueGet (OTRS 3.3.11) Line: 38
   Module: Kernel::System::DynamicField::Backend::ValueGet (OTRS 3.3.11) Line: 850
   Module: Kernel::System::Ticket::TicketGet (OTRS 3.3.11) Line: 6005
   Module: Kernel::System::Ticket::TicketAcl (OTRS 3.3.11) Line: 3151
   Module: Kernel::Output::HTML::TicketOverviewSmall::Run (OTRS 3.3.11) Line: 536
   Module: Kernel::Output::HTML::LayoutTicket::TicketListShow (OTRS 3.3.11) Line: 2184
   Module: Kernel::Modules::AgentTicketEscalationView::Run (OTRS 3.3.11) Line: 405
   Module: Kernel::System::Web::InterfaceAgent::Run (OTRS 3.3.11) Line: 910
   Module: ModPerl::ROOT::ModPerl::Registry::opt_otrs_bin_cgi_2dbin_index_2epl::handler (unknown version) Line: 56
   Module: (eval) (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::run (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::default_handler (v1.99) Line: 170
   Module: ModPerl::Registry::handler (v1.99) Line: 31
   

and

Code: Select all

[Tue Mar 10 10:08:01 2015] -e: DBD::mysql::st execute failed: MySQL server has gone away at /opt/otrs/KIX4OTRS/Kernel/System/DB.pm line 650.
ERROR: OTRS-CGI-01 Perl: 5.10.1 OS: linux Time: Tue Mar 10 10:08:01 2015

 Message: MySQL server has gone away, SQL: 'SELECT id, name, calendar_name, first_response_time, first_response_notify, update_time, update_notify, solution_time, solution_notify, valid_id, comments, create_time, create_by, change_time, change_by , type_id, min_time_bet_incidents FROM sla WHERE id = ? LIMIT 1'

 RemoteAddress: 10.36.150.7
 RequestURI: /otrs/index.pl?Action=AgentTicketEscalationView;SortBy=EscalationTime;OrderBy=Up;View=;Filter=NextWeek

 Traceback (29716): 
   Module: Kernel::System::SLA::SLAGet (OTRS 3.3.11) Line: 272
   Module: Kernel::System::Ticket::TicketEscalationPreferences (OTRS 3.3.11) Line: 2353
   Module: Kernel::System::Ticket::TicketEscalationDateCalculation (OTRS 3.3.11) Line: 2436
   Module: Kernel::System::Ticket::TicketGet (OTRS 3.3.11) Line: 6073
   Module: Kernel::Output::HTML::TicketOverviewSmall::Run (OTRS 3.3.11) Line: 487
   Module: Kernel::Output::HTML::LayoutTicket::TicketListShow (OTRS 3.3.11) Line: 2184
   Module: Kernel::Modules::AgentTicketEscalationView::Run (OTRS 3.3.11) Line: 405
   Module: Kernel::System::Web::InterfaceAgent::Run (OTRS 3.3.11) Line: 910
   Module: ModPerl::ROOT::ModPerl::Registry::opt_otrs_bin_cgi_2dbin_index_2epl::handler (unknown version) Line: 56
   Module: (eval) (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::run (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::default_handler (v1.99) Line: 170
   Module: ModPerl::Registry::handler (v1.99) Line: 31

[Tue Mar 10 10:08:01 2015] -e: DBD::mysql::st fetchrow_array failed: fetch() without execute() at /opt/otrs/KIX4OTRS/Kernel/System/DB.pm line 720.
ERROR: OTRS-CGI-01 Perl: 5.10.1 OS: linux Time: Tue Mar 10 10:08:01 2015

 Message: No such SLAID (3)!

 RemoteAddress: 10.36.150.7
 RequestURI: /otrs/index.pl?Action=AgentTicketEscalationView;SortBy=EscalationTime;OrderBy=Up;View=;Filter=NextWeek

 Traceback (29716): 
   Module: Kernel::System::SLA::SLAGet (OTRS 3.3.11) Line: 316
   Module: Kernel::System::Ticket::TicketEscalationPreferences (OTRS 3.3.11) Line: 2353
   Module: Kernel::System::Ticket::TicketEscalationDateCalculation (OTRS 3.3.11) Line: 2436
   Module: Kernel::System::Ticket::TicketGet (OTRS 3.3.11) Line: 6073
   Module: Kernel::Output::HTML::TicketOverviewSmall::Run (OTRS 3.3.11) Line: 487
   Module: Kernel::Output::HTML::LayoutTicket::TicketListShow (OTRS 3.3.11) Line: 2184
   Module: Kernel::Modules::AgentTicketEscalationView::Run (OTRS 3.3.11) Line: 405
   Module: Kernel::System::Web::InterfaceAgent::Run (OTRS 3.3.11) Line: 910
   Module: ModPerl::ROOT::ModPerl::Registry::opt_otrs_bin_cgi_2dbin_index_2epl::handler (unknown version) Line: 56
   Module: (eval) (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::run (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::default_handler (v1.99) Line: 170
   Module: ModPerl::Registry::handler (v1.99) Line: 31

ERROR: OTRS-CGI-01 Perl: 5.10.1 OS: linux Time: Tue Mar 10 10:08:01 2015

 Message: MySQL server has gone away, SQL: '
            SELECT article_key, article_value
            FROM article_flag
            WHERE article_id = ?
                AND create_by = ? LIMIT 1500'

 RemoteAddress: 10.36.165.10
 RequestURI: /otrs/index.pl?Action=AgentTicketEscalationView

 Traceback (29818): 
   Module: Kernel::System::Ticket::Article::ArticleFlagGet (OTRS 3.3.11) Line: 3191
   Module: Kernel::Output::HTML::TicketOverviewSmall::Run (OTRS 3.3.11) Line: 646
   Module: Kernel::Output::HTML::LayoutTicket::TicketListShow (OTRS 3.3.11) Line: 2184
   Module: Kernel::Modules::AgentTicketEscalationView::Run (OTRS 3.3.11) Line: 405
   Module: Kernel::System::Web::InterfaceAgent::Run (OTRS 3.3.11) Line: 910
   Module: ModPerl::ROOT::ModPerl::Registry::opt_otrs_bin_cgi_2dbin_index_2epl::handler (unknown version) Line: 56
   Module: (eval) (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::run (v1.99) Line: 204
   Module: ModPerl::RegistryCooker::default_handler (v1.99) Line: 170
   Module: ModPerl::Registry::handler (v1.99) Line: 31

Could anyone point me in the direction of further troubleshooting or a potential fix please?

[wurzel]

Hi,

see here http://bit.ly/HJj9PB

Flo

[retro2707]

Wurzel,

Fantastic response, managing to be both insightful and witty and at the same time not being irritating in the slightest... Bravo?

I had actually already googled said error message however it doesn't help me. Take for example the top result:

It mentions 2 things

Server timed out and closed the connection. To fix, check that “wait_timeout” mysql variable in your my.cnf configuration file is large enough.

For one, I dont have this in the my.cnf file. Secondly, why do I suddenly need this value in there? what has changed? This doesent answer the question on why it was fine but now is not. We have only had the system in for 3 months so its not exactly a massive database (about 0.7GB).

The 2nd recommendation is equally as pointless. Increase your max_allowed_packet size to 128MB. Great, but actually when I look at support info from OTRS, it tells me that my current value of 25MB is fine, and again, doesent answer the question of why should I change this now when it was fine before?

Apologies for not being a MySQL DBA and knowing how to troubleshoot this. By all means I can add or change these values if this is appropriate for OTRS, but telling me to google a generic error and make random changes to a production system is not really constructive.

Any further (actual) help would be appreciated.

[crythias]

But this is a DB error ... unless you can demonstrate otherwise, it doesn't (yet) belong here.
https://www.google.com/search?q=site:fo ... gws_rd=ssl

[EXG133]

The max_allowed_packet and wait_timeout are most likely not relevant, they only matter when you are executing a large SQL statement. Based on these two values MySQL decides when to cut a connection to avoid keeping connections open too long.

You will have to investigate why your DB is isn't responding or crashing. You surely have a my.cnf (try using 'locate my.cnf' to find it). Check where your log files are being saved (eg var/log/mysql/mysql.log ) and check these logs. The chance this is an OTRS problem is very small.

[retro2707]

Thanks crythias and EXG133. I dont know much about MySQL (I only installed it because of OTRS) but I will see what I can find with the logs. It was the fact that the issues seems to happen after the latest OTRS update which made me wonder if the problem was related to that.

I will post if I find anything constructive.