502 errors after upgrade from 6.0.12 to 6.0.34.

Moderator: crythias

Post Reply
jeansantos
Znuny newbie
Posts: 3
Joined: 29 Sep 2021, 22:42
Znuny Version: 6.0.36
Real Name: Jean Santos

502 errors after upgrade from 6.0.12 to 6.0.34.

Post by jeansantos »

We upgraded the OTRS version from 6.0.12 to 6.0.34 mainly for security reasons but also to change the repository to the Znuny repository. After the change, the team that uses OTRS started to complain that they're experiencing a lot of 502 Bad Gateway errors. Checking the Apache error log, I've found that the apache process is being killed by segmentation fault:
The following PID (1083) is an example:

Code: Select all

[Wed Sep 29 11:13:05.225943 2021] [mpm_event:debug] [pid 1083:tid 140042582447872] event.c(2094): AH02471: start_threads: Using epoll
[Wed Sep 29 13:51:55.918160 2021] [core:notice] [pid 1304:tid 140042880669568] AH00051: child pid 1083 exit signal Segmentation fault (11), possible coredump in /etc/apache2

Here's also an example from a PID that died in 4 minutes:

Code: Select all

[Tue Sep 28 14:45:04.026821 2021] [mpm_event:debug] [pid 21198:tid 140193719068416] event.c(2094): AH02471: start_threads: Using epoll
[Tue Sep 28 15:49:11.123843 2021] [core:notice] [pid 19763:tid 140194017290112] AH00051: child pid 21198 exit signal Segmentation fault (11), possible coredump in /etc/apache2
(These Segmentation faults are related to server usage, so the crashes are mostly during commercial hours.)

Using one crash file to extract the core dump, I got the following:

Code: Select all

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 11748]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f816a30b1dd in modperl_handler_resolve () from /usr/lib/apache2/modules/mod_perl.so
(gdb)
Looks that the segmentation fault comes from the Perl mod, so from OTRS.

Code: Select all

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f816a30b1dd in modperl_handler_resolve () from /usr/lib/apache2/modules/mod_perl.so
I checked the version of the things that we're running. Still, I didn't find anything useful so far. Our apache configuration is relatively standard. The base for the configuration is the example configuration with some additional improvements to use OneLogin as an SSO tool.
  • Server: Ubuntu xenial (16.04)
    OTRS: 6.0.34
    Apache: 2.4.18-2ubuntu3.17
    perl: 5.22.1-9ubuntu0.9
    libapache2-mod-perl2: 2.0.9-4ubuntu1.2
I would like some direction on how to troubleshoot this or get more directions from OTRS.

Thanks in advance
Jean
Sysadmin
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: 502 errors after upgrade from 6.0.12 to 6.0.34.

Post by root »

Hi,

Let's start with disabling mpm_event and enable mpm_prefork.

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
jeansantos
Znuny newbie
Posts: 3
Joined: 29 Sep 2021, 22:42
Znuny Version: 6.0.36
Real Name: Jean Santos

Re: 502 errors after upgrade from 6.0.12 to 6.0.34.

Post by jeansantos »

Looks like that this solved the issue, we install the apache using ansible but we don't touch the MPM, so we were using the default MPM (event).
I'll now follow the logs for few days to check if we need to change the configurations.
Thank you, Roy.
Jean
Sysadmin
Post Reply