100% CPU usage after changing the server

[Snaake]

Hello everyone,
First of all, sorry for my english.

In my company, we use OTRS+ITSM 6.0.10 for one year in a test environment. Since one week, we migrate OTRS in a production environment. It means that OTRS is installed on a new VM.
Here is the steps followed :
1. Stop all services relevant to OTRS (except the database).
2. Make a full backup of OTRS from the test environment.
3. Restore the full backup on the production environment.
4. Change the differents settings of the configuration files (for example : Config.pm).
5. Start all services relevant to OTRS.
There is no difference between the test environment and the production environment, except the PostgreSQL version : from 10.4 to 10.8.

Since we make this change, we have 100% CPU usage pics sometimes, but we never had this before... See below :



In the httpd's log, I find this :
[Fri Jun 28 10:22:59 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:24:17 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:27:14 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:28:54 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:30:58 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.

The errors appear in the same time of the 100% CPU usage pics.

Furthermore, I have a zombie process. Even after a reboot, the zombie process is still alive !


Could you please help me to resolve this issue ?
Snaake

[fcasal]

I would update the otrs to the last patch version, they have fixed many errors

Regards!

[Snaake]

Hello fcasal,

Thanks for your fast reply !
But I have an another problem when I update OTRS to the last version (from 6.0.10 to 6.0.19).
My test environment is now a duplication of the production environment. So I make the update on the test environment, and I have an error.

Here is the steps followed :
1. Stop all services relevant to OTRS (httpd, crond, Cron.sh, otrs.Daemon.pl) in this order.
2. Make a full backup of OTRS.
3. Install the new version of OTRS (rpm -Uvh otrs-6.0.19-02.noarch.rpm)


4. Run the migration script (sudo -u otrs ./script/DBUpdate-to-6.pl)

Code: Select all

 Migration started ...                                                                                                                                                    
                                                                                                                                                                          
 Checking requirements ...                                                                                                                                                
                                                                                                                                                                          
    Requirement check for: Check framework version ...                                                                                                                    
    Requirement check for: Check required Perl version ...                                                                                                                
    Requirement check for: Check required database version ...                                                                                                            
    Requirement check for: Check database charset ...                                                                                                                     
    Requirement check for: Check required Perl modules ...                                                                                                                
    Requirement check for: Check if database has been backed up ...                                                                                                       
                                                                                                                                                                          
        Did you backup the database? [Y]es/[N]o: Y                                                                                                                        
                                                                                                                                                                          
    Requirement check for: Upgrade database structure ...                                                                                                                 
    Requirement check for: Migrating time zone configuration ...                                                                                                          
    Requirement check for: Update calendar appointment future tasks ...                                                                                                   
    Requirement check for: Migrate GenericAgent jobs configuration ...                                                                                                    
    Requirement check for: Migrate TicketAppointment rules configuration ...                                                                                              
    Requirement check for: Create entries in new article table ...                                                                                                        
    Requirement check for: Migrate ArticleType in ProcessManagement Data ...                                                                                              
    Requirement check for: Migrate ArticleType in PostMaster filters ...                                                                                                  
                                                                                                                                                                          
 Executing tasks ...                                                                                                                                                      
                                                                                                                                                                          
    Step 1 of 42: Check framework version ...                                                                                                                             
    Step 2 of 42: Check required Perl version ...                                                                                                                         
    Step 3 of 42: Check required database version ...                                                                                                                     
    Step 4 of 42: Check database charset ...                                                                                                                              
    Step 5 of 42: Check required Perl modules ...                                                                                                                         
    Step 6 of 42: Check if database has been backed up ...                                                                                                                
    Step 7 of 42: Upgrade database structure ...                                                                                                                          
    Step 8 of 42: Migrate configuration ...                                                                                                                               
    Step 9 of 42: Refresh configuration cache after migration of OTRS 5 settings ...                                                                                      
    Step 10 of 42: Migrating ticket storage configuration ...                                                                                                             
    Step 11 of 42: Migrating article search index configuration ...                                                                                                       
    Step 12 of 42: Migrating ticket zoom customer information widget configuration ...                                                                                    
    Step 13 of 42: Drop deprecated table gi_object_lock_state ...                                                                                                         
    Step 14 of 42: Migrate PossibleNextActions setting ...                                                                                                                
    Step 15 of 42: Migrate ZoomExpand setting ...                                                                                                                         
    Step 16 of 42: Migrating time zone configuration ...                                                                                                                  
    Step 17 of 42: Create appointment calendar tables ...                                                                                                                 
    Step 18 of 42: Create ticket number counter tables ...                                                                                                                
    Step 19 of 42: Update calendar appointment future tasks ...                                                                                                           
    Step 20 of 42: Add basic appointment notification for reminders ...                                                                                                   
    Step 21 of 42: Create Form Draft tables ...                                                                                                                           
    Step 22 of 42: Clean and drop group_user permission_value column ...                                                                                                  
    Step 23 of 42: Migrate GenericAgent jobs configuration ...                                                                                                            
    Step 24 of 42: Migrate TicketAppointment rules configuration ...                                                                                                      
    Step 25 of 42: Migrate Merged Ticket history name values ...
    Step 26 of 42: Migrate ticket statistics ...
    Step 27 of 42: Migrate ticket notifications ...
    Step 28 of 42: Create entries in new article table ...
    Step 29 of 42: Post changes on article related tables ...
    Step 30 of 42: Migrate ArticleType in ProcessManagement Data ...
    Step 31 of 42: Migrate ArticleType in PostMaster filters ...
    Step 32 of 42: Migrate chat articles ...
    Step 33 of 42: Initialize default cron jobs ...
    Step 34 of 42: Migrate web service configuration ...
    Step 35 of 42: Migrate package repository configuration ...
    Step 36 of 42: Migrate ticket search profiles ...
    Step 37 of 42: Uninstall Merged Feature Add-Ons ...
    Step 38 of 42: Clean up the cache ...
    Step 39 of 42: Refresh configuration cache another time ...
    Step 40 of 42: Deploy ACLs ...
    Step 41 of 42: Deploy processes ...
    Step 42 of 42: Check invalid settings ...



 Migration completed!

5. Start all services relevant to OTRS (httpd, crond, Cron.sh) in this order (the Daemon will start with the Cron.sh).
6. Access to the Web Interface :

In English : Your navigator is too old. This software runs with a huge lists of browsers, please upgrade to one of these. Thanks to refer to the documentation or ask to your system administrator for more informations.

In the Apache error log after the update :

Code: Select all

[Wed Jul 03 09:51:05.818904 2019] [suexec:notice] [pid 24421] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jul 03 09:51:06.021560 2019] [auth_digest:notice] [pid 24421] AH01757: generating secret for digest authentication ...
[Wed Jul 03 09:51:06.022153 2019] [lbmethod_heartbeat:notice] [pid 24421] AH02282: No slotmem from mod_heartmonitor
[Wed Jul 03 09:51:06.026513 2019] [mpm_prefork:notice] [pid 24421] AH00163: Apache/2.4.6 (CentOS) mod_perl/2.0.10 Perl/v5.16.3 configured -- resuming normal operations
[Wed Jul 03 09:51:06.026544 2019] [core:notice] [pid 24421] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

In the Apache access log after the update :

Code: Select all

y.y.y.y - - [03/Jul/2019:09:58:49 +0200] "GET /favicon.ico HTTP/1.1" 404 209 "http://xxx/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:58:49 +0200] "GET /otrs/index.pl HTTP/1.1" 200 3287 "http://xxx/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/skins/Agent/default/css/thirdparty/ui-theme/jquery-ui.css HTTP/1.1" 200 4062 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/skins/Agent/default/css-cache/CommonCSS_936d8ae02adc3ce7b51fa1962917e1cf.css HTTP/1.1" 200 28735 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/TranslationJS_fr_7f254087f4b89ef983965683023a8bf3.js HTTP/1.1" 200 6868 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/TemplateJS_5a290dca9c2fba9dd847bed0bd2a0130.js HTTP/1.1" 200 4194 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/ModuleJS_66a2bf5fe03b98da7330a53ccd96f093.js HTTP/1.1" 200 875 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/skins/Agent/default/css-cache/ResponsiveCSS_ffb4bb42e7ff6f25d7652ed19e3ed120.css HTTP/1.1" 200 4475 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/CommonJS_1ed8dcf8d778b1234f2659adf2efe8b1.js HTTP/1.1" 200 353583 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
::1 - - [03/Jul/2019:09:59:07 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_perl/2.0.10 Perl/v5.16.3 (internal dummy connection)"

I see in the Apache access log the skin loaded is the default skin, but we use a modified skin. Perhaps the error is from here ? But how to resolve this ?

[fcasal]

There is a bug about this, check https://bugs.otrs.org/show_bug.cgi?id=14540

Regards

[Snaake]

Hello fcasal,

Thanks for your fast reply again !
I'm waiting for the OTRS version 6.0.20, the issue will be fix (I hope).
I will update OTRS from 6.0.10 to 6.0.20 in the test environment. If everything is OK, I will update OTRS on the production environment and I will see if my problem is fix or not.
I will tell you. Thanks for your help.

Snaake

[Snaake]

Hello !

Some news here :
I update OTRS from 6.0.10 to 6.0.20 on my test environment with the OTRS documentation.
After that, I use the new otrs.console.pl command with "Dev::Code::CPANAudit" and I update these modules with "cpan -f <module>".
But, I have 100% CPU usage pics again... even if nobody use the Web interface of OTRS !

Anyone can help me please ?
Snaake

[fcasal]

Could you monitor which module o service are doing the 100% problem?

Regards!

[Snaake]

I'm sorry but how to do that ?
I think it's a perl module (cgi module) from Apache.

EDIT : I found a trail ! It's maybe caused by the RefreshTime of the dashboard...

Snaake

[Snaake]

Hello,

Some news here. During breakfast, when nobody use the application, we have 100% CPU usage pics because the agents don't disconnect from OTRS.
So yesterday I kill every session and nobody was connected to OTRS. We don't have 100% CPU usage. So I think this occur when the web interface is refreshed by the agents.

Could you help me please ?

[root]

Hi,

Do you have escalations configured? And if yes how many tickets are right now escalated and what'ds the average age of them?

- Roy

[Snaake]

Hello,
Thanks for the reply.

If you mean the escalation in the queues setting, I don't have any escalations. For all the queues, I set "0" :


EDIT : We have different queues and we escalate the tickets with a manual action (from the "queue" button).
We have 400~500 open tickets and ~10000 closed tickets. The average age of all the tickets are 160 days (from 374 days to several minutes).

Snaake

[root]

Hi,

From my experience, this could be a reason. The date calculation needs a lot of CPU time, especially when you have older, escalated tickets.
I'm sorry, I only knew a commercial solution to this.

- Roy

[Snaake]

Hello,
Thanks for your information but could you clarify for me please ? I want to fully understand

Do the escalation mean change a ticket from a queue to another ? Because in the queues settings it always be to "0" so it's desactivated, right ?
OTRS doesn't like to have a lot of tickets ? Does OTRS make the difference for opened and closed tickets (in this case) ?
And finally, which solution do we have ?

EDIT : This problem doesn't exist on the older server, it appear since the new server (at the beginning, with exact same tickets). So strange...
EDIT 2 : Does the problem disappear if the performance increase ? Actualy, it's a virtual machine with 2 vCPU (with one socket per core), 8Go RAM and a thin provisionning disk.

Thanks a lot for your time.
Snaake

[root]

Hi,

It seems to be related to the calculation of the escalations and the timezone feature which was introduced in OTRS 6.

- Roy

[Snaake]

Hello,

Do you have any advice ?

Snaake

[root]

Do you have any advice ?

Without cost, in favor of your customer: reduce escalations, use pending states. suspend escalations.

- Roy

[solown]

Hi Snaake,

I had the same problem. The solution for me was to change the MaxRequestWorkers line in the httpd.conf (#MPM Prefork section).

I didn't read everything about your problem but you can try this.


Regards,
Solown.

PS : English is not my native language, sorry if there is misstakes.

[Snaake]

The solution for me was to change the MaxRequestWorkers line in the httpd.conf (#MPM Prefork section).

Hello,


I don't have any "MaxRequestWorkers" line in the httpd.conf and the zzz_otrs.conf.

I only have
"MaxRequestsPerChild 4000" (in any section, just at the root of the file, it's the last line) in the zzz_otrs.conf
and
"MaxRequestsPerChild 0" ("<IfModule mpm_prefork_module>" section) in the httpd.conf.

I don't know which file is used for this option.
From which value to which value did you modify the "MaxRequestWorkers" line ?

PS : Sorry, English isn't my native language too.


Snaake

[jlima]

Hello

I have the same problem, sudden 100% CPU spikes on all four CPUs that can last for 15min.
As you can see, I have lots of Daemon as well as /opt/otrs/bin/c -DFOREGROUND processes

I don't know what they're doing.

Any help would be greatly appreciated.

Thanks,
Jorge

[Snaake]

Hello,

I still have the same problem...
I don't find a fix yet.

Snaake

[root]

Hi,

Did you checked disk i/o with vmstat or iotop? Found this as the reason with nearly 2/3 of all performance issues, especially in virtual environments.

- Roy

[obione85]

same issue here ... we migrated to a new server, when we upgraded to OTRS 6

our testenvironment is like 3 times faster than our production OTRS. The testsystem has only 1/4 of RAM and CPU.

All benchmarks were slower on the production system. We were not able to fix this until now

[Snaake]

same issue here ... we migrated to a new server, when we upgraded to OTRS 6

our testenvironment is like 3 times faster than our production OTRS. The testsystem has only 1/4 of RAM and CPU.

All benchmarks were slower on the production system. We were not able to fix this until now

Hello,

I still have the problem...
The only difference between you and me : I don't update OTRS. I kept the exactly same configuration, and I just import the full backup to one server to another one.
Nobody has a solution ?

Snaake

[zzz]

Hello Snaake,

It's always hard to find a remote solution for this kind of individual problem.

I'm sorry but how to do that ?
I think it's a perl module (cgi module) from Apache.

EDIT : I found a trail ! It's maybe caused by the RefreshTime of the dashboard...

Snaake

Have you ever tried to disable the 'PreferencesGroups###Refresh' configuration?
Did you also ensure that ./bin/otrs.CheckModules.pl does not show you any uninstalled modules and that the support data collecter is not showing any errors?

Best regards
Emin

[Snaake]

Hello,

Have you ever tried to disable the 'PreferencesGroups###Refresh' configuration?

I never disable this, but I reduce the time until the refresh of the dashboard happens => not better.

Did you also ensure that ./bin/otrs.CheckModules.pl does not show you any uninstalled modules?

Code: Select all

[root@XXX otrs]# sudo -u otrs ./bin/otrs.CheckModules.pl                                                                                                          
perl: warning: Setting locale failed.                                                                                                                                  
perl: warning: Please check that your locale settings:                                                                                                                 
        LANGUAGE = (unset),                                                                                                                                            
        LC_ALL = (unset),                                                                                                                                              
        LANG = "fr.UTF-8"                                                                                                                                              
    are supported and installed on your system.                                                                                                                        
perl: warning: Falling back to the standard locale ("C").
  o Apache::DBI......................ok (v1.12)                                                                                                                        
  o Apache2::Reload..................ok (v0.13)                                                                                                                        
  o Archive::Tar.....................ok (v1.92)                                                                                                                        
  o Archive::Zip.....................ok (v1.30)                                                                                                                        
  o Crypt::Eksblowfish::Bcrypt.......ok (v0.009)                                                                                                                       
  o Crypt::SSLeay....................ok (v0.64)                                                                                                                        
  o Date::Format.....................ok (v2.24)                                                                                                                        
  o DateTime.........................ok (v1.04)                                                                                                                        
  o DBI..............................ok (v1.627)                                                                                                                       
  o DBD::mysql.......................Not installed! Use: 'yum install "perl(DBD::mysql)"' (optional - Required to connect to a MySQL database.)                        
  o DBD::ODBC........................Not installed! (optional - Required to connect to a MS-SQL database.)                                                             
  o DBD::Oracle......................Not installed! (optional - Required to connect to a Oracle database.)                                                             
  o DBD::Pg..........................ok (v2.19.3)                                                                                                                      
  o Digest::SHA......................ok (v5.85)                                                                                                                        
  o Encode::HanExtra.................ok (v0.23)                                                                                                                        
  o IO::Socket::SSL..................ok (v1.94)                                                                                                                        
  o JSON::XS.........................ok (v3.01)                                                                                                                        
  o List::Util::XS...................ok (v1.27)                                                                                                                        
  o LWP::UserAgent...................ok (v6.26)                                                                                                                        
  o Mail::IMAPClient.................ok (v3.37)                                                                                                                        
       o IO::Socket::SSL................ok (v1.94)                                                                                                                        
       o Authen::SASL...................ok (v2.15)                                                                                                                        
       o Authen::NTLM...................ok (v1.09)                                                                                                                        
  o ModPerl::Util....................ok (v2.000010)                                                                                                                    
  o Net::DNS.........................ok (v0.72)                                                                                                                        
  o Net::LDAP........................ok (v0.56)                                                                                                                        
  o Template.........................ok (v2.24)                                                                                                                        
  o Template::Stash::XS..............ok (undef)                                                                                                                        
  o Text::CSV_XS.....................ok (v1.00)                                                                                                                        
  o Time::HiRes......................ok (v1.9725)                                                                                                                      
  o XML::LibXML......................ok (v2.0018)                                                                                                                      
  o XML::LibXSLT.....................ok (v1.80)                                                                                                                        
  o XML::Parser......................ok (v2.41)                                                                                                                        
  o YAML::XS.........................ok (v0.54)

Modules are up-to-date but I have perl warning with user root (not display with user otrs).

Did you also ensure that the support data collecter is not showing any errors?

Code: Select all

ERROR: duplicate key value violates unique constraint "ticket_flag_per_user"

The only error we have is this. But since the beginning, not since the migration in production. We never found an issue for this error.



Snaake

[Snaake]

Hi everyone,

My issue is now resolved ! No more pics of CPU usage and no more latency.
I just give more memory on the VM (from 8Go to 12Go). Before changing the MaxSpareServers of the httpd service (from 30 to 50), the entire memory was used and the server swaps. Now, with enough memory, it's OK.


Sincerely,
Snaake