100% CPU usage after changing the server

Moderator: crythias

Post Reply
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

100% CPU usage after changing the server

Post by Snaake »

Hello everyone,
First of all, sorry for my english.

In my company, we use OTRS+ITSM 6.0.10 for one year in a test environment. Since one week, we migrate OTRS in a production environment. It means that OTRS is installed on a new VM.
Here is the steps followed :
1. Stop all services relevant to OTRS (except the database).
2. Make a full backup of OTRS from the test environment.
3. Restore the full backup on the production environment.
4. Change the differents settings of the configuration files (for example : Config.pm).
5. Start all services relevant to OTRS.
There is no difference between the test environment and the production environment, except the PostgreSQL version : from 10.4 to 10.8.

Since we make this change, we have 100% CPU usage pics sometimes, but we never had this before... See below :
Image
Image

In the httpd's log, I find this :
[Fri Jun 28 10:22:59 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:24:17 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:27:14 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:28:54 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.
[Fri Jun 28 10:30:58 2019] -e: Use of uninitialized value $NewTo in concatenation (.) or string at /opt/otrs//Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 383.

The errors appear in the same time of the 100% CPU usage pics.

Furthermore, I have a zombie process. Even after a reboot, the zombie process is still alive !
Image

Could you please help me to resolve this issue ?
Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
fcasal
Znuny wizard
Posts: 336
Joined: 21 Apr 2014, 16:14
Znuny Version: 6.0.18

Re: 100% CPU usage after changing the server

Post by fcasal »

I would update the otrs to the last patch version, they have fixed many errors

Regards!
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello fcasal,

Thanks for your fast reply !
But I have an another problem when I update OTRS to the last version (from 6.0.10 to 6.0.19).
My test environment is now a duplication of the production environment. So I make the update on the test environment, and I have an error.

Here is the steps followed :
1. Stop all services relevant to OTRS (httpd, crond, Cron.sh, otrs.Daemon.pl) in this order.
2. Make a full backup of OTRS.
3. Install the new version of OTRS (rpm -Uvh otrs-6.0.19-02.noarch.rpm)
Image

4. Run the migration script (sudo -u otrs ./script/DBUpdate-to-6.pl)

Code: Select all

 Migration started ...                                                                                                                                                    
                                                                                                                                                                          
 Checking requirements ...                                                                                                                                                
                                                                                                                                                                          
    Requirement check for: Check framework version ...                                                                                                                    
    Requirement check for: Check required Perl version ...                                                                                                                
    Requirement check for: Check required database version ...                                                                                                            
    Requirement check for: Check database charset ...                                                                                                                     
    Requirement check for: Check required Perl modules ...                                                                                                                
    Requirement check for: Check if database has been backed up ...                                                                                                       
                                                                                                                                                                          
        Did you backup the database? [Y]es/[N]o: Y                                                                                                                        
                                                                                                                                                                          
    Requirement check for: Upgrade database structure ...                                                                                                                 
    Requirement check for: Migrating time zone configuration ...                                                                                                          
    Requirement check for: Update calendar appointment future tasks ...                                                                                                   
    Requirement check for: Migrate GenericAgent jobs configuration ...                                                                                                    
    Requirement check for: Migrate TicketAppointment rules configuration ...                                                                                              
    Requirement check for: Create entries in new article table ...                                                                                                        
    Requirement check for: Migrate ArticleType in ProcessManagement Data ...                                                                                              
    Requirement check for: Migrate ArticleType in PostMaster filters ...                                                                                                  
                                                                                                                                                                          
 Executing tasks ...                                                                                                                                                      
                                                                                                                                                                          
    Step 1 of 42: Check framework version ...                                                                                                                             
    Step 2 of 42: Check required Perl version ...                                                                                                                         
    Step 3 of 42: Check required database version ...                                                                                                                     
    Step 4 of 42: Check database charset ...                                                                                                                              
    Step 5 of 42: Check required Perl modules ...                                                                                                                         
    Step 6 of 42: Check if database has been backed up ...                                                                                                                
    Step 7 of 42: Upgrade database structure ...                                                                                                                          
    Step 8 of 42: Migrate configuration ...                                                                                                                               
    Step 9 of 42: Refresh configuration cache after migration of OTRS 5 settings ...                                                                                      
    Step 10 of 42: Migrating ticket storage configuration ...                                                                                                             
    Step 11 of 42: Migrating article search index configuration ...                                                                                                       
    Step 12 of 42: Migrating ticket zoom customer information widget configuration ...                                                                                    
    Step 13 of 42: Drop deprecated table gi_object_lock_state ...                                                                                                         
    Step 14 of 42: Migrate PossibleNextActions setting ...                                                                                                                
    Step 15 of 42: Migrate ZoomExpand setting ...                                                                                                                         
    Step 16 of 42: Migrating time zone configuration ...                                                                                                                  
    Step 17 of 42: Create appointment calendar tables ...                                                                                                                 
    Step 18 of 42: Create ticket number counter tables ...                                                                                                                
    Step 19 of 42: Update calendar appointment future tasks ...                                                                                                           
    Step 20 of 42: Add basic appointment notification for reminders ...                                                                                                   
    Step 21 of 42: Create Form Draft tables ...                                                                                                                           
    Step 22 of 42: Clean and drop group_user permission_value column ...                                                                                                  
    Step 23 of 42: Migrate GenericAgent jobs configuration ...                                                                                                            
    Step 24 of 42: Migrate TicketAppointment rules configuration ...                                                                                                      
    Step 25 of 42: Migrate Merged Ticket history name values ...
    Step 26 of 42: Migrate ticket statistics ...
    Step 27 of 42: Migrate ticket notifications ...
    Step 28 of 42: Create entries in new article table ...
    Step 29 of 42: Post changes on article related tables ...
    Step 30 of 42: Migrate ArticleType in ProcessManagement Data ...
    Step 31 of 42: Migrate ArticleType in PostMaster filters ...
    Step 32 of 42: Migrate chat articles ...
    Step 33 of 42: Initialize default cron jobs ...
    Step 34 of 42: Migrate web service configuration ...
    Step 35 of 42: Migrate package repository configuration ...
    Step 36 of 42: Migrate ticket search profiles ...
    Step 37 of 42: Uninstall Merged Feature Add-Ons ...
    Step 38 of 42: Clean up the cache ...
    Step 39 of 42: Refresh configuration cache another time ...
    Step 40 of 42: Deploy ACLs ...
    Step 41 of 42: Deploy processes ...
    Step 42 of 42: Check invalid settings ...



 Migration completed!
5. Start all services relevant to OTRS (httpd, crond, Cron.sh) in this order (the Daemon will start with the Cron.sh).
6. Access to the Web Interface :
Image
In English : Your navigator is too old. This software runs with a huge lists of browsers, please upgrade to one of these. Thanks to refer to the documentation or ask to your system administrator for more informations.

In the Apache error log after the update :

Code: Select all

[Wed Jul 03 09:51:05.818904 2019] [suexec:notice] [pid 24421] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jul 03 09:51:06.021560 2019] [auth_digest:notice] [pid 24421] AH01757: generating secret for digest authentication ...
[Wed Jul 03 09:51:06.022153 2019] [lbmethod_heartbeat:notice] [pid 24421] AH02282: No slotmem from mod_heartmonitor
[Wed Jul 03 09:51:06.026513 2019] [mpm_prefork:notice] [pid 24421] AH00163: Apache/2.4.6 (CentOS) mod_perl/2.0.10 Perl/v5.16.3 configured -- resuming normal operations
[Wed Jul 03 09:51:06.026544 2019] [core:notice] [pid 24421] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
In the Apache access log after the update :

Code: Select all

y.y.y.y - - [03/Jul/2019:09:58:49 +0200] "GET /favicon.ico HTTP/1.1" 404 209 "http://xxx/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:58:49 +0200] "GET /otrs/index.pl HTTP/1.1" 200 3287 "http://xxx/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/skins/Agent/default/css/thirdparty/ui-theme/jquery-ui.css HTTP/1.1" 200 4062 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/skins/Agent/default/css-cache/CommonCSS_936d8ae02adc3ce7b51fa1962917e1cf.css HTTP/1.1" 200 28735 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/TranslationJS_fr_7f254087f4b89ef983965683023a8bf3.js HTTP/1.1" 200 6868 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/TemplateJS_5a290dca9c2fba9dd847bed0bd2a0130.js HTTP/1.1" 200 4194 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/ModuleJS_66a2bf5fe03b98da7330a53ccd96f093.js HTTP/1.1" 200 875 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/skins/Agent/default/css-cache/ResponsiveCSS_ffb4bb42e7ff6f25d7652ed19e3ed120.css HTTP/1.1" 200 4475 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
y.y.y.y - - [03/Jul/2019:09:59:01 +0200] "GET /otrs-web/js/js-cache/CommonJS_1ed8dcf8d778b1234f2659adf2efe8b1.js HTTP/1.1" 200 353583 "http://xxx/otrs/index.pl" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
::1 - - [03/Jul/2019:09:59:07 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) mod_perl/2.0.10 Perl/v5.16.3 (internal dummy connection)"
I see in the Apache access log the skin loaded is the default skin, but we use a modified skin. Perhaps the error is from here ? But how to resolve this ?
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
fcasal
Znuny wizard
Posts: 336
Joined: 21 Apr 2014, 16:14
Znuny Version: 6.0.18

Re: 100% CPU usage after changing the server

Post by fcasal »

There is a bug about this, check https://bugs.otrs.org/show_bug.cgi?id=14540

Regards
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello fcasal,

Thanks for your fast reply again !
I'm waiting for the OTRS version 6.0.20, the issue will be fix (I hope).
I will update OTRS from 6.0.10 to 6.0.20 in the test environment. If everything is OK, I will update OTRS on the production environment and I will see if my problem is fix or not.
I will tell you. Thanks for your help.

Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello !

Some news here :
I update OTRS from 6.0.10 to 6.0.20 on my test environment with the OTRS documentation.
After that, I use the new otrs.console.pl command with "Dev::Code::CPANAudit" and I update these modules with "cpan -f <module>".
But, I have 100% CPU usage pics again... even if nobody use the Web interface of OTRS ! :(

Anyone can help me please ?
Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
fcasal
Znuny wizard
Posts: 336
Joined: 21 Apr 2014, 16:14
Znuny Version: 6.0.18

Re: 100% CPU usage after changing the server

Post by fcasal »

Could you monitor which module o service are doing the 100% problem?

Regards!
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

I'm sorry but how to do that ? :?
I think it's a perl module (cgi module) from Apache.

EDIT : I found a trail ! It's maybe caused by the RefreshTime of the dashboard...

Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello,

Some news here. During breakfast, when nobody use the application, we have 100% CPU usage pics because the agents don't disconnect from OTRS.
So yesterday I kill every session and nobody was connected to OTRS. We don't have 100% CPU usage. So I think this occur when the web interface is refreshed by the agents.

Could you help me please ?
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: 100% CPU usage after changing the server

Post by root »

Hi,

Do you have escalations configured? And if yes how many tickets are right now escalated and what'ds the average age of them?

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello,
Thanks for the reply.

If you mean the escalation in the queues setting, I don't have any escalations. For all the queues, I set "0" :
Image

EDIT : We have different queues and we escalate the tickets with a manual action (from the "queue" button).
We have 400~500 open tickets and ~10000 closed tickets. The average age of all the tickets are 160 days (from 374 days to several minutes).

Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: 100% CPU usage after changing the server

Post by root »

Hi,

From my experience, this could be a reason. The date calculation needs a lot of CPU time, especially when you have older, escalated tickets.
I'm sorry, I only knew a commercial solution to this.

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello,
Thanks for your information but could you clarify for me please ? I want to fully understand :D

Do the escalation mean change a ticket from a queue to another ? Because in the queues settings it always be to "0" so it's desactivated, right ?
OTRS doesn't like to have a lot of tickets ? Does OTRS make the difference for opened and closed tickets (in this case) ?
And finally, which solution do we have ?

EDIT : This problem doesn't exist on the older server, it appear since the new server (at the beginning, with exact same tickets). So strange...
EDIT 2 : Does the problem disappear if the performance increase ? Actualy, it's a virtual machine with 2 vCPU (with one socket per core), 8Go RAM and a thin provisionning disk.

Thanks a lot for your time.
Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: 100% CPU usage after changing the server

Post by root »

Hi,

It seems to be related to the calculation of the escalations and the timezone feature which was introduced in OTRS 6.

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello,

Do you have any advice ?

Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: 100% CPU usage after changing the server

Post by root »

Snaake wrote: 26 Jul 2019, 10:59 Do you have any advice ?
Without cost, in favor of your customer: reduce escalations, use pending states. suspend escalations.

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
solown
Znuny newbie
Posts: 29
Joined: 05 Nov 2018, 16:28
Znuny Version: OTRS5

Re: 100% CPU usage after changing the server

Post by solown »

Hi Snaake,

I had the same problem. The solution for me was to change the MaxRequestWorkers line in the httpd.conf (#MPM Prefork section).

I didn't read everything about your problem but you can try this.


Regards,
Solown.

PS : English is not my native language, sorry if there is misstakes.
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

solown wrote: 29 Jul 2019, 13:25 The solution for me was to change the MaxRequestWorkers line in the httpd.conf (#MPM Prefork section).
Hello,


I don't have any "MaxRequestWorkers" line in the httpd.conf and the zzz_otrs.conf.

I only have
"MaxRequestsPerChild 4000" (in any section, just at the root of the file, it's the last line) in the zzz_otrs.conf
and
"MaxRequestsPerChild 0" ("<IfModule mpm_prefork_module>" section) in the httpd.conf.

I don't know which file is used for this option.
From which value to which value did you modify the "MaxRequestWorkers" line ?

PS : Sorry, English isn't my native language too.


Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
jlima
Znuny newbie
Posts: 34
Joined: 03 Jul 2014, 13:48
Znuny Version: 6.0.18
Real Name: Jorge Lima
Location: Braga, Portugal

Re: 100% CPU usage after changing the server

Post by jlima »

Hello

I have the same problem, sudden 100% CPU spikes on all four CPUs that can last for 15min.
As you can see, I have lots of Daemon as well as /opt/otrs/bin/c -DFOREGROUND processes

I don't know what they're doing.

Any help would be greatly appreciated.

Thanks,
Jorge

Image
OTRS 6.0.18 (public/testing) on CentOS with Postgres database
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello,

I still have the same problem...
I don't find a fix yet.

Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: 100% CPU usage after changing the server

Post by root »

Hi,

Did you checked disk i/o with vmstat or iotop? Found this as the reason with nearly 2/3 of all performance issues, especially in virtual environments.

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
obione85
Znuny newbie
Posts: 13
Joined: 30 Nov 2018, 12:44
Znuny Version: 6.0.14

Re: 100% CPU usage after changing the server

Post by obione85 »

same issue here ... we migrated to a new server, when we upgraded to OTRS 6

our testenvironment is like 3 times faster than our production OTRS. The testsystem has only 1/4 of RAM and CPU.

All benchmarks were slower on the production system. We were not able to fix this until now :(
OTRS 6.0.14 (test/live) on CentOS7 with MySQL database (MariaDB)
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

obione85 wrote: 24 Jan 2020, 12:42 same issue here ... we migrated to a new server, when we upgraded to OTRS 6

our testenvironment is like 3 times faster than our production OTRS. The testsystem has only 1/4 of RAM and CPU.

All benchmarks were slower on the production system. We were not able to fix this until now :(
Hello,

I still have the problem...
The only difference between you and me : I don't update OTRS. I kept the exactly same configuration, and I just import the full backup to one server to another one.
Nobody has a solution ?

Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
zzz
Znuny superhero
Posts: 888
Joined: 15 Dec 2016, 15:13
Znuny Version: All
Real Name: Emin
Company: Efflux GmbH
Contact:

Re: 100% CPU usage after changing the server

Post by zzz »

Hello Snaake,

It's always hard to find a remote solution for this kind of individual problem.
Snaake wrote: 18 Jul 2019, 14:25 I'm sorry but how to do that ? :?
I think it's a perl module (cgi module) from Apache.

EDIT : I found a trail ! It's maybe caused by the RefreshTime of the dashboard...

Snaake
Have you ever tried to disable the 'PreferencesGroups###Refresh' configuration?
Did you also ensure that ./bin/otrs.CheckModules.pl does not show you any uninstalled modules and that the support data collecter is not showing any errors?

Best regards
Emin
Professional OTRS, Znuny & OTOBO services: efflux.de | efflux.de/en/

Free and premium add-ons: German | English
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hello,


zzz wrote: 05 Feb 2020, 11:30 Have you ever tried to disable the 'PreferencesGroups###Refresh' configuration?
I never disable this, but I reduce the time until the refresh of the dashboard happens => not better.

zzz wrote: 05 Feb 2020, 11:30 Did you also ensure that ./bin/otrs.CheckModules.pl does not show you any uninstalled modules?

Code: Select all

[root@XXX otrs]# sudo -u otrs ./bin/otrs.CheckModules.pl                                                                                                          
perl: warning: Setting locale failed.                                                                                                                                  
perl: warning: Please check that your locale settings:                                                                                                                 
        LANGUAGE = (unset),                                                                                                                                            
        LC_ALL = (unset),                                                                                                                                              
        LANG = "fr.UTF-8"                                                                                                                                              
    are supported and installed on your system.                                                                                                                        
perl: warning: Falling back to the standard locale ("C").
  o Apache::DBI......................ok (v1.12)                                                                                                                        
  o Apache2::Reload..................ok (v0.13)                                                                                                                        
  o Archive::Tar.....................ok (v1.92)                                                                                                                        
  o Archive::Zip.....................ok (v1.30)                                                                                                                        
  o Crypt::Eksblowfish::Bcrypt.......ok (v0.009)                                                                                                                       
  o Crypt::SSLeay....................ok (v0.64)                                                                                                                        
  o Date::Format.....................ok (v2.24)                                                                                                                        
  o DateTime.........................ok (v1.04)                                                                                                                        
  o DBI..............................ok (v1.627)                                                                                                                       
  o DBD::mysql.......................Not installed! Use: 'yum install "perl(DBD::mysql)"' (optional - Required to connect to a MySQL database.)                        
  o DBD::ODBC........................Not installed! (optional - Required to connect to a MS-SQL database.)                                                             
  o DBD::Oracle......................Not installed! (optional - Required to connect to a Oracle database.)                                                             
  o DBD::Pg..........................ok (v2.19.3)                                                                                                                      
  o Digest::SHA......................ok (v5.85)                                                                                                                        
  o Encode::HanExtra.................ok (v0.23)                                                                                                                        
  o IO::Socket::SSL..................ok (v1.94)                                                                                                                        
  o JSON::XS.........................ok (v3.01)                                                                                                                        
  o List::Util::XS...................ok (v1.27)                                                                                                                        
  o LWP::UserAgent...................ok (v6.26)                                                                                                                        
  o Mail::IMAPClient.................ok (v3.37)                                                                                                                        
       o IO::Socket::SSL................ok (v1.94)                                                                                                                        
       o Authen::SASL...................ok (v2.15)                                                                                                                        
       o Authen::NTLM...................ok (v1.09)                                                                                                                        
  o ModPerl::Util....................ok (v2.000010)                                                                                                                    
  o Net::DNS.........................ok (v0.72)                                                                                                                        
  o Net::LDAP........................ok (v0.56)                                                                                                                        
  o Template.........................ok (v2.24)                                                                                                                        
  o Template::Stash::XS..............ok (undef)                                                                                                                        
  o Text::CSV_XS.....................ok (v1.00)                                                                                                                        
  o Time::HiRes......................ok (v1.9725)                                                                                                                      
  o XML::LibXML......................ok (v2.0018)                                                                                                                      
  o XML::LibXSLT.....................ok (v1.80)                                                                                                                        
  o XML::Parser......................ok (v2.41)                                                                                                                        
  o YAML::XS.........................ok (v0.54)
Modules are up-to-date but I have perl warning with user root (not display with user otrs).

zzz wrote: 05 Feb 2020, 11:30 Did you also ensure that the support data collecter is not showing any errors?

Code: Select all

ERROR: duplicate key value violates unique constraint "ticket_flag_per_user"
The only error we have is this. But since the beginning, not since the migration in production. We never found an issue for this error.



Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
Snaake
Znuny newbie
Posts: 14
Joined: 27 Jun 2019, 17:14
Znuny Version: 6.0.10

Re: 100% CPU usage after changing the server

Post by Snaake »

Hi everyone,

My issue is now resolved ! No more pics of CPU usage and no more latency. :D
I just give more memory on the VM (from 8Go to 12Go). Before changing the MaxSpareServers of the httpd service (from 30 to 50), the entire memory was used and the server swaps. Now, with enough memory, it's OK.


Sincerely,
Snaake
OTRS+ITSM 6.0.10 on CentOS 7.6 with Apache 2.4.6, mod_perl 2.0.10, PostgreSQL 10.8 and Perl 5.16.3.
Post Reply