Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Moderator: crythias

Locked
MrAsp
Znuny newbie
Posts: 7
Joined: 12 Sep 2014, 09:55
Znuny Version: 3.3.8
Real Name: Isak

Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by MrAsp »

Hi.

I just started at a new job and my first task was to upgrade the companys OTRS-ticketsystem.
The system works fine except some complains about slow search. Over 60sec is not unusually. And kind of the main reason for the upgrade.

System today:
1 virtual server
openSUSE 11.2
2CPU @ 2.8GHz
2GB RAM
mysql-5.1.49
apache2-2.2.13
OTRS 3.0.5 15100 tickets

Code: Select all

Insert Time:	10000	5 s	Ok
Update Time:	10000	5 s :-)	Looks fine!
Select Time:	10000	4 s :-)	Looks fine!
Delete Time:	10000	5 s	Ok
Multiplier:	* 1	s
New system:
1 virtual server
openSUSE 12.3
2CPU @ 3.8GHz
3GB RAM
mariadb-5.5.33
apache2-2.2.22
OTRS 3.3.8

Code: Select all

Insert Time:	10000	109 s :-(	Should not take more than 5's on an average system.
Update Time:	10000	111 s :-(	Should not take more than 9's on an average system.
Select Time:	10000	1 s :-)	Looks fine!
Delete Time:	10000	112 s :-(	Should not take more than 5's on an average system.
Multiplier:	* 1	s
The old server was cloned and upgraded in those steps required to get to 3.3. Then I ran a backup on the system from which i did a restore into
the new server.

As you can see the response time from the new OTRS system is worthless.
I have never worked with databases before, so I don’t know anything about optimizing them.

The otrs.CheckDB.pl doesn’t show any errors.
Apache and mysql(mariaDB) are started, OTRS-service not. I don’t want the new system to fetch mail and snitch them from the old,active system.

This is the my.cnf that was on the old server with som smaller tweaks to handle the upgraded memory and InnoDB.

Code: Select all

[client]
port		= 3306
socket		= /var/run/mysql/mysql.sock
[mysqld]
port		= 3306
socket		= /var/run/mysql/mysql.sock
datadir	= /var/lib/mysql
skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 25M
query_cache_size=256M
table_open_cache = 64
sort_buffer_size = 1024K
net_buffer_length = 100K
read_buffer_size = 512K
read_rnd_buffer_size = 1024K
myisam_sort_buffer_size = 16M
server-id	= 1
innodb_data_home_dir = /var/lib/mysql
innodb_buffer_pool_size = 2000M
thread_cache_size = 4
query_cache_limit = 100M
log-slow-queries = 1
innodb_file_per_table = 1
[safe_mysqld]
log-error	= /var/log/mysql/mysqld.log
socket		= /var/run/mysql/mysql.sock
!includedir /etc/mysql
[mysqldump]
socket		= /var/run/mysql/mysql.sock
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
[mysqld_multi]
mysqld     = /usr/bin/mysqld_safe
mysqladmin = /usr/bin/mysqladmin
log        = /var/log/mysqld_multi.log
Do you need more info to help me? If so, what do you need.

Best regards
schulmann
Znuny wizard
Posts: 477
Joined: 20 Nov 2011, 16:08
Znuny Version: 6.5.11
Real Name: Schulmann

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by schulmann »

MrAsp wrote:The system works fine except some complains about slow search. Over 60sec is not unusually. And kind of the main reason for the upgrade.
How many concurrent agents are on your system?
How many articles are in your system?
What does "vmstat 1" say while searching?
How quick is the search operation when there is only one agent logged in?
Did you check viewtopic.php?f=62&t=23757&hilit=+keepalive#p94002 ?
Why don't you use openSUSE 13.1?
Znuny6/Debian/ESXi
MrAsp
Znuny newbie
Posts: 7
Joined: 12 Sep 2014, 09:55
Znuny Version: 3.3.8
Real Name: Isak

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by MrAsp »

Thank you for your reply.

What does "vmstat 1" say while searching?

Code: Select all

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0   9980 222396 209352 1901696    0    0     2    12   12   34  0  0 99  0  0
 0  0   9980 222372 209352 1901668    0    0     0     0   26   73  0  0 100  0  0
 0  0   9980 222372 209352 1901668    0    0     0     0   22   52  0  0 100  0  0
 0  0   9980 218804 209352 1901668    0    0     0   112  173  527 19  1 70 11  0
 0  0   9980 218652 209352 1901688    0    0     0    12   30   80  0  0 99  1  0
 0  0   9980 218348 209352 1901688    0    0     0     0   33   90  0  0 100  0  0
 0  0   9980 218196 209352 1901668    0    0     0     0   33   70  0  0 100  0  0
 0  0   9980 218044 209352 1901668    0    0     0     4   32   84  0  0 100  0  0
 0  0   9980 217892 209364 1901656    0    0     0   244   44   95  0  0 95  5  0
 0  0   9980 217740 209364 1901668    0    0     0     0   28   68  0  0 100  0  0
 0  0   9980 217588 209364 1901668    0    0     0     0   35   93  0  0 100  0  0
 0  0   9980 217476 209364 1901656    0    0     0     0   38   79  1  0 100  0  0
 0  0   9980 217324 209364 1901656    0    0     0     0   31   74  0  0 100  0  0
 0  1   9980 217172 209368 1901652    0    0     0   528   57  139  0  0 68 32  0
 0  0   9980 216564 209372 1901664    0    0     0    12   42   99  1  0 99  1  0
 0  0   9980 216108 209372 1901664    0    0     0     0   36   79  0  0 100  0  0
 0  0   9980 215988 209372 1901664    0    0     0  1232  116  124  0  0 100  0  0
 0  0   9980 215836 209372 1901656    0    0     0     0   74  116  0  0 100  0  0
 0  0   9980 215684 209372 1901656    0    0     0     0   33   74  1  0 100  0  0
 0  0   9980 215684 209384 1901644    0    0     0    76   40   97  0  0 96  4  0
 0  0   9980 215380 209384 1901656    0    0     0     0   40   96  0  0 100  0  0
 0  0   9980 215380 209384 1901640    0    0     0     0   32   78  0  0 100  0  0
 0  0   9980 212948 209384 1901640    0    0     0    36   92  208  6  0 87  6  0
 0  0   9980 212948 209388 1901636    0    0     0   348   38   99  0  0 93  7  0
 0  0   9980 212948 209388 1901640    0    0     0     0   22   50  0  0 100  0  0
 0  0   9980 212948 209396 1901632    0    0     0    20   29   60  0  0 97  3  0

How many concurrent agents are on your system?
On the new server 1, me. On the old live system, 10.

How many articles are in your system?
81772

How quick is the search operation when there is only one agent logged in?
About 25sec on the new server. Cannot try it on the live system.

Why don't you use openSUSE 13.1?
Its not supported by the XenServer-environment we use. And for now we cant upgrade it
due to other dependencies.


I have not read your link yet, will do it now.
Rooobaaat
Znuny wizard
Posts: 432
Joined: 11 Sep 2014, 16:28
Znuny Version: OTRS 5.0.x

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by Rooobaaat »

stupid question... mod_perl is activated?
My english is better than your german :P

"Produktiv": OTRS: 5.0.x, OTRS::ITSM 5.0.x
"Testing": OTRS 6 git
OS: Debian 8.0 (Jessie)
Apache2.4.10/MySQL 5.5.41
MrAsp
Znuny newbie
Posts: 7
Joined: 12 Sep 2014, 09:55
Znuny Version: 3.3.8
Real Name: Isak

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by MrAsp »

No questions is stupid. I'm out of knowledge and ideas so I'm glad for all the help I can get :)

mod_perl is added to the "APACHE_MODULES" row in /etc/sysconfig/apache2
And under the Support Assessment category it shows green status. (mod_perl/2.0.6)
MrAsp
Znuny newbie
Posts: 7
Joined: 12 Sep 2014, 09:55
Znuny Version: 3.3.8
Real Name: Isak

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by MrAsp »

Quick question:
For every 5 concurrent agents 1 additional Apache process
For every Apache process 250 MB of RAM

Where do i change this? Is in the server-tuning.conf as the other settings? In such case, which settings?
srenon
Znuny newbie
Posts: 60
Joined: 20 Mar 2013, 14:26
Znuny Version: 3.3.8

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by srenon »

I've the same issues
schulmann
Znuny wizard
Posts: 477
Joined: 20 Nov 2011, 16:08
Znuny Version: 6.5.11
Real Name: Schulmann

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by schulmann »

MrAsp wrote:Quick question:
For every 5 concurrent agents 1 additional Apache process
For every Apache process 250 MB of RAM

Where do i change this? Is in the server-tuning.conf as the other settings? In such case, which settings?
In my environment (apache prefork with openSUSE 13.1) it's in the file /etc/apache2/server-tuning.conf:
  • In the section prefork.c: "MaxClients 10"
  • KeepAlive Off
But I think it doesn't solve your problem.

If you connect to your database with tpc/ip: Did you check DNS resolving? tcpdump/wireshark may help.
Znuny6/Debian/ESXi
MrAsp
Znuny newbie
Posts: 7
Joined: 12 Sep 2014, 09:55
Znuny Version: 3.3.8
Real Name: Isak

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by MrAsp »

schulmann wrote:
MrAsp wrote:Quick question:
For every 5 concurrent agents 1 additional Apache process
For every Apache process 250 MB of RAM

Where do i change this? Is in the server-tuning.conf as the other settings? In such case, which settings?
In my environment (apache prefork with openSUSE 13.1) it's in the file /etc/apache2/server-tuning.conf:
  • In the section prefork.c: "MaxClients 10"
  • KeepAlive Off
But I think it doesn't solve your problem.

If you connect to your database with tpc/ip: Did you check DNS resolving? tcpdump/wireshark may help.
Hi
The MaxClients are set to 5 already and KeepAlive is off. And as you say, that doesnt solve my problem.
A normal search still takes minimum 25sec, and the benchmark around 120sec.

The DB-server is stored att localhost, and as far as I can see the DNS-resolving is working as it should.
MrAsp
Znuny newbie
Posts: 7
Joined: 12 Sep 2014, 09:55
Znuny Version: 3.3.8
Real Name: Isak

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by MrAsp »

On the old server the database use MyISAM. On the new InnoDB, can this be the reason why the benchmark take so long time?

http://stackoverflow.com/questions/9819 ... rt-so-slow
schulmann
Znuny wizard
Posts: 477
Joined: 20 Nov 2011, 16:08
Znuny Version: 6.5.11
Real Name: Schulmann

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by schulmann »

Sorry, but I also can't see why your search operation is so slow.
Znuny6/Debian/ESXi
srenon
Znuny newbie
Posts: 60
Joined: 20 Mar 2013, 14:26
Znuny Version: 3.3.8

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by srenon »

try to increase innodb_log_buffer_size in your my.cnf. For me 256M seems to be a good value!

for me it is great...
MrAsp
Znuny newbie
Posts: 7
Joined: 12 Sep 2014, 09:55
Znuny Version: 3.3.8
Real Name: Isak

Re: Upgrade 3.0.5 > 3.3.8, new server Extremely SLOW

Post by MrAsp »

srenon wrote:try to increase innodb_log_buffer_size in your my.cnf. For me 256M seems to be a good value!

for me it is great...
That actually helpt me with the search.. its now around 5-10 seconds faster..big thanks.

The benchmark is still very slow. And I'm worried that adding new tickets will take long time.

Code: Select all

Insert Time:	10000	108 s :-(	Should not take more than 5's on an average system.
Update Time:	10000	121 s :-(	Should not take more than 9's on an average system.
Select Time:	10000	2 s :-)	Looks fine!
Delete Time:	10000	115 s :-(	Should not take more than 5's on an average system.
Multiplier:	* 1	s	
Locked