Znuny is slow and traversing my messages

Moderator: crythias

Post Reply
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Znuny is slow and traversing my messages

Post by RoyK »

Hi all

I just upgraded from OTRS 5 to 6 and then to Znuny 6.0.36 and if I now try to visit https://znuny.my.site/otrs/index.pl?Act ... aCollector, some process only identified as /opt/otrs/bin/c (along the httpd process) keeps using tons of cpu and I/O and according to strace, is calling stat() and lstat() for every file under /opt/otrs/var/article/. Since there are 13 millions of them, this takes some time. To me, this looked like something not indexed properly, so I ran 'bin/otrs.Console.pl Maint::Ticket::FulltextIndex --rebuild' and it finished in two seconds.

Any idea how to debug this further?
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

I tried to reinstalled on a new machine to eliminate potential issues during the installation and it I see the same happening there.
Reedu
Znuny newbie
Posts: 29
Joined: 11 Mar 2013, 15:08
Znuny Version: Znuny 6.1.2
Real Name: Adam

Re: Znuny is slow and traversing my messages

Post by Reedu »

When updating from 5 to 6 did you restore Kernel/Config/Files/ZZZAuto.pm ?
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

Reedu wrote: 13 Oct 2021, 12:26 When updating from 5 to 6 did you restore Kernel/Config/Files/ZZZAuto.pm ?
yes
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

I tried to reindex everything and it didn't help.
The reinstallation didn't get as far as the upgrade to znuny, so I guess this is an OTRS issue. Still, I have no idea where to look, really, since OTRS is out of support and I'm not migrated yet.
skullz
Znuny superhero
Posts: 618
Joined: 24 Feb 2012, 03:58
Znuny Version: LTS and Features
Real Name: Mo Azfar
Location: Kuala Lumpur, MY
Contact:

Re: Znuny is slow and traversing my messages

Post by skullz »

What db ? Postgresql?
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

MariaDB so far. I'm considering moving to PostgreSQL, but I doubt it'll help me on this alone. I doubt the bug is in the db connection.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: Znuny is slow and traversing my messages

Post by root »

Hi,

Did you tried the AdminDataCollector before upgrading with a better result?

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

I have tried it again several times with the old install without issues
Reedu
Znuny newbie
Posts: 29
Joined: 11 Mar 2013, 15:08
Znuny Version: Znuny 6.1.2
Real Name: Adam

Re: Znuny is slow and traversing my messages

Post by Reedu »

Isn't

Code: Select all

Maint::Ticket::FulltextIndex  -- rebuild 
used only to flag it?

Look at

Code: Select all

Maint::Ticket::FulltextIndexRebuildWorker

and it's parameters.

Another solution could be moving from FS to DB.
Ticket::Article::Backend::MIMEBase::ArticleStorage
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

With 2594180 articles, the docs are quite clear that I should use fs storage. Any change on that?

Also, from the support data collector

Information Article Search Index Status
Information Indexed Articles 100.0 % (2594180/2594180)

also, ps axfww just shows me this when I look for the process. Which script can it be that's traversing the directory structure? Why is it doing it? Shouldn't it use the index?

130325 ? Dl 2:03 \_ /opt/otrs/bin/c -DFOREGROUND
hkais
Znuny expert
Posts: 280
Joined: 16 Apr 2016, 08:55
Znuny Version: see in post
Real Name: Hans
Contact:

Re: Znuny is slow and traversing my messages

Post by hkais »

RoyK wrote: 12 Oct 2021, 11:45 strace, is calling stat() and lstat() for every file under /opt/otrs/var/article/. Since there are 13 millions of them, this takes some time. To me, this looked like something not indexed properly, so I ran 'bin/otrs.Console.pl Maint::Ticket::FulltextIndex --rebuild' and it finished in two seconds.
looks for me you switched the filesystem type also.

what was your filesystem of /opt/otrs/var/article/ before you migrated and what is it now?
Had a similar issue with many files on an environment too (not OTRS/Znuny)
Elected 2022-06 as an IT Governance Portal Expert. The portal for Znuny, OTRS and OTOBO users
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

The old machine runs rhel6 and ext4. the new one has rhel8 and was originally setup with xfs. this was later changed to ext4, since xfs is horribly slow on metadata operations compared to ext4. I can't see the old system doing any massive traversal of the articles when I visit the mentioned link. It takes some seconds to open the link, sure, but not half an hour. The underlying storage is also the same, a Dell Compellent system with about half a peta on HDDs and a caching layer of 40TB on SSDs and the problem is not I/O, but something in OTRS/znuny.
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

Anyone here that might know what can cause this? It seems it's not using the indices at all. That is, perhaps it's reading through one or more index first, but then it still traverses the data.
schulmann
Znuny wizard
Posts: 470
Joined: 20 Nov 2011, 16:08
Znuny Version: 6.5.4
Real Name: Schulmann

Re: Znuny is slow and traversing my messages

Post by schulmann »

RoyK wrote: 14 Oct 2021, 12:35 With 2594180 articles, the docs are quite clear that I should use fs storage. Any change on that?
Sometimes it's better not to follow the docs.
But then you should know what you are doing.

In our case there are spare CPU cycles so we can use up to 16 CPUs in parallel.

We decided to store the attachments in the PostgreSQL database and to use the same server for the ticket system and for the database.
We have more than a million articles.

To open a new phone ticket or to open a random ticket needs less than 500 ms.

We disabled index search (Ticket::SearchIndex::ForceUnfilteredStorage is enabled).
The average fulltext search time is less than 5 seconds.

We ignore some warnings:
  • You have more than 60,000 tickets and should use the StaticDB backend. See admin manual (Performance Tuning) for more information.
  • The indexing process forces the storage of the original article text in the article search index, without executing filters or applying stop word lists. This will increase the size of the search index and thus may slow down fulltext searches.
Znuny6/Debian/ESXi
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: Znuny is slow and traversing my messages

Post by root »

Hi,

I double checked the source and did not find any change related to this. Do you changed anything related to the file/folder structure?

- Roy
RoyK wrote: 14 Oct 2021, 12:35 With 2594180 articles, the docs are quite clear that I should use fs storage. Any change on that?

Also, from the support data collector

Information Article Search Index Status
Information Indexed Articles 100.0 % (2594180/2594180)

also, ps axfww just shows me this when I look for the process. Which script can it be that's traversing the directory structure? Why is it doing it? Shouldn't it use the index?

130325 ? Dl 2:03 \_ /opt/otrs/bin/c -DFOREGROUND
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

The mountpoint for the articles is changed. For some reason, the old install used /home/var/article - possibly someone didn't know how mounting should be done in linux, so I moved it back to /opt/otrs/var/article. I have gone through all of the config and changed. I've even searched through the database in search for anything there.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: Znuny is slow and traversing my messages

Post by root »

RoyK wrote: 22 Oct 2021, 16:44 The mountpoint for the articles is changed. For some reason, the old install used /home/var/article - possibly someone didn't know how mounting should be done in linux, so I moved it back to /opt/otrs/var/article. I have gone through all of the config and changed. I've even searched through the database in search for anything there.
Hi,

This might be the reason. The system is searching for files which not belong to the system when creating a support assessment. I recommend moving this directory to somewhere outside /opt/otrs and configure the system configuration Ticket::Article::Backend::MIMEBase::ArticleDataDir properly

-Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
RoyK
Znuny newbie
Posts: 20
Joined: 12 Oct 2021, 11:10
Znuny Version: 6.0.36
Real Name: Roy Sigurd Karlsbakk
Company: OsloMet

Re: Znuny is slow and traversing my messages

Post by RoyK »

Thanks, Roy. It has taken me quite a while to try to sort this out and I'm really glad this was the issue. I moved the articles out to /opt/otrs_articles and *poff*, this problem disappeared.

btw, I beleive there was a benchmark of sorts in otrs to check for total system performance. Since I'm in the middle of migrating/upgrading, I can probably postpone that a wee bit to test further, be it file or db storage or whatever. Is there such a thing as a builtin benchmark?
Post Reply