Hello,
we are running Znuny 6.5.4 and are facing a strange issue, that only occurs with a small number of seemingly random tickets.
The issues has existed for a while and was already present in some OTRS versions, before we migrated to Znuny.
When clicking on affected ticket entries, the browser starts loading the view and shows the message content.
However, the page does not finish loading and Apache2 stops handling requests all together, so all subsequent requests fail.
Restarting the Apache2 service makes Znuny work again, but the issue can be reproduced by opening the same affected ticket entry again.
It seems to be random which tickets are affected, but they all have this in common:
- HTML emails
- Formatted signature/footer with links
- Images in the message, which are embedded
When the issue occurs, if often does after a few emails have been exchanged, meaning the first entries of the ticket can be viewed without issues, but after the issue occurred, all later entries will cause Apache2 to get stuck.
Each message of course contains the previous one below, meaning some bit of content is introduced at some point and kept in the message, which Znuny/Apache2 has trouble handling.
To narrow it down we have set up Nginx on the same server, running on a different port.
When using Nginx to view the affected tickets, there is no issue. However, Nginx is much slower at serving OTRS/Znuny than Apache2, so we don't want to switch.
The Apache2 debug log level and the Znuny performance log have given us no clue so far.
Also inspecting the message source and comparing the HTML of affected and not affected messages has not shown any obvious difference.
I hope anyone here has an idea and can suggest things we can try to narrow it down further or even solve it.
Thanks a lot in advance!
Best regards
Alexander
Apache stops working when viewing specific tickets
Moderator: crythias
-
- Znuny superhero
- Posts: 629
- Joined: 24 Feb 2012, 03:58
- Znuny Version: LTS and Features
- Real Name: Mo Azfar
- Location: Kuala Lumpur, MY
- Contact:
Re: Apache stops working when viewing specific tickets
Do also check browser console log
See whatever script / files / network that got stuck..
See whatever script / files / network that got stuck..
My Github
OTRS CE/LTS Discord Channel
Cant Update Package Anymore ? Check This
Professional OTRS, Znuny & OTOBO services: efflux.de/en
Free and premium add-ons: English
OTRS CE/LTS Discord Channel
Cant Update Package Anymore ? Check This
Professional OTRS, Znuny & OTOBO services: efflux.de/en
Free and premium add-ons: English
-
- Administrator
- Posts: 3976
- Joined: 18 Dec 2007, 12:23
- Znuny Version: Znuny and Znuny LTS
- Real Name: Roy Kaldung
- Company: Znuny
- Contact:
Re: Apache stops working when viewing specific tickets
Hi,
The Performance Log won't help here. If you can identify the tickets that cause the problem, it might be a problem during parsing for displaying.
This might be related to sanitizing the content before displaying it to prevent XSS and other attacks. With the causing email, we could probably fix this issue.
- root
The Performance Log won't help here. If you can identify the tickets that cause the problem, it might be a problem during parsing for displaying.
This might be related to sanitizing the content before displaying it to prevent XSS and other attacks. With the causing email, we could probably fix this issue.
- root
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO
Use a test system - always.
Do you need professional services? Check out https://www.znuny.com/
Do you want to contribute or want to know where it goes ?
Use a test system - always.
Do you need professional services? Check out https://www.znuny.com/
Do you want to contribute or want to know where it goes ?
Re: Apache stops working when viewing specific tickets
Thanks for the suggestion.
We had a look at the browser console and apache2 logs before, but could not see any obvious differences between loading entires with and without the issue.
Browser console for a ticket entry without issues:
- CommonJS_38d6c19c5c4a4843df4a8adac8a4fb86.js:1 Error while parsing the 'sandbox' attribute: 'ms-allow-popups' is an invalid sandbox flag.
- index.pl:1 Unrecognized Content-Security-Policy directive 'referrer'.
- index.pl:100 GET https://*****/otrs/image002.jpg@01D698A1.D5B6C550 404 (Not Found)
- index.pl:76 GET https://*****/otrs/image005.png@01D9987C.215474D0 404 (Not Found)
- index.pl:100 GET https://*****/otrs/image003.jpg@01D698A1.D5B6C550 404 (Not Found)
Browser console for a ticket entry with issues:
- CommonJS_38d6c19c5c4a4843df4a8adac8a4fb86.js:1 Error while parsing the 'sandbox' attribute: 'ms-allow-popups' is an invalid sandbox flag.
- /otrs/index.pl?Action=AgentTicketArticleContent;Subaction=HTMLView;TicketID=*****;ArticleID=*****;FileID=;:1 Unrecognized Content-Security-Policy directive 'referrer'.
- index.pl:392 GET https://*****/otrs/image002.jpg@01D698A1.D5B6C550 404 (Not Found)
When the issue occurs, Znuny shows the following message and some request remain pending, as the webserver stops responding.
Znuny detected possible network issues. You could either try reloading this page manually or wait until your browser has re-established the connection on its own.
It makes no difference which browser and OS is used. We have tried Windows with Chrome, Firefox, Edge, Opera, and Mac with Safari, Chrome and Firefox.
Re: Apache stops working when viewing specific tickets
Thanks for your reply.root wrote: ↑05 Dec 2023, 13:25 Hi,
The Performance Log won't help here. If you can identify the tickets that cause the problem, it might be a problem during parsing for displaying.
This might be related to sanitizing the content before displaying it to prevent XSS and other attacks. With the causing email, we could probably fix this issue.
- root
I have now tried to load every single URL that is requested when loading a ticket entry having issues and could narrow it down further.
The issue can only be reproduced when loading the article content and never when loading the attachments.
Example:
https://*****/otrs/index.pl?Action=AgentTicketArticleContent;Subaction=HTMLView;TicketID=*****;ArticleID=*****;FileID=
I have copied the content of the ArticleID in the filesystem and assume the issue must be related to how the HTML is parsed.
As the email contains sensitive information I can't share it here. I will try to reproduce the issue with a dummy email, containing the same HTML.
If that works, I will afterwards do the same with fragments of it, is I can isolate the troublesome parts and share them here.
Or would other steps make more sense?
-
- Administrator
- Posts: 3976
- Joined: 18 Dec 2007, 12:23
- Znuny Version: Znuny and Znuny LTS
- Real Name: Roy Kaldung
- Company: Znuny
- Contact:
Re: Apache stops working when viewing specific tickets
Hi,
These steps are fine so far. If you have any chance to reproduce it without sharing sensitive information let us know.
Thank you.
- Roy
These steps are fine so far. If you have any chance to reproduce it without sharing sensitive information let us know.
Thank you.
- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO
Use a test system - always.
Do you need professional services? Check out https://www.znuny.com/
Do you want to contribute or want to know where it goes ?
Use a test system - always.
Do you need professional services? Check out https://www.znuny.com/
Do you want to contribute or want to know where it goes ?
-
- Administrator
- Posts: 3976
- Joined: 18 Dec 2007, 12:23
- Znuny Version: Znuny and Znuny LTS
- Real Name: Roy Kaldung
- Company: Znuny
- Contact:
Re: Apache stops working when viewing specific tickets
Please create a issue at GitHub if you were able to anonymize the email.awietz wrote: ↑06 Dec 2023, 10:54
I have copied the content of the ArticleID in the filesystem and assume the issue must be related to how the HTML is parsed.
As the email contains sensitive information I can't share it here. I will try to reproduce the issue with a dummy email, containing the same HTML.
If that works, I will afterwards do the same with fragments of it, is I can isolate the troublesome parts and share them here.
- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO
Use a test system - always.
Do you need professional services? Check out https://www.znuny.com/
Do you want to contribute or want to know where it goes ?
Use a test system - always.
Do you need professional services? Check out https://www.znuny.com/
Do you want to contribute or want to know where it goes ?