Dashboard Error: This ticket has no title or subject

Moderator: crythias

Post Reply
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Dashboard Error: This ticket has no title or subject

Post by disciple »

i am getting a new error, which I have never seen before, nor my users have seen them before.

getting to the facts:

- I enabled ExternalTicketNumberRecognition 2.5d ago
- In the night at about 2:13 from my change day to the next day (yesterday) I have got tons of tickets all at the same time processed
- all these tickets are NOT received via POP3, they are received via (see below, the Communication Log). I have no clue from where they come
- all these tickets where added to a alphabetically first queue name (makes no sense to all the tickets)
- all these tickets have really no subject
- all these tickets are corrupted, means have no FROM, no TO, body seems to be naked (html or base64 encoded)
- all these tickets look like they have been received during the day (yesterday) and have been deferred on processing into the night
- as of other threads here, I should have checked database (mariaDB), which gives a OK for analyze and for check on all tables
- I did not run "repair" since the checks where ok
- free diskspace is also checked an enough free

Code: Select all

1 		Kernel::System::Console::Command::Maint::PostMaster::Read 	Read email from STDIN. 	08/12/2022 02:13:12
2 		Kernel::System::Console::Command::Maint::PostMaster::Read 	Email with 758 lines successfully read from STDIN. 	08/12/2022 02:13:12
3 		Kernel::System::Console::Command::Maint::PostMaster::Read 	Closing connection from STDIN. 	08/12/2022 02:13:13

so here are multiple questions:

1. where do these tickets come from?
2. is there any system/module in Znuny which explains this odd behavior?
3. why are these ticket corrupted?
4. how to "re-process" them. It costed yesterday already tons of time, to process them manually


best wishes
hkais
Znuny expert
Posts: 280
Joined: 16 Apr 2016, 08:55
Znuny Version: see in post
Real Name: Hans
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by hkais »

Can you check if you find something like in your znuny.log / otrs.log

Code: Select all

PostMaster::SpoolMailsReprocess 
and 
Successfully reprocessed email /opt/otrs/var/spool/problem-email
Elected 2022-06 as an IT Governance Portal Expert. The portal for Znuny, OTRS and OTOBO users
hkais
Znuny expert
Posts: 280
Joined: 16 Apr 2016, 08:55
Znuny Version: see in post
Real Name: Hans
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by hkais »

very probably something in the log like

Code: Select all

Could not process message. Raw mail saved
If so, you have misconfigured your ExternalTicketNumberRecognition


One hint from many years of usage:

avoid using anything around PostmasterFIlter, like ExternalTicketNumberRecognition

consider the real life example how the postmaster filter works:
- You get a envelope
- you open the envelope
- you read the mail
- you modify the mail and/or envelope (header, content, meta, whatever)
- you pass it to your officials in the team (in the team noone know that you modified the mail)

This is how the PostmasterFilters works.
You would never do such an approach in real life, right? But this is the PostmasterFilter

The second and major thing is, if you do a mistake at the level of Postmasterfilter you can hardly reproduce it.
AND, if you do a mistake, often (like in your case) the consequences are very high. Means high effort to fix the issue

Do no use PostmasterFilters, try to use a Generic Agent
1. easy to test, by simulating tickets
2. easy to reproduce (you cannot ask your businesspartner to resend some mails, if your postmasterfilter breaks)
3. other non IT-guys like power users can handle the Generic Agent on their own (and test them, or disable them, or whatever)


Report a bug
And finally, it seems the processing of either
- saving the mail to spool
or
- reprocessing the mail from spool
has a bug, if your e-mail fields like FROM, TO and so on are missing
Last edited by hkais on 26 Aug 2022, 14:49, edited 2 times in total.
Elected 2022-06 as an IT Governance Portal Expert. The portal for Znuny, OTRS and OTOBO users
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Re: Dashboard Error: This ticket has no title or subject

Post by disciple »

yes exactly my issues

Code: Select all

[Thu Aug 11 08:20:21 2022][Error][Kernel::System::MailAccount::POP3::Fetch][486] Exception while processing mail: Quantifier follows nothing in regex; marked by <-- HERE in m/* <-- HERE @example.com/ at /opt/znu
ny-6.4.2/Kernel/System/PostMaster/Filter/ExternalTicketNumberRecognition.pm line 63.

[Thu Aug 11 08:20:21 2022][Error][Kernel::System::CommunicationLog::_LogError][535] CommunicationLog(ID:1708815,AccountType:-,AccountID:-,Direction:Incoming,Transport:Email,ObjectLogType:Message,ObjectLogID:173
9563)::Kernel::System::MailAccount::POP3TLS => Could not process message. Raw mail saved (/opt/otrs/var/spool/problem-email-de3e20cac513ada55172a2eca9e6c9d6, report it on https://github.com/znuny/Znuny/issues)!
[Thu Aug 11 08:20:22 2022][Error][Kernel::System::MailAccount::POP3::Fetch][486] Exception while processing mail: Quantifier follows nothing in regex; marked by <-- HERE in m/* <-- HERE @example.com/ at /opt/znu
ny-6.4.2/Kernel/System/PostMaster/Filter/ExternalTicketNumberRecognition.pm line 63.
and this night

Code: Select all

[Fri Aug 12 02:13:28 2022][Info][Kernel::System::Console::Command::Maint::PostMaster::SpoolMailsReprocess::Run] Successfully reprocessed email /opt/otrs/var/spool/problem-email-de3e20cac513ada55172a2eca9e6c9d6.
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Re: Dashboard Error: This ticket has no title or subject

Post by disciple »

can I somehow fix the broken tickets? yesterday it took 3 person over 1h each to fix the half amount of tickets..

Means can I rerun the recognition of the fields, so that my tickets are not all corrupted and repaired?
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Re: Dashboard Error: This ticket has no title or subject

Post by disciple »

We had to go through all ticket.
Took 3 persons the full morning till now :(

Will be very careful in the future about Postmaster changes.
Johannes
Moderator
Posts: 391
Joined: 30 Jan 2008, 02:26
Znuny Version: All of them ^^
Real Name: Hannes
Company: Znuny|OTTERHUB

Re: Dashboard Error: This ticket has no title or subject

Post by Johannes »

disciple wrote: 12 Aug 2022, 10:50 yes exactly my issues

Code: Select all

[Thu Aug 11 08:20:21 2022][Error][Kernel::System::MailAccount::POP3::Fetch][486] Exception while processing mail: Quantifier follows nothing in regex; marked by <-- HERE in m/* <-- HERE @example.com/ at /opt/znu
ny-6.4.2/Kernel/System/PostMaster/Filter/ExternalTicketNumberRecognition.pm line 63.

[Thu Aug 11 08:20:21 2022][Error][Kernel::System::CommunicationLog::_LogError][535] CommunicationLog(ID:1708815,AccountType:-,AccountID:-,Direction:Incoming,Transport:Email,ObjectLogType:Message,ObjectLogID:173
9563)::Kernel::System::MailAccount::POP3TLS => Could not process message. Raw mail saved (/opt/otrs/var/spool/problem-email-de3e20cac513ada55172a2eca9e6c9d6, report it on https://github.com/znuny/Znuny/issues)!
[Thu Aug 11 08:20:22 2022][Error][Kernel::System::MailAccount::POP3::Fetch][486] Exception while processing mail: Quantifier follows nothing in regex; marked by <-- HERE in m/* <-- HERE @example.com/ at /opt/znu
ny-6.4.2/Kernel/System/PostMaster/Filter/ExternalTicketNumberRecognition.pm line 63.
and this night

Code: Select all

[Fri Aug 12 02:13:28 2022][Info][Kernel::System::Console::Command::Maint::PostMaster::SpoolMailsReprocess::Run] Successfully reprocessed email /opt/otrs/var/spool/problem-email-de3e20cac513ada55172a2eca9e6c9d6.
Sorry to hear that things went south for you and your team.

But:
You have a simple config issue in your filter. And I really can't second the opinion of hkais. This is more than questionable. If you don't know how to use them, learn it or ask. It's not that hard. If you try the hard way, you get the hard way. Thats the same for basically everything.
You don't know how to use an oven. Don't try, use the microwave. :?

Filters are made to be used, you just have to check them for errors after you deploy them.
It is more a question of change management not of "never use filters".
Would be interesting how you to build the external ticket number recognition using a generic agent.


back to topic:
I would bet, that you have a mistake in your regex.
System Configuration Group - Admin - znuny-6040 2022-08-12 16-08-00.png
You used a "*" without any modifier like ".*" or similar, this is not valid and throws an error.

Every mail that matches this filter gets an error and is parked in ../var/spool and are reprocessed in the night.
At least the systems tries too. Because failures in mail handling can happen al the time.

To sum things up:
- Use Filters. Use them a lot and learn to use them
- Use the basics of change and release management -> Use a test instance, verify, deploy to prod
- Visit a course, read the docs (if available) or by a book to understand those topics
- Ask upfront and wait for answers (we have a discord to, for a more instant help)

If you still need help, let me know.


p.s.
I'm not able to check your setup, but if this happens again.
Check var/spool. If the mails are still there, delete the empty mails using a generic agent.
Fix your filter and reprocess them again. Saves a lot of manual labor.
You do not have the required permissions to view the files attached to this post.
hkais
Znuny expert
Posts: 280
Joined: 16 Apr 2016, 08:55
Znuny Version: see in post
Real Name: Hans
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by hkais »

hkais wrote: 12 Aug 2022, 10:44 Report a bug
And finally, it seems the processing of either
- saving the mail to spool
or
- reprocessing the mail from spool
has a bug, if your e-mail fields like FROM, TO and so on are missing
can you clarify this behaviour?

IMHO if an untouched e-mail is getting serialized into a file, the reprocessing of this mail should NEVER loose details like

- subject
- from
- to
- mail structure

and of course the envelope header fields
Elected 2022-06 as an IT Governance Portal Expert. The portal for Znuny, OTRS and OTOBO users
Johannes
Moderator
Posts: 391
Joined: 30 Jan 2008, 02:26
Znuny Version: All of them ^^
Real Name: Hannes
Company: Znuny|OTTERHUB

Re: Dashboard Error: This ticket has no title or subject

Post by Johannes »

I can,

the error is triggered because of a wrong regex. This brings the filter to fail. And before something worse than empty tickets happens, the filter stops, returns an error code to the postmaster process, which then moves mails to the spool folder.

The result is pretty straight forward. The processing has to stop. Because no one knows what the filter is actually doing / needed for.

This can surely be discussed if a feature is created like a flag "allowed to fail" but as I wrote before. Don't bring your instance to this state in the "first place" is the way to go. Especially when you don't know what / how to do.

The E-Mail is not lost, the content is not lost. Just the ticket creation fails. But we need a ticket id to perform the needed changes. Is more like hen & egg problem. Ticket has to be there to store "external" information. Processing fails, ticket is still there, maybe empty, but this can easily be resolved by:
1) fixing the filter
2) delete old tickets with those criteria (e.g. time created)
3) reprocess

All you "lose" is the real create time stamp

Hope this clears things up for you
jojo
Znuny guru
Posts: 15019
Joined: 26 Jan 2007, 14:50
Znuny Version: Git Master
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by jojo »

hkais wrote: 12 Aug 2022, 10:44 consider the real life example how the postmaster filter works:
- You get a envelope
- you open the envelope
- you read the mail
- you modify the mail and/or envelope (header, content, meta, whatever)
- you pass it to your officials in the team (in the team noone know that you modified the mail)
Thats not correct, the postmaster filter adds new headers, but does not alter the content from a mail.
"Production": OTRS™ 8, OTRS™ 7, STORM powered by OTRS
"Testing": ((OTRS Community Edition)) and git Master

Never change Defaults.pm! :: Blog
Professional Services:: http://www.otrs.com :: enjoy@otrs.com
jojo
Znuny guru
Posts: 15019
Joined: 26 Jan 2007, 14:50
Znuny Version: Git Master
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by jojo »

hkais wrote: 12 Aug 2022, 10:44
Do no use PostmasterFilters, try to use a Generic Agent
1. easy to test, by simulating tickets
2. easy to reproduce (you cannot ask your businesspartner to resend some mails, if your postmasterfilter breaks)
3. other non IT-guys like power users can handle the postmaster filter on their own (and test them, or disable them, or whatever)
Thats also not the best suggestion. Generic Agents will use up much more resources and they also do not support regular expressions
"Production": OTRS™ 8, OTRS™ 7, STORM powered by OTRS
"Testing": ((OTRS Community Edition)) and git Master

Never change Defaults.pm! :: Blog
Professional Services:: http://www.otrs.com :: enjoy@otrs.com
hkais
Znuny expert
Posts: 280
Joined: 16 Apr 2016, 08:55
Znuny Version: see in post
Real Name: Hans
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by hkais »

jojo wrote: 13 Aug 2022, 09:58 Thats also not the best suggestion. Generic Agents will use up much more resources and they also do not support regular expressions
cannot confirm this, if you use event based handling.
E.g. on TicketCreate. It may consume a few more CPU and IO cycles but not really relevant compared to the gains won.
jojo wrote: 13 Aug 2022, 09:57 Thats not correct, the postmaster filter adds new headers, but does not alter the content from a mail.
you are right, some headers are extended, but as I have noted before, you cannot identify them in the later processing. And this behavior is like changing a mail or/and the envelop in real life, without letting later readers know
Especially you have no real way to test, change, retest without recreating new tickets
Johannes wrote: 12 Aug 2022, 20:23 I can,

the error is triggered because of a wrong regex. This brings the filter to fail. And before something worse than empty tickets happens, the filter stops, returns an error code to the postmaster process, which then moves mails to the spool folder.
fully confirm to this, but a mail delivery and the later processing should be IMHO to separate steps
Johannes wrote: 12 Aug 2022, 20:23 The E-Mail is not lost, the content is not lost. Just the ticket creation fails. But we need a ticket id to perform the needed changes. Is more like hen & egg problem. Ticket has to be there to store "external" information. Processing fails, ticket is still there, maybe empty, but this can easily be resolved by:
yes, if decoupled you will not loose anything.

Also here it sounds for me like a bug, if parts of the message on the reprocessing are getting lost
The report is, that e.g. the subject is missing. And if a clean reprocessing is taking in place, how can the subject be lost?


To the other question, we have implemented a GA, which is capable of detecting external tickets and merge them accordingly.
The big difference:
We can rerun the "detection" and the "merging" independently at any time and rerun how often we need.
Without recreating any additional tickets...

If there is interest in the community I may ask for a release/publishing of the code.
Elected 2022-06 as an IT Governance Portal Expert. The portal for Znuny, OTRS and OTOBO users
jojo
Znuny guru
Posts: 15019
Joined: 26 Jan 2007, 14:50
Znuny Version: Git Master
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by jojo »

hkais wrote: 15 Aug 2022, 22:07
jojo wrote: 13 Aug 2022, 09:58 Thats also not the best suggestion. Generic Agents will use up much more resources and they also do not support regular expressions
cannot confirm this, if you use event based handling.
E.g. on TicketCreate. It may consume a few more CPU and IO cycles but not really relevant compared to the gains won.
Checking all filters on the same event .e.g TicketCreate will cause a huge load on large systems an can even lock the system. Also the processing of the E-Mail willl be done after e.g. a Ticket or an Article was creates. The PostMaster Filters are in the preporcessing. Much faster, much more relieable and also will work on each Mail and may even cause not to trigger a TicketCreate due to the possibility to ignore Mails.

We have systems with morge than 100.000 incoming mails a day. Postmaster Filters are indentend to do the work.
hkais wrote: 15 Aug 2022, 22:07
jojo wrote: 13 Aug 2022, 09:57 Thats not correct, the postmaster filter adds new headers, but does not alter the content from a mail.
you are right, some headers are extended, but as I have noted before, you cannot identify them in the later processing. And this behavior is like changing a mail or/and the envelop in real life, without letting later readers know
Especially you have no real way to test, change, retest without recreating new tickets
The communication log contains all needed information. A "normal" reader typically has no interest in such data. They just want to have the mail/ticket on the right place.
"Production": OTRS™ 8, OTRS™ 7, STORM powered by OTRS
"Testing": ((OTRS Community Edition)) and git Master

Never change Defaults.pm! :: Blog
Professional Services:: http://www.otrs.com :: enjoy@otrs.com
hkais
Znuny expert
Posts: 280
Joined: 16 Apr 2016, 08:55
Znuny Version: see in post
Real Name: Hans
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by hkais »

jojo wrote: 16 Aug 2022, 07:18 Checking all filters on the same event .e.g TicketCreate will cause a huge load on large systems an can even lock the system. Also the processing of the E-Mail willl be done after e.g. a Ticket or an Article was creates. The PostMaster Filters are in the preporcessing. Much faster, much more relieable and also will work on each Mail and may even cause not to trigger a TicketCreate due to the possibility to ignore Mails.

We have systems with morge than 100.000 incoming mails a day. Postmaster Filters are indentend to do the work.
I am sure you have tons of more knowledge on huge Znuny installations, no interest in discussing about this.

But be sure in my area we work with really huge enterprise systems too.

and taking your number 100k tickets
results into typically 100k DB transactions

results with a "short" day of 10h into
10k transactions/h
167 transaction/min
2.8 transactions/s

We work on systems with 1k+ transactions per second. So 3 transactions per second is nothing for a well organized database and well structured schema like the Znuny schema is.
And if this amount of transactions create a table locks, which it looks like you refer to, the DB locking is badly misconfigured...

So even if we take 1M DB transactions, respectively 1.000.000 tickets in a 10h day, we still are in a lower 2 figure transaction count ( 27.8 ) per second. Even somehow good configured mysqls can handle that properly...

But in sum you are right, that a simple text operations with PostmasterFilter is more CPU and disk efficient. But from total efficiency point incl. work time, which costs usually factors of CPU or IO, the consideration is IMHO a different one.
Elected 2022-06 as an IT Governance Portal Expert. The portal for Znuny, OTRS and OTOBO users
Johannes
Moderator
Posts: 391
Joined: 30 Jan 2008, 02:26
Znuny Version: All of them ^^
Real Name: Hannes
Company: Znuny|OTTERHUB

Re: Dashboard Error: This ticket has no title or subject

Post by Johannes »

To bring this back on track for the OP and stop the thread steal:

@disciple
Is the problem in the filter resolved? Are mails processed in the correct way?
hkais
Znuny expert
Posts: 280
Joined: 16 Apr 2016, 08:55
Znuny Version: see in post
Real Name: Hans
Contact:

Re: Dashboard Error: This ticket has no title or subject

Post by hkais »

Johannes wrote: 16 Aug 2022, 12:23 To bring this back on track for the OP and stop the thread steal:

@disciple
Is the problem in the filter resolved? Are mails processed in the correct way?
you are right, we stole mutually the thread

@disciple: you can take a look here
use dynamic fields to link or merge tickets
maybe this will help you for the future tasks?
Elected 2022-06 as an IT Governance Portal Expert. The portal for Znuny, OTRS and OTOBO users
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Re: Dashboard Error: This ticket has no title or subject

Post by disciple »

@all, thx for the help
I was a while off, due to unplanned additional effort caused by this issue.

We had to cleanup the mess created by the wrong PostmasterFilter config I created. So now we workarounded the issue, but it is not fixed.

as hkais wrote, I do not understand why my messages got totally corrupted.
body was mostly base64 in ticket view
subject was missing
header fields from, to have also being missing.

So it took us a lot of time to fix all these manually. My mates have been pretty unhappy, not to use another bad word they have said...


@hkais, your links look good. will message you on your posts
Johannes
Moderator
Posts: 391
Joined: 30 Jan 2008, 02:26
Znuny Version: All of them ^^
Real Name: Hannes
Company: Znuny|OTTERHUB

Re: Dashboard Error: This ticket has no title or subject

Post by Johannes »

Hi,

can you post a screenshot of your config please. I help you to get it right.

Regards
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Re: Dashboard Error: This ticket has no title or subject

Post by disciple »

Johannes wrote: 16 Aug 2022, 22:08 can you post a screenshot of your config please. I help you to get it right.
you mean the postmasterfilter regex for external ticket detection?

As long as I have no clearance why tickets got messed up, for me the postmasterfilter is not an option to use.
I understand it was initially my mistake, but I cannot accept (nor my collegues) that a simple "search filter mistake" is breaking tons of emails and all email details (subject, body, from to) are lost
Here I see it as hkais. I will skip this feature until it is fixed or I found a better approach.
Anyway many thanks for your help here

If you need the config for fixing the reason of the message mess up, I can share you the regex.
Johannes
Moderator
Posts: 391
Joined: 30 Jan 2008, 02:26
Znuny Version: All of them ^^
Real Name: Hannes
Company: Znuny|OTTERHUB

Re: Dashboard Error: This ticket has no title or subject

Post by Johannes »

Hi,

ok. No, I just wanted to help.
I already know what the error is / was.

Thanks
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Re: Dashboard Error: This ticket has no title or subject

Post by disciple »

Johannes wrote: 17 Aug 2022, 16:39 Hi,

ok. No, I just wanted to help.
I already know what the error is / was.

Thanks
what was the error of the message format mess up?
Johannes
Moderator
Posts: 391
Joined: 30 Jan 2008, 02:26
Znuny Version: All of them ^^
Real Name: Hannes
Company: Znuny|OTTERHUB

Re: Dashboard Error: This ticket has no title or subject

Post by Johannes »

Just, a wild guess based on you error message, because you never posted your actual config.

The issue was|is a wrong regex.

correct wildcard

Code: Select all

.*
vs
your version

Code: Select all

*
In you filter config somewhere. I think you wanted to use a wildcard and missed the error in the regex.
disciple
Znuny newbie
Posts: 62
Joined: 31 Jul 2022, 12:27
Znuny Version: 6.4.2
Real Name: David Müller

Re: Dashboard Error: This ticket has no title or subject

Post by disciple »

okay now got it.
But thank you for clarification and for your time.


I had a mess up in the e-mail structure, and this can be caused by the regex wrongly created by the user
I thought it maybe a bug, why the regex matching was messing up the e-mail structure.

I will stay on my possition to avoid using postmasterFilters, since the consequences are too criritical too me. And my colleagues have been pretty pi..ed to me
My expectation here would be that the regex fails, but that the message structure is not getting messed up
Johannes
Moderator
Posts: 391
Joined: 30 Jan 2008, 02:26
Znuny Version: All of them ^^
Real Name: Hannes
Company: Znuny|OTTERHUB

Re: Dashboard Error: This ticket has no title or subject

Post by Johannes »

As often in life. Expectation and reality differ some times.
We can't catch every error. And as I wrote before. Some things need to be processed / created before one can work on it.
Your filter could also be mission critical and has to run. Maybe to sort things in specific queues, creates links, take information out of it and trigger a webservice.

So the system goes to failsafe. Put the mails to spool, tell the admin in the log-> a fix is needed, something is wrong.
And then you can reprocess everything using the spool job.

You can always see this behaviour both ways. But an empty ticket, is less than a problem than a complete ticket which was not processed correctly and maybe triggers the wrong actions / gives information to people who should not have them.

The solution would have been simple, as describe. The use of a proper staging would be even better. Try and error and prod will always lead to such kind of issues.
Post Reply