Problem fetching e-mail.

[ofca]

Since some time now OTRS randomly hangs when fetching e-mail.

It doesn't matter whether POP3 or IMAP is used.

Code: Select all

bin/otrs.Console.pl Maint::PostMaster::MailAccountFetch --force-pid --debug
Fetching incoming mails from mail accounts...

...imap server... (IMAPS)...
(...)
[...cpan-lib/Net/IMAP/Simple.pm line 1221 in sub _read_multiline] [230] (16232) )\r\n
[...cpan-lib/Net/IMAP/Simple.pm line 730 in sub _process_cmd] 3 OK FETCH completed\r\n
[...cpan-lib/Net/IMAP/Simple.pm line 1227 in sub _cmd_ok] 3 OK FETCH completed\r\n

After this it hangs using 100% CPU.

Any pointers as to how to debug this? Removing this one e-mail returns things to working condition for some time, until some other e-mail causes same behavior. I've updated to patch 13, and it didn't help.

[root]

I've updated to patch 13

That's far away from a supported system. Getting into details with such an old version isn't easy. Whats your OS and version?
I really recommend updating at least to 4.0.x, better 5.0.x

[ofca]

Wait, what? OTRS 5 Patch Level 13 is an old system? - I thought 5.0.13 was the newest stable version. OS is FreeBSD, however I tried moving the installation to Debian and same thing happens. I'll try to collect more e-mail that trigger this behavior to find some commonalities, but maybe it happened to someone already?

[root]

Keep cool, you have a 3.1.4 in your signature and did not mention you use OTRS 5. And 3.1.15 still exists.

2016-10-28 at 8.10 PM.png

Is direct SMTP an option for you?

[ofca]

Thanks for noticing. Updated.

Direct SMTP? I thought SMTP is for sending e-mail, not receiving it. Like I've said, I've tried both IMAP and POP3, so it has something to do with contents of the message. Pointer as to what is done after mentioned debug lines happen would be nice.

[ofca]

Anyway, I tried running:
PERLDB_OPTS="N f A L=listing" perl -d bin/otrs.Console.pl Maint::PostMaster::MailAccountFetch --force-pid --debug

and it seems things die here:

672: print STDERR "\nspecials_name => $specials_name\n" if DEBUG;
673: $SWASH->{'SPECIALS'} = \%$specials_name;
675: $SWASH->{'INVERT_IT'} = $invert_it;
678: pop @recursed if @recursed && $type;
680: return $SWASH;

Can't find this anywhere in OTRS codebase, so it must be some external module. But since it happens on both FreeBSD and Debian, and judging by earlier calls, it's probably related to postgresql and UTF-8.

Any ideas? Maybe someone good with perl debugger has some pointers?

[ofca]

After killing the process (with normal SIGTERM), I've noticed some lines were added to 'listing':

Code: Select all

           entering DB::print_lineinfo
         76:            my $MatchedResult = '';
          entering CODE(0x80ae1ef90)
          entering DB::Obj::_init
          entering CODE(0x8022f0300)
          entering CODE(0x8022f0378)
           entering CODE(0x8022f0300)
           entering CODE(0x8022f0300)
          entering DB::Obj::_curr_line
          entering CODE(0x802061108)
          entering CODE(0x8022f0300)
          entering CODE(0x8022f0300)
          entering DB::Obj::_curr_line
          entering CODE(0x802061108)
          entering CODE(0x8022f0180)
          entering CODE(0x8022f0300)
          entering CODE(0x8022ed2b8)
          entering CODE(0x8022f0180)
          entering DB::Obj::_my_print_lineinfo
           entering CODE(0x802061108)
           entering DB::print_lineinfo
         77:            for ( sort keys %Match ) {
          entering CODE(0x80afb4c30)
          entering CODE(0x80afb4c30)
          entering DB::Obj::_init
          entering CODE(0x8022f0300)
          entering CODE(0x8022f0378)
           entering CODE(0x8022f0300)
           entering CODE(0x8022f0300)
          entering DB::Obj::_curr_line
          entering CODE(0x802061108)
          entering CODE(0x8022f0300)
          entering CODE(0x8022f0300)
          entering DB::Obj::_curr_line
          entering CODE(0x802061108)
          entering CODE(0x8022f0180)
          entering CODE(0x8022f0300)
          entering CODE(0x8022ed2b8)
          entering CODE(0x8022f0180)
          entering DB::Obj::_my_print_lineinfo
           entering CODE(0x802061108)
           entering DB::print_lineinfo
         80:                if ( defined $Param{GetParam}->{$_} && $Match{$_} =~ /^EMAILADDRESS:(.*)$/ ) {
          entering CODE(0x80ae1ec48)

This can be located in Kernel/System/PostMaster/Filter/MatchDBSource.pm

Not sure if kill made the code go forward a bit, or if there's some buffering issue here with perl debugger, but well, that's all I have.

[ofca]

Found the bug...

I had multiple postmaster filters that checked body for things like:

Code: Select all

.*V.*p.*g.*b.*H.*s.*m.*A.*W.*x.*c.*m.*r.*L.*4.*q.*X.*n.*j.*

It was an anti-spam measure and it worked great for matching the spam mails, however for reasons unknown it causes perl to sometimes hang. Oh well. I've deleted the filters and e-mail that previously caused perl to hang are now delivered correctly.

[reneeb]

If that's a real example for your regex, perl seems to "hang" most likely for backtracking reasons.

[ofca]

What do you mean?

We've received multitude spam e-mails with URLs containing hashed e-mail address. URL was pretty much the same, but was randomly broken with superficial content (&, #, and other characters that were probably filtered out by spammers after someone clicked the link).

Anyway, that's a real example and this filter worked on thousand of e-mails, with rare exceptions. For all purposes and intents, perl hung. It would process the damn thing at the line I've posted above for hours.

[ofca]

This spam started again.

What's the proper way to filter such mail?

It has URLs such as this in common.
http://cpsinterier.sk/template.php?f=11 ... BQQ&k=ABD2

[jojo]

using a proper mailsystem with spamprotection...

[ofca]

It mostly does the job done, but not in case of these e-mails. SpamAssassin passes them through. URLs are pretty much common for each destination e-mail address, except for minor differences.

[jojo]

you still should create filter rules on the mail server side...