Open Communications/Failing Fetchmails

[RichH]

Znuny 6.4
CentOS Linux release 7.9.2009
MariaDB 5.5.68
Perl 5.16.3
O365 Mailboxes using OAuth2 for fetchmail.

Hi,

In the Znuny Communication Log we have 89 open communications. The oldest one has been open for more than 8 hours.

I expected a timeout of 600 seconds.

When I run the sql query below, I get 361 results dating back to early yesterday.

select * from communication_log
where
status = 'Processing'
and
direction = 'Incoming'

Yesterday we stopped receiving email in several queues. I had to reboot the server, stop the Daemon, edit the crontab to stop the Daemon from restarting, open the mailboxes, move all emails out of the Inbox, then move 10 emails at a time back into the Inbox and manually run Fetchmail via the Znuny Admin interface.

Each of the problematic mailboxes has more than 50 emails in the Inbox.

The Crontab is set to run the Daemon every 5 minutes.

I don't know if the Open Communications are causing a problem, but is there a way to shut them down manually?

Would it help to increase or decrease the Daemon startup interval in the Crontab?

Thanks!

[flyinguser23]

Hello,
did you find any solution for your problem?
I've also got multiple connections which won't close.

I've found some old threads which explained how to close the connections manually, they dont seem to work for me though.

viewtopic.php?t=36637

1. Stop the OTRS Agent ( sudo -u otrs /opt/otrs/bin/otrs.Daemon.pl stop )
2. In MySQL run:
2a. truncate otrs.scheduler_task;
2b. truncate otrs.process_id;
3. In CentOS:
3a. ps -u otrs
3b. Kill all running processes ( sudo kill -9 [process#] )
4. Start the OTRS Agent ( sudo -u otrs /opt/otrs/bin/otrs.Daemon.pl start )

The problem started with version 6.4.3.
After trying to fix the problem, I've also decided to upgrade 6.5.3 which didn't help at all.

Manually fetching with the force parameter simpy gets stuck on the same step which I can see in the communication protocoll on all the open instances:

~$ sudo -u otrs /opt/otrs/bin/otrs.Console.pl Maint::PostMaster::MailAccountFetch --force-pid

Spawning child process to fetch incoming messages from mail accounts...

outlook.office365.com (IMAPTLS)...


_____________________________________________
select * from communication_log
where
status = 'Processing'
and
direction = 'Incoming'

Currently returns 48 entries: 1 from june 6. and two june 19. Everything else is from june 25.

Systeminformation:
Debian 10 LTS
OTRS 6.5.3 (and previously 6.4.3)
O365 Mailboxes using OAuth2 for fetchmail
10.3.39-MariaDB-0+deb10u1
Perl v5.28.1

[thowelt]

Same problem here.
OAuth2 + Office is currently not working.

Maybe there is a connection between Office Graph Connector update "mc545914"
"Graph Connector Content Results Experience Switches to Results Inline in All Vertical"

Updated from 6.4 to 6.5 LTS and also tried solutions like "--force-pid"

Problem occurred first time on 24.06.2023 at ~15:00

#OTRS-otrs.Console.pl-Maint::Email::MailQueue-42
There was an error executing Execute() in Kernel::System::Console::Command::Maint::Email::MailQueue: Error: Message sending already in progress! Skipping...

#OTRS-otrs.Console.pl-Maint::PostMaster::MailAccountFetch-42
There was an error executing Execute() in Kernel::System::Console::Command::Maint::PostMaster::MailAccountFetch: Error: Timeout of 600 seconds reached, killing child process!

[Johannes]

Hi,

I would say the admin center gives enough information at the moment.
There is an active incident at MS for this, see the attached file.

Regards
Johannes

Edit:

More info: The following services are impacted: - Users may be unable to access the Exchange Online service from Internet Message Access Protocol (IMAP) and Post Office Protocol (POP3) connection methods.

[zzz]

Edit: Before making changes, please read the next comments!

Hello,

If you are facing problems fetching mails via IMAP and Microsoft Exchange Online, you might try the following workaround, which worked for me and a couple of my customers who also experienced the same issue:

Code: Select all

sed -i 's/use Mail::IMAPClient;/use Mail::IMAPClient;\nuse IO::Socket::SSL;/' /opt/otrs/Kernel/System/MailAccount/IMAP.pm

Code: Select all

sed -i '/my %StartTLSOptions = $Self->_GetStartTLSOptions();/a\
\
    if ( $Param{Host} eq '\''outlook.office365.com'\'' ) {\
        $StartTLSOptions{Socket} = IO::Socket::SSL->new(\
            PeerAddr => $Param{Host},\
            PeerPort => 993,\
            SSL_verify_mode => 0,\
        );\

Feedback is appreciated.

Best regards
Emin

[thowelt]

Switching to POP3S worked for us.

@zzz
Your two commands didn't help, sorry.
But thanks for posting!

[Johannes]

Thanks Emin. As discussed earlier. This is another problem due to context switches during a connection.

For everyone that do not like to modify code, please consider a switch to POP3S as long as the incident is live.

Note:
Officially StartTLS is not support and you might get the next trouble when they
disable this for IMAP.

See also:
https://support.microsoft.com/en-gb/off ... 6c4ac95353

https://github.com/znuny/Znuny/issues/4 ... 1607325194

[zzz]

Johannes, thanks for pointing out that a switch to POP3 is also possible for some system.

STARTTLS seems to be the problem. Instead of switching to POP3, users might also try to switch from IMAPTLS to IMAPS.

This might be a bit confusing, but Port 993 is for SSL/TLS. IMAPTLS in Znuny, however, stands for STARTTLS, which is something completely different.

Best regards
Emin

[Johannes]

Hi,

we had several calls and tickets today.

We can summarize this:
- Incident on Microsoft Office365 side, as posted already
- STARTTLS is deprecated not in use at Office365 and should not be used
- IMAPS and POP3S should be used
- The incident may or may not be related to the switching off of STARTTLS on the IMAP ports (which was announced) - which was not support / documented anyway.

So two things came together: Incident at MS and the fact that STARTTLS will not work anymore.

Please see correct ports and hosts:
https://support.microsoft.com/en-gb/off ... 6c4ac95353

If the process did crash / hang you do not need to do anything. Just wait (max 10 min)

If you are not sure:
please run
bin/znuny.Console.pl Maint::Daemon::Summary
or
bin/otrs.Console.pl Maint::Daemon::Summary

192.168.1.105 2023-06-26 16-48-09.png

If you see a long running MailAccount fetch job, please run the job manually.
It will reset the pid and the mails should be fetched.

bin/znuny.Console.pl Maint::PostMaster::MailAccountFetch --force-pid
or
bin/otrs.Console.pl Maint::PostMaster::MailAccountFetch --force-pid


Regards
Johannes

[devber]

Hi guys!

I see that MS "resolved" the issue but I still have this issue present - does it work for you all again?

https://support.nhs.net/2023/06/microso ... -services/

[flyinguser23]

Hi,

thank you for your answers!
Since I've switched to IMAPS everything is working as intended.
Thank you!
Still thinking about switching to procmail once I've got more time.

[Johannes]

Hi guys!

I see that MS "resolved" the issue but I still have this issue present - does it work for you all again?

https://support.nhs.net/2023/06/microso ... -services/

Yes, everything back to normal.
Have you switched to IMAPS / POP3S?

[RichH]

Thanks to everyone for your responses.

We'll test switching to IMAPS or POP3S Thursday when our email admin is back from holidays...

Thanks again!

[RichH]

Hi,

Another (related) question:

The timeout for each fetchmail session is 600 seconds. It takes about 10 seconds per email, and only downloads a maximum of 15 emails per session.

So it takes about 180 seconds per session at most.

Is there anyway to decrease the timeout for the fetchmail sessions? I'd think that 300 seconds would be enough.

Also, I found the the retry time for failed sessions is 300 seconds. Could this be increased?

Daemon::SchedulerGenericInterfaceTaskManager::FutureTaskTimeDiff

Thanks!

[RichH]

Hi again....

Any answer for how to decrease the Fetchmail timeout (600 seconds)?

We're occasionally getting duplicate emails:

if a sequence of Fetchmails exceeds 10 minutes, a new sequence starts. That means that there can be 2 connections to the same mailbox downloading the same email.

I know that I could increase the fetchmail interval, but 10 minutes works well. Decreasing the timeout would work better, but I can't find that setting anywhere.

Thanks!

[root]

Hi,

Code: Select all

➜  Znuny git:(rel-6_5) grep -Rn 600 Kernel/System/Console
Kernel/System/Console/Command/Maint/PostMaster/MailAccountFetch.pm:55:            "Timeout in seconds to kill the child process, that does the mail fetching (default: 600).",
Kernel/System/Console/Command/Maint/PostMaster/MailAccountFetch.pm:72:        TTL   => 600,                             # 10 minutes
Kernel/System/Console/Command/Maint/PostMaster/MailAccountFetch.pm:99:    my $Timeout       = $Self->GetOption('timeout') || 600;

There is a timeout parameter, you can add this to the Crontask configuration (sysconfig)

But I would try to fix the root cause of why the process runs into the timeout.

- Roy

[RichH]

Thanks for your reply.

We have 45 mailboxes to fetch from Exchange Online using IMAPS/OAUTH2. Most fetches take about 8-10 seconds per email, with 0-15 emails per fetch, sometimes more.

If any one of the fetches runs for long time, all the other fetches are delayed and can run the entire sequence of fetches over the 10 minute cycle.

In the last case, a single fetch took 384 seconds which caused the subsequent fetches to be delayed.


I've been unable to find any CronTask or MailAccountFetch timeout in the Znuny sysconfig settings. Could you be more specific in how I can set this in sysconfig?

I'd prefer not to edit MAilAccountFetch.pm directly.

Thanks!

[root]

Hi,

You can't find the timeout parameter because I said "add". Should probably look then like this after yu changed it:

SCR-20230803-qdyf.png

- Roy

[RichH]

Thanks Roy!