OTRS not handling procmail

[crythias]

Summary: .procmail on OTRS version 5.0.24 (git) isn't generating tickets.

Any Ideas? I can't seem to get this working ... it makes spool tickets and then I have to run the spool process. This isn't good because I can't check if the process spool is already running and it doubles, triples up tickets.

I'm on CentOS 7.4.1708
mariadb (not relevant)
factory (slightly modified) .procmailrc with logging enabled.

Code: Select all

# --
# Copyright (C) 2001-2017 xxx, http://otrs.com/
# --
# This software comes with ABSOLUTELY NO WARRANTY. For details, see
# the enclosed file COPYING for license information (AGPL). If you
# did not receive this file, see http://www.gnu.org/licenses/agpl.txt.
# --

#SYS_HOME=/opt/otrs/
SYS_HOME=$HOME

UMASK=003
SHELL=/bin/sh
PATH=/bin:/usr/bin:/usr/local/bin
MONTHFOLDER=`date +%Y-%m`
YEARFOLDER=`date +%Y`
LOGFILE=$SYS_HOME/var/log/procmail-$MONTHFOLDER.log
VERBOSE=on


# Remove all X-OTRS Header (allow this only for trusted email)
# e. g. from *@example.com

:0 fhw :
* !^From.*@example.com
| grep -vi '^X-OTRS-'

# or just no emails with "X-OTRS-Ident: ahfiw2Fw32r230dddl2foeo3r" header!
#:0 fhw :
#* !^X-OTRS-Ident: ahfiw2Fw32r230dddl2foeo3r
#| grep -vi '^X-OTRS-'



# Examples for queue presorting.

:0 fhw :
* TO:.*@example.com
| formail -I "X-OTRS-Queue: example"

:0 fhw :
* ^List-Id:.*OpenAntiVirus
| formail -I "X-OTRS-Queue: OpenAntiVirus"


# Example for add free form key and value (max 3).

:0 fhw :
* TO:.*@example.com
| formail -I "X-OTRS-ArticleKey1: Test" |  formail -I "X-OTRS-ArticleValue1: True!"


# Example for assigning the "email domain as customer ID" automaticaly.

#:0hc
#CUSTOMERID=|formail -X "From:"|perl -e '$i=<STDIN>; $i=~s/^From: //;$i=~s/\(.*?\)//;$i=~s/<(.*?)>/$1/;$i=~s/ //g;$i=~s/.*@(.*)/$1/;print $i'
#
#:0 fhw :
#| formail -I "X-OTRS-CustomerNo: $CUSTOMERID"

# Backup of all incoming emails.
# It's always better to have a backup of all incoming emails!

:0 c :
$SYS_HOME/var/INBOX.Backup.$MONTHFOLDER


# Pipe all email into the PostMaster process.

:0 :
| $SYS_HOME/bin/otrs.Console.pl Maint::PostMaster::Read


# spool all the rest (which the otrs.PostMaster.pl can't process!)
# If the database is down or the otrs.PostMaster.pl exit was not '0'!

:0 :
$SYS_HOME/var/spool/.


# end of .procmailrc

The permissions on .procmailrc:

Code: Select all

-rw-r-----  1 otrs otrs     1859 Jan 11 08:01 .procmailrc

Error message in Syslog when messages arrive:

Code: Select all

Thu Jan 11 08:02:08 2018	error	OTRS-otrs.Console.pl-Maint::PostMaster::Read-10	mkdir /opt/otrs/var/tmp/CacheFileStorable/Valid/2: Permission denied at /opt/otrs/Kernel/System/Cache/FileStorable.pm line 87.

The folder exists:

Code: Select all

drwxrws---  3 apache apache 4096 Jan  4 07:44 2

procmail log:

Code: Select all

procmail: [12028] Thu Jan 11 08:10:38 2018
procmail: Match on ! "^From.*@example.com"
procmail: Couldn't determine implicit lockfile from "grep"
procmail: Locking ".lock"
procmail: Executing "grep,-vi,^X-OTRS-"
procmail: Unlocking ".lock"
procmail: No match on "TO:.*@example.com"
procmail: No match on "^List-Id:.*OpenAntiVirus"
procmail: No match on "TO:.*@example.com"
procmail: Locking "/opt/otrs/var/INBOX.Backup.2018-01.lock"
procmail: Assigning "LASTFOLDER=/opt/otrs/var/INBOX.Backup.2018-01"
procmail: Opening "/opt/otrs/var/INBOX.Backup.2018-01"
procmail: Acquiring kernel-lock
procmail: Unlocking "/opt/otrs/var/INBOX.Backup.2018-01.lock"
procmail: Couldn't determine implicit lockfile from "/opt/otrs/bin/otrs.Console.pl"
procmail: Locking ".lock"
procmail: Executing "/opt/otrs/bin/otrs.Console.pl,Maint::PostMaster::Read"
ERROR: OTRS-otrs.Console.pl-Maint::PostMaster::Read-10 Perl: 5.16.3 OS: linux Time: Thu Jan 11 08:10:39 2018

 Message: mkdir /opt/otrs/var/tmp/CacheFileStorable/Valid/2: Permission denied at /opt/otrs/Kernel/System/Cache/FileStorable.pm line 87.


 Traceback (12035):
   Module: Kernel::System::Console::Command::Maint::PostMaster::Read::Run Line: 138
   Module: (eval) Line: 460
   Module: Kernel::System::Console::BaseCommand::Execute Line: 454
   Module: Kernel::System::Console::InterfaceConsole::Run Line: 86
   Module: /opt/otrs/bin/otrs.Console.pl Line: 38

procmail: [12028] Thu Jan 11 08:10:39 2018
procmail: Non-zero exitcode (75) from "/opt/otrs/bin/otrs.Console.pl"
procmail: Assigning "LASTFOLDER=/opt/otrs/bin/otrs.Console.pl Maint::PostMaster::Read"
procmail: Unlocking ".lock"
procmail: Locking "/opt/otrs/var/spool/..lock"
procmail: Assigning "LASTFOLDER=/opt/otrs/var/spool/8"
procmail: Opening "/opt/otrs/var/spool/8"
procmail: Acquiring kernel-lock
procmail: Unlocking "/opt/otrs/var/spool/..lock"
procmail: Notified comsat: "otrs@0:/opt/otrs/var/spool/8"
From name.redacted@domainredacted.com  Thu Jan 11 08:10:38 2018
 Subject: RE: Subject Redacted
  Folder: /opt/otrs/var/spool/8                                           26873

I can't get procmail command

Code: Select all

| $SYS_HOME/bin/otrs.Console.pl Maint::PostMaster::Read

to debug. even if I add --debug to the entry.

[root]

Hi,

The user should be otrs with the primary group apache when using CentOS. I assume you use the RPM.

This is how it looks on my reference system, fresh installed:

Code: Select all

[root@support ~]# ls -ld /opt/otrs/.procmailrc
-rw-r-----. 1 otrs root 1892 Nov  1  2016 /opt/otrs/.procmailrc

Did you tried it after clearing the cache and running otrs.SetPermissions.pl

- Roy

[crythias]

I didn't use RPM. I installed via source.
The only difference for your procmailrc and mine is group root. Which ... shouldn't really be a thing, but okay.

Code: Select all

> groups otrs
otrs : otrs apache
> ls -la .procmailrc
-rw-r----- 1 otrs root 1859 Jan 11 08:51 .procmailrc

Code: Select all

Locking ".lock"
procmail: Executing "/opt/otrs/bin/otrs.Console.pl,Maint::PostMaster::Read"
ERROR: Permission denied: /opt/otrs/Kernel/Config/Files/ZZZAAuto.pm
ERROR: Permission denied: /opt/otrs/Kernel/Config/Files/ZZZAuto.pm
ERROR: OTRS-otrs.Console.pl-Maint::PostMaster::Read-10 Perl: 5.16.3 OS: linux Time: Thu Jan 11 09:31:55 2018

 Message: mkdir /opt/otrs/var/tmp/CacheFileStorable/Valid/2: Permission denied at /opt/otrs/Kernel/System/Cache/FileStorable.pm line 87.


 Traceback (5174):
   Module: Kernel::System::Console::Command::Maint::PostMaster::Read::Run Line: 138
   Module: (eval) Line: 460
   Module: Kernel::System::Console::BaseCommand::Execute Line: 454
   Module: Kernel::System::Console::InterfaceConsole::Run Line: 86
   Module: /opt/otrs/bin/otrs.Console.pl Line: 38

procmail: [5167] Thu Jan 11 09:31:55 2018
procmail: Non-zero exitcode (75) from "/opt/otrs/bin/otrs.Console.pl"

I'm guessing this is because it can't make the CacheDirectory that exists but doesn't.

[crythias]

Roy

Are you available to address this? I'm really in need of looking at this ... I can't seem to get Postmaster Read to debug.

[wurzel]

Hi,

my .procmailrc is
932287 4 -rw-r----- 1 otrs root 1913 Jan 10 09:53 .procmailrc

It seems you have a permission issue and on your /opt/otrs filesystem, too

Regards
Florian

[crythias]

It seems you have a permission issue and on your /opt/otrs filesystem, too

But my .procmailrc runs. it's not a permission with procmailrc. I get logs. It's OTRS not doing the thing.

I would guess so, but there's no particular difference in permissions between ticket cacheing and postmaster read cacheing.

[wurzel]

Hi,

OK. Fine.

I just wanted to mention my permissions on the file

in any case... your log shows

ERROR: Permission denied: /opt/otrs/Kernel/Config/Files/ZZZAuto.pm

so it is actually a permission issue.
Either your ./Files directory is not allowed to be touched by otrs user (this usually happens if you perform the Rebuild Config via root user (should not be possible in versions 5+6)

Or your otrs user is in wrong permission group (apache/httpd/www-data/your-apache-group)
or your apache webserver is running with wrong user/group (and writes files with wrong permissions)


I doubt that my tipps will help you, because you ususally are more expert than me...

Sorry that I can not help you further


but maybe you can show your file permissions of
/opt/otrs
/opt/otrs/Kernel/Config/Files
/opt/otrs/var/tmp/ + subdirectories

and your group membership of your otrs user.

Regards
Florian

[crythias]

in any case... your log shows

ERROR: Permission denied: /opt/otrs/Kernel/Config/Files/ZZZAuto.pm

so it is actually a permission issue.
Either your ./Files directory is not allowed to be touched by otrs user (this usually happens if you perform the Rebuild Config via root user (should not be possible in versions 5+6)

And this is what ticks me off regarding OTRS (not you).

Code: Select all

[root@sp Config]# ls -lad /opt/otrs/Kernel/Config
drwxrwsr-x 3 otrs apache 4096 Dec 22 15:22 /opt/otrs/Kernel/Config
[root@sp Config]# ls -lad /opt/otrs/Kernel/Config/Files
drwxrwsr-x 2 otrs apache 4096 Jan 11 09:27 /opt/otrs/Kernel/Config/Files
[root@sp Config]# groups otrs
otrs : otrs apache
[]ls -la /opt/otrs/Kernel/Config/Files/ZZZ*
-rw-rw---- 1 apache apache 371535 Jan 11 09:27 /opt/otrs/Kernel/Config/Files/ZZZAAuto.pm
-rw-rw---- 1 apache apache  10368 Jan 11 09:26 /opt/otrs/Kernel/Config/Files/ZZZAuto.pm

Every time apache rewrites ZZZ*.pm it changes the owner from otrs to apache. and uses 660 permissions when I need (at least) 644

Code: Select all

 ls -la
total 2128
drwxrwsr-x 2 otrs   apache   4096 Jan 11 09:27 .
drwxrwsr-x 3 otrs   apache   4096 Dec 22 15:22 ..
-rw-rw---- 1 otrs   apache   3795 Nov 25 15:06 CloudServices.xml
-rw-rw---- 1 otrs   apache  37825 Nov 25 15:06 Daemon.xml
-rw-r--r-- 1 apache apache 111045 Nov 25 15:36 FAQ.xml
-rw-rw---- 1 otrs   apache 351711 Nov 25 15:06 Framework.xml
-rw-r--r-- 1 apache apache   1636 Nov 25 15:42 FrameworkITSMCore.xml
-rw-r--r-- 1 apache apache   3208 Nov 25 15:41 GeneralCatalog.xml
-rw-rw---- 1 otrs   apache  27122 Nov 25 15:06 GenericInterface.xml
-rw-r--r-- 1 apache apache 255612 Nov 25 15:44 ITSMChangeManagement.xml
-rw-r--r-- 1 apache apache  51564 Nov 25 15:44 ITSMConfigItem.xml
-rw-r--r-- 1 apache apache   5722 Nov 25 15:44 ITSMConfigItemGenericInterface.xml
-rw-r--r-- 1 apache apache   3711 Nov 25 15:44 ITSMConfigItemTicketLink.xml
-rw-r--r-- 1 apache apache  21863 Nov 25 15:42 ITSMCore.xml
-rw-r--r-- 1 apache apache  11836 Nov 25 15:42 ITSMService.xml
-rw-r--r-- 1 apache apache  40896 Nov 25 15:46 ITSMTicket.xml
-rw-r--r-- 1 apache apache   1682 Nov 25 15:47 ImportExport.xml
-rw-r--r-- 1 apache apache  29327 Nov 25 15:37 OTRSMasterSlave.xml
-rw-r--r-- 1 apache apache   2380 Nov 25 15:49 PasswordPolicy.xml
-rw-rw---- 1 otrs   apache  25448 Nov 25 15:06 ProcessManagement.xml
-rw-r--r-- 1 apache apache   8851 Nov 25 15:38 SystemMonitoring.xml
-rw-rw---- 1 otrs   apache 637050 Nov 25 15:06 Ticket.xml
-rw-r--r-- 1 apache apache    675 Nov 25 15:42 TicketITSMService.xml
-rw-r--r-- 1 apache apache   6859 Nov 25 15:46 TicketITSMTicket.xml
-rw-r--r-- 1 apache apache  21090 Nov 25 15:38 TimeAccounting.xml
-rw-rw---- 1 apache apache 371535 Jan 11 09:27 ZZZAAuto.pm
-rw-rw---- 1 apache apache  10368 Jan 11 09:26 ZZZAuto.pm
-rw-r--r-- 1 apache apache   4135 Nov 25 15:48 ZZZZZZnuny4OTRSRepo.pm
-rw-r--r-- 1 apache apache  24785 Nov 25 15:48 Znuny4OTRSRepo.xml
-rw-r--r-- 1 apache apache   2369 Nov 25 15:50 Znuny4OTRSShowPendingTimeIfNeeded.xml

What's interesting is that the procmail dump of Backup emails is otrs/apache:

Code: Select all

-rw-rw-r--  1 otrs   apache  56485107 Jan 11 12:10 INBOX.Backup.2018-01

procmailrc is running as otrs. It knows $HOME is /opt/otrs.
what's strange is that even as procmail runs otrs ... stuff, otrs seems not to run as otrs user. It seems to run as something that's not otrs and not apache.

[jojo]

SELINUX?

[crythias]

Code: Select all

sestatus
SELinux status:                 disabled

[root]

What't the primary group of the OTRS user? Should be apache on a CentOS.

- Roy

[seberget2]

Every time apache rewrites ZZZ*.pm it changes the owner from otrs to apache. and uses 660 permissions when I need (at least) 644

So what is your umask setting for the apache user?

[crythias]

So what is your umask setting for the apache user?

Code: Select all

 su -s /bin/bash apache
bash-4.2$ umask
0022

/etc/profile:

Code: Select all

# By default, we want umask to get set. This sets it for login shell
# Current threshold for system reserved uid/gids is 200
# You could check uidgid reservation validity in
# /usr/share/doc/setup-*/uidgid file
if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ]; then
    umask 002
else
    umask 022
fi

/etc/passwd

Code: Select all

apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin

[seberget2]

And the otrs user is member of the apache group?

[crythias]

I've already posted this twice, but since it's asked again:

Code: Select all

 groups otrs
otrs : otrs apache

[crythias]

removing

[crythias]

Code: Select all

# Pipe all email into the PostMaster process.

:0 :
| $SYS_HOME/bin/otrs.Console.pl Maint::PostMaster::Read

procmail says the current running user is otrs.
$SYS_HOME certainly resolves to /opt/otrs

So far, I've narrowed this down to
Kernel/System/PostMaster.pm:

Code: Select all

        # add Dynamic Field headers
        my $DynamicFields = $DynamicFieldObject->DynamicFieldList(
            Valid      => 1,
            ObjectType => [ 'Ticket', 'Article' ],
            ResultType => 'HASH',
        );

[crythias]

Something more interesting:
Changes to DynamicField.pm for debugging:

Code: Select all

            $Kernel::OM->Get('Kernel::System::Log')->Log(
                Priority => 'info',
                Message  => 'CacheGet!',
            );
    my $Cache = $CacheObject->Get(
        Type => 'DynamicField',
        Key  => $CacheKey,
    );

            $Kernel::OM->Get('Kernel::System::Log')->Log(
                Priority => 'info',
                Message  => 'CacheGot!',
            );
    if ($Cache) {

            $Kernel::OM->Get('Kernel::System::Log')->Log(
                Priority => 'info',
                Message  => 'is Cache!',
            );

If procmail does this, it doesn't Cache. When cleanup does it, it does Cache.

[crythias]

Code: Select all

            $SQL .= ' WHERE valid_id IN (' . join ', ', $ValidObject->ValidIDsGet() . ')';

This is where it stops. I'm bouncing to the next place.

[crythias]

In some places Queue.pm, it's

Code: Select all

" st.valid_id IN ( ${\(join ', ', $ValidObject->ValidIDsGet())} )"

or Salutation.pm:

Code: Select all

WHERE valid_id IN ( ${\(join ', ', $Kernel::OM->Get('Kernel::System::Valid')->ValidIDsGet())} )

in MailAccount it's

Code: Select all

'WHERE valid_id IN ( ' . join ', ', $ValidObject->ValidIDsGet() . ' )'

in LinkObject it's

Code: Select all

my $ValidIDs = join ', ', $Kernel::OM->Get('Kernel::System::Valid')->ValidIDsGet();

[crythias]

DynamicField doesn't want to grab ValidIDsGet (it fails on Cache-ing)
And it's having trouble Cache-ing DynamicField Hash data

So there's a problem where procmail doesn't try to use the cache and spooler does.
ValidIDsGet doesn't return values from line 732 (the call fails with permissions to cache):

Code: Select all

    else {
        my $SQL = 'SELECT id, name, field_order FROM dynamic_field';

        # get database object
        my $DBObject = $Kernel::OM->Get('Kernel::System::DB');

        if ($Valid) {

            # get valid object
            my $ValidObject = $Kernel::OM->Get('Kernel::System::Valid');

            $SQL .= ' WHERE valid_id IN (' . join ', ', $ValidObject->ValidIDsGet() . ')';

So, why is that ValidObject->ValidIDsGet() failing? It's exactly the same near 982.
https://github.com/OTRS/otrs/blob/460b0 ... ld.pm#L732
https://github.com/OTRS/otrs/blob/460b0 ... ld.pm#L982

Somehow, only when the procmail engages, 732 kicks in and crashes. When SpoolMailsReprocess does it, 732 is ignored, and everything runs.

Anyone have any idea?

[crythias]

The difference between procmailrc and spool:

When procmailrc requests a CacheKey:

Code: Select all

DynamicFieldList::Valid::1::ObjectType::Article_Ticket::ResultType::HASH

When Spool reprocess requests a CacheKey:

Code: Select all

DynamicFieldList::Valid::1::ObjectType::Ticket::ResultType::HASH

[crythias]

So here's the fun part:

Both of these have permissions issues:

Valid.pm:
https://github.com/OTRS/otrs/blob/a79e8 ... lid.pm#L84

Code: Select all

    while ( my @Row = $DBObject->FetchrowArray() ) {
        $Data{ $Row[0] } = $Row[1];
    }

    # set cache
    $Kernel::OM->Get('Kernel::System::Cache')->Set(
        Type  => $Self->{CacheType},
        TTL   => $Self->{CacheTTL},
        Key   => $CacheKey,
        Value => \%Data
    );

DynamicField.pm:
https://github.com/OTRS/otrs/blob/a79e8 ... ld.pm#L774

Code: Select all

            while ( my @Row = $DBObject->FetchrowArray() ) {
                $Data{ $Row[0] } = $Row[1];
            }

            # set cache
            $CacheObject->Set(
                Type  => 'DynamicField',
                Key   => $CacheKey,
                Value => \%Data,
                TTL   => $Self->{CacheTTL},
            );

[crythias]

I gotta find a different way to address. all the Cache Sets have permission issues via procmail, even though procmail is running (as far as I can tell) in the context of otrs.

Spool reprocess is also running via crontab under the otrs user and has no problem processing. But also doesn't seem to touch cache.

[crythias]

From SpoolMailsReprocess.pm

# Here we use a system call because Maint::PostMaster::Read has special exception handling
# and will die if certain problems occur.
my $Result = system("$^X $Home/bin/otrs.Console.pl Maint::PostMaster::Read < $File ");

Maybe this is a hint? If so, then procmailrc doesn't handle this the same way.

[crythias]

welp, I went the wrong way I guess:
running as otrs,
procmail .procmailrc <test-email.box
works exactly as designed.

So it's a CentOS permissions issue.

[crythias]

After all this, I'm throwing my towel in the ring.

I changed the crontab to every minute to clean up the spool and flock to not run this multiple times and duplicate tickets.

Code: Select all

* * * * *  /usr/bin/flock -n /tmp/noynim-otrs-mail.lockfile  $HOME/bin/otrs.Console.pl Maint::PostMaster::SpoolMailsReprocess >> /dev/null