Various kind of errors on Apache error.log

[dolfiz]

Hi all,

first of all sorry for the generic topic title but, though I think that may be a unique reason behind (resources? apache bug?), I really don't know how to aggregate on a single matter the following problems. I thought that they can be correlated due to the fact that they appeared on the same days and that before all worked well. Maybe I'm wrong, but maybe not, so I would like to give you all the evidences that we collected.

Our scenario:
- OTRS 3.3.3
- ubuntu 12.04.4 LTS
- mysql 5.5.34-0
- perl 5.14.2
- apache 2.2.22
- it has been used for more than a month with increasing load (from 700 tt/day till the current 2-3k tt/day)
- web server: virtual machine with 6GB ram
- db server: vm with 8GB ram
- agents ~50

I must say that until now the whole system worked well without inexplicable anomalys. We did not perform big changes to the system lately, just the load increases a bit (from 1.5k tt/day to 2-3k tt/day).

Checking the log for the few last days we found several errors that we do not understand:

ERROR [1]

Code: Select all

Scalars leaked: 1

here and there, without error declaration, just those words, with a value that can change (positive, negative, sometimes small like the example, other times bigger as 158): what does those rows mean? Do I need to care about them?

ERROR [2]

Code: Select all

*** glibc detected *** /usr/sbin/apache2: double free or corruption (out): 0x00007f32f15ac6f0 ***

that followed by a spool of many files (see attachment "glibc_detected.txt"). It seems an apache/perl problem and googling it we found that probably it is a bug within apache or a module used. It seems that 2 times a day apache restart itself after this error

ERROR [3]

Code: Select all

[Mon Mar 03 12:47:20 2014] [error] panic: attempt to copy freed scalar 7f32eec47cc8 to 7f32d82de540 at /opt/otrs//Kernel/System/Group.pm line 533.\n
[Mon Mar 03 12:47:24 2014] [error] Not an ARRAY reference at /opt/otrs//Kernel/System/Ticket.pm line 6310.\n
[Mon Mar 03 12:47:25 2014] [error] Not an ARRAY reference at /opt/otrs//Kernel/System/Ticket.pm line 6310.\n
[Mon Mar 03 12:47:28 2014] [error] Not an ARRAY reference at /opt/otrs//Kernel/System/Ticket.pm line 6310.\n
[Mon Mar 03 12:47:33 2014] [error] Not an ARRAY reference at /opt/otrs//Kernel/System/Ticket.pm line 6310.\n
[Mon Mar 03 12:47:33 2014] [error] Not an ARRAY reference at /opt/otrs//Kernel/System/Ticket.pm line 6310.\n
[Mon Mar 03 12:47:34 2014] [error] Not an ARRAY reference at /opt/otrs//Kernel/System/Ticket.pm line 6310.\n
[Mon Mar 03 12:47:38 2014] [error] Not an ARRAY reference at /opt/otrs//Kernel/System/Ticket.pm line 6310.\n

This is the first clearly otrs-related error. I really don't know if it underlines a possible mistake on our implementation (wrong configurations, wrong values on Config.pm, ...) or something else. Of course, we did not modified Group.pm nor Ticket.pm.

ERROR [4]

Code: Select all

[Mon Mar 03 14:15:20 2014] [error] [client xxx.xxx.xxx.xx] File does not exist: /var/www/avcp_skin, referer: http://xxx.xxx.xxx.xx/otrs/index.pl?Action=AgentTicketProcess;Subaction=DisplayActivityDialog;TicketID=15054;ProcessEntityID=P2;ActivityDialogEntityID=AD6
[Mon Mar 03 14:15:20 2014] [error] [client xxx.xxx.xxx.xx] File does not exist: /var/www/portal, referer: http://xxx.xxx.xxx.xx/otrs/index.pl?Action=AgentTicketProcess;Subaction=DisplayActivityDialog;TicketID=15054;ProcessEntityID=P2;ActivityDialogEntityID=AD6
[Mon Mar 03 14:15:20 2014] [error] [client xxx.xxx.xxx.xx] File does not exist: /var/www/portal, referer: http://xxx.xxx.xxx.xx/otrs/index.pl?Action=AgentTicketProcess;Subaction=DisplayActivityDialog;TicketID=15054;ProcessEntityID=P2;ActivityDialogEntityID=AD6
[Mon Mar 03 14:15:20 2014] [error] [client xxx.xxx.xxx.xx] File does not exist: /var/www/eXoResources, referer: http://xxx.xxx.xxx.xx/otrs/index.pl?Action=AgentTicketProcess;Subaction=DisplayActivityDialog;TicketID=15054;ProcessEntityID=P2;ActivityDialogEntityID=AD6
[Mon Mar 03 14:15:20 2014] [error] [client xxx.xxx.xxx.xx] File does not exist: /var/www/eXoResources, referer: http://xxx.xxx.xxx.xx/otrs/index.pl?Action=AgentTicketProcess;Subaction=DisplayActivityDialog;TicketID=15054;ProcessEntityID=P2;ActivityDialogEntityID=AD6

There are various errors like these ones. One time /var/www/js is missing, another time /var/www/css is missing, etc... Currently that folder is empty except for a file index.html.

ERROR [5]

Code: Select all

[Mon Mar 03 15:27:17 2014] [error] Can't call method "new" without a package or object reference at /opt/otrs//Kernel/System/SLA.pm line 90.
[Mon Mar 03 15:27:18 2014] [error] [client x.x.x.x] panic: attempt to copy freed scalar 7f33047bac80 to 7f3305c73968 at /usr/lib/perl/5.14/File/Spec/Unix.pm line 113.\nCompilation failed in require at /opt/otrs/Kernel/cpan-lib/Apache2/Reload.pm line 179.

After those errors many (>50) of the following ones have been spooled:

Code: Select all

[Mon Mar 03 15:27:19 2014] [error] [client x.x.x.x] panic: attempt to copy freed scalar 7f33047bac80 to 7f33064f3188 at /opt/otrs/Kernel/cpan-lib/Apache2/Reload.pm line 138.\n, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketPhone

Those "panic" words make me very uncomfortable. Did you see something similar before?

ERROR [6]

Code: Select all

[Tue Mar 04 12:56:29 2014] [error] slurp_filename('/opt/otrs/bin/cgi-bin/js') / opening: (2) No such file or directory at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 541

Actually there's no "js" file/folder on that folder. I don't know what to say about :/

Those are the problems that appear on the last days. I must say that agents did not complain about the web-application, but I don't know if they experienced problems using the web interface without letting us know about it. I asked the virtual environment responsible for log of occupied resources on those VMs in order to check if errors appear during memory peaks or something similar. I'm looking forward for those logs.

Do you expert guys have other suggestions? Do the above evidences make sense together or do you think that could be more than a problem behind them?

Any help is appreciated. Sorry for the loooooong topic.
Dolfiz

[reneeb]

wrt ERROR[3]: Can you post those lines mentioned in the error message

wrt ERROR[4]: Is the ticket 15054 a HTML mail? Have a look at the plain view (you might activate the PlainView option in SysConfig). I guess there are some links in the HTML message that are relative not absolute

Which mod_perl version do you use?

[dolfiz]

Thank you reneeb for your time.

wrt ERROR[3]: Can you post those lines mentioned in the error message

Group.pm (rows 508-615)

Code: Select all

sub GroupMemberList {
    my ( $Self, %Param ) = @_;

    # check needed stuff
    for (qw(Result Type)) {
        if ( !$Param{$_} ) {
            $Self->{LogObject}->Log( Priority => 'error', Message => "Need $_!" );
            return;
        }
    }
    if ( !$Param{UserID} && !$Param{GroupID} ) {
        $Self->{LogObject}->Log( Priority => 'error', Message => 'Need UserID or GroupID!' );
        return;
    }

    # create cache key
    my $CacheKey = 'GroupMemberList::' . $Param{Type} . '::' . $Param{Result} . '::';
    if ( $Param{UserID} ) {
        $CacheKey .= "UserID::$Param{UserID}";
    }
    else {
        $CacheKey .= "GroupID::$Param{GroupID}";
    }

    # check cache
    my $Cache = $Self->{CacheInternalObject}->Get( Key => $CacheKey );   <------ row 533
    if ($Cache) {
        return @{$Cache} if ref $Cache eq 'ARRAY';
        return %{$Cache} if ref $Cache eq 'HASH';
    }

    # return result
    if ( $Param{Result} eq 'ID' || $Param{Result} eq 'Name' ) {
        my @Result = $Self->GroupGroupMemberList(%Param);

        # get roles of user
        if ( $Param{UserID} ) {
            my @Member = $Self->GroupUserRoleMemberList(
                UserID => $Param{UserID},
                Result => 'ID',
            );
            if (@Member) {
                my @ResultGroupRole = $Self->GroupRoleMemberList( %Param, RoleIDs => \@Member, );
                push @Result, @ResultGroupRole;
            }
        }

        # get roles of group
        elsif ( $Param{GroupID} ) {
            my @Roles = $Self->GroupRoleMemberList(
                GroupID => $Param{GroupID},
                Type    => $Param{Type},
                Result  => 'ID',
            );
            if (@Roles) {
                my @ResultGroupUserRole = $Self->GroupUserRoleMemberList(
                    %Param,
                    RoleIDs => \@Roles,
                );
                push @Result, @ResultGroupUserRole;
            }
        }

        # set cache
        $Self->{CacheInternalObject}->Set( Key => $CacheKey, Value => \@Result );

        return @Result;
    }

    # get group member list as hash
    my %Result = $Self->GroupGroupMemberList(%Param);

    # get roles of user
    if ( $Param{UserID} ) {
        my @Member = $Self->GroupUserRoleMemberList(
            UserID => $Param{UserID},
            Result => 'ID',
        );

        if (@Member) {
            my %ResultGroupRole = $Self->GroupRoleMemberList( %Param, RoleIDs => \@Member );
            %Result = ( %Result, %ResultGroupRole );
        }
    }

    # get roles of group
    elsif ( $Param{GroupID} ) {
        my @Roles = $Self->GroupRoleMemberList(
            GroupID => $Param{GroupID},
            Type    => $Param{Type},
            Result  => 'ID',
        );
        if (@Roles) {

            my %ResultGroupUserRole = $Self->GroupUserRoleMemberList(
                %Param,
                RoleIDs => \@Roles,
            );
            %Result = ( %Result, %ResultGroupUserRole );
        }
    }

    # set cache
    $Self->{CacheInternalObject}->Set( Key => $CacheKey, Value => \%Result );

    return %Result;

}

Ticket.pm (rows 6285-6316)

Code: Select all

    # use user data
    if ( $Param{UserID} ) {
        my %User = $Self->{UserObject}->GetUserData(
            UserID => $Param{UserID},
        );
        for my $Type ( @{ $Self->{ConfigObject}->Get('System::Permission') } ) {
            my @Groups = $Self->{GroupObject}->GroupMemberList(
                UserID => $Param{UserID},
                Result => 'Name',
                Type   => $Type,
            );
            $User{"Group_$Type"} = \@Groups;
        }

        my @RoleIDs = $Self->{GroupObject}->GroupUserRoleMemberList(
            UserID => $Param{UserID},
            Result => 'ID',
        );
        my @Roles;
        ROLEID:
        for my $RoleID (@RoleIDs) {
            my $RoleName = $Self->{GroupObject}->RoleLookup(
                RoleID => $RoleID,
            );
            next ROLEID if !$RoleName;
            push @Roles, $RoleName;       <------- row 6310
        }
        $User{Role} = \@Roles;

        $Checks{User}         = \%User;
        $ChecksDatabase{User} = \%User;
    }

wrt ERROR[4]: Is the ticket 15054 a HTML mail? Have a look at the plain view (you might activate the PlainView option in SysConfig). I guess there are some links in the HTML message that are relative not absolute

No, it is not. Just a ticket with 3 articles of simple plain text (I did not used plain view but I queried directly the db).
Actually there is HTML in that body. Here's the plain body obtained as you suggested (previously I queried the article table :/):

Code: Select all

Email Text Plain View: 2014022288006134 - richiesta attivazione xxx
Close window
Plain Ticket#: 2014022288006134 / TicketID: 15054 / ArticleID: 31571 - Download this email
Received: from foo.goo.com (x.x.x.x) by exmcas01.master.base.local (x.x.x.x) with Microsoft SMTP Server (TLS) id 14.3.158.1; Wed, 19 Feb 2014 09:44:40 +0100
Received: from outrelay01.domain.com (outrelay01.domain.com [x.x.x.x]) by avlp-fortimail2-mancini.avlp.it  with ESMTP id s1J8iehD001190-s1J8iehE001190 for <foo@goo.com>; Wed, 19 Feb 2014 09:44:40 +0100
X-CTCH-Spam: Unknown
X-CTCH-Refid: str=0001.0A0C0207.53046EF8.000C,ss=1,re=0.000,fgs=0
X-Libjamoibt: 1587
Received: from wmail76 (x.x.x.x) by outrelay01.domain.com (x.x.x.x) id 53020E9500439151 for foo@goo.com; Wed, 19 Feb 2014 09:44:39 +0100
Message-ID: <105104.9784751392799479677.JavaMail.defaultUser@defaultHost>
Date: Wed, 19 Feb 2014 09:44:39 +0100
From: "john-smith@domain.com" <john-smith@domain.com>
Reply-To: "john-smith@domain.com" <john-smith@domain.com>
To: <foo@goo.com>
Subject: richiesta attivazione xxxx
Content-Type: multipart/mixed; boundary="----=_Part_875595_9358072.1392799479675"
X-Senderip: 95.243.177.240
X-Libjamv: cDmHbHKfcaI=
X-Libjamsun: sJSK8AI74m6ID5InZ99Ok3BguJfIJ0EsQ4TrrC10+8A=
Return-Path: john-smith@domain.com
X-MS-Exchange-Organization-Authsource: EXMCAS01.master.base.local
X-MS-Exchange-Organization-Authas: Anonymous
MIME-Version: 1.0

------=_Part_875595_9358072.1392799479675 
Content-Type: multipart/alternative; 
boundary="----=_Part_875596_11497594.1392799479676" 

------=_Part_875596_11497594.1392799479676 
Content-Type: text/plain; charset="UTF-8" 
Content-Transfer-Encoding: quoted-printable 

Gentilissimi, 
considerando l'urgenza per la partecipazione alla gara di appalto, relativa= 
ai lavori di Ristrutturazione Palazzo Stancati Comune di Domanico CS, ci v= 
iene richiesto il codice xxx.Si allega copia CCIAA e documento di ricono= 
scimento. GRAZIE Cordialmente geom. john smit=C3=A0 

------=_Part_875596_11497594.1392799479676 
Content-Type: text/html; charset="UTF-8" 
Content-Transfer-Encoding: quoted-printable 

<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8"><p= 
style=3D"margin: 0px; padding: 0px;">Gentilissimi,</p><p style=3D"margin: = 
0px; padding: 0px;"><br></p><p style=3D"margin: 0px; padding: 0px;">conside= 
chiesto il codice PASSoe.</p><p style=3D"margin: 0px; padding: 0px;">Si all= 
ega copia frefe e documento di riconoscimento. GRAZIE </p><p style=3D"margi= 
n: 0px; padding: 0px;">Cordialmente geom. john smit=C3=A0<br></p>= 

------=_Part_875596_11497594.1392799479676-- 

------=_Part_875595_9358072.1392799479675 
Content-Type: application/pdf; name="file.pdf" 
Content-Transfer-Encoding: base64 
Content-Disposition: attachment; filename="file.pdf"; size=671075 

JVBERi0xLjYNJeLjz9MNCjIyIDAgb2JqDTw8L0xpbmVhcml6ZWQgMS9MIDY3MTA3NS9PIDI0L0Ug 
MzY0MjYyL04gMi9UIDY3MDc0My9IIFsgNDQyIDE3OV0+Pg1lbmRvYmoNICAgICAgICAgICAgICAg 
[...]
ASZGluLJDEwMDIy3QSSrM5gtCSJ594PZfSCS4RFQ5Z8KDjCbgRFBMv3DLsLIABBgAKK5CCkNCmVu 
ZHN0cmVhbQ1lbmRvYmoNc3RhcnR4cmVmDQoxMTYNCiUlRU9GDQo= 

------=_Part_875595_9358072.1392799479675-- 

Which mod_perl version do you use?

Code: Select all

otrsuser@itssg-vlotrs01:~$ perl -Mmod_perl2 -E 'say $mod_perl2::VERSION'
2.000005
otrsuser@itssg-vlotrs01:~$

UPDATE:
Checking the log today I did not find "glibc detected error" but I can see again a bunch of:

Code: Select all

[Tue Mar 04 17:23:27 2014] [error] [client x.x.x.x.] File does not exist: /var/www/avcp_skin, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=15280
[Tue Mar 04 17:23:27 2014] [error] [client x.x.x.x.] File does not exist: /var/www/portal, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=15280
[Tue Mar 04 17:23:27 2014] [error] [client x.x.x.x.] File does not exist: /var/www/portal, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=15280

The referred tickets (more than one) are all simple tickets, with no HTML code on their body, with and without attachments. I cannot see peculiarity on them.

Beside that, new kind of errors appear:

Code: Select all

[Wed Mar  5 09:18:43 2014] -e: Can't cd to (/opt/otrs/var/tmp/CacheFileStorable/CacheInternalGroup/1/) 5: No such file or directory
[Wed Mar  5 09:18:43 2014] -e:  at /opt/otrs//Kernel/System/Cache/FileStorable.pm line 204
[Wed Mar  5 09:18:43 2014] -e: Can't cd to (/opt/otrs/var/tmp/CacheFileStorable/CacheInternalGroup/1/) 7: No such file or directory
[Wed Mar  5 09:18:43 2014] -e:  at /opt/otrs//Kernel/System/Cache/FileStorable.pm line 204
[Wed Mar  5 09:18:43 2014] -e: Can't cd to (/opt/otrs/var/tmp/CacheFileStorable/CacheInternalGroup/1/) a: No such file or directory
[Wed Mar  5 09:18:43 2014] -e:  at /opt/otrs//Kernel/System/Cache/FileStorable.pm line 204
[Wed Mar  5 09:18:43 2014] -e: Can't cd to (/opt/otrs/var/tmp/CacheFileStorable/CacheInternalGroup/5/) b: No such file or directory
[Wed Mar  5 09:18:43 2014] -e:  at /opt/otrs//Kernel/System/Cache/FileStorable.pm line 204
[Wed Mar  5 09:18:43 2014] -e: Can't cd to (/opt/otrs/var/tmp/CacheFileStorable/CacheInternalGroup/5/) 0: No such file or directory
[Wed Mar  5 09:18:43 2014] -e:  at /opt/otrs//Kernel/System/Cache/FileStorable.pm line 204
[Wed Mar  5 09:18:43 2014] -e: Can't cd to (/opt/otrs/var/tmp/CacheFileStorable/CacheInternalGroup/5/) 6: No such file or directory
[Wed Mar  5 09:18:43 2014] -e:  at /opt/otrs//Kernel/System/Cache/FileStorable.pm line 204

...and many more rows like these.

FileStorable.pm (rows 157-207)

Code: Select all

sub CleanUp {
    my ( $Self, %Param ) = @_;

    my @TypeList = $Self->{MainObject}->DirectoryRead(
        Directory => $Self->{CacheDirectory},
        Filter => $Param{Type} || '*',
    );

    return 1 if !@TypeList;

    my $FileCallback = sub {

        my $CacheFile = $File::Find::name;

        # Remove directory if it is empty
        if ( -d $CacheFile ) {
            rmdir $CacheFile;
            return;
        }

        # For expired filed, check the content and TTL
        if ( $Param{Expired} ) {
            my $Content = $Self->{MainObject}->FileRead(
                Location        => $CacheFile,
                Mode            => 'binmode',
                DisableWarnings => 1,
            );

            if ( ref $Content eq 'SCALAR' ) {
                my $Storage = eval { Storable::thaw( ${$Content} ); };
                return if ( ref $Storage eq 'HASH' && $Storage->{TTL} > time() );
            }
        }

        # Delete all cache files; don't error out when the file doesn't
        # exist anymore, it was probably just another process deleting it.
        my $Success = unlink $CacheFile;
        if ( !$Success && $! != POSIX::ENOENT ) {
            $Self->{LogObject}->Log(
                Priority => 'error',
                Message  => "Can't remove file $CacheFile: $!",
            );
        }
    };

    # We use finddepth so that the most deeply nested files will be deleted first,
    #   and then the directories above are already empty and can just be removed.
    File::Find::finddepth( $FileCallback, @TypeList );                                            <--------- row 204

    return 1;
}

[dolfiz]

UPDATE:
I've obtained the stats of memory conspumtion and cpu load for the virtual machine web server. As you can see there is no evidence of overload. By the way, I did not attach them but the same is true for the db-server, I see the stats and you can trust me.

Anyway the log continues to show error of "File does not exist" pointing the always-empty /var/www/. I'm not skilled on linux systems... should that be the folder of htdocs? Is it correct that is always empty? If so, why otrs tries to point files there?

Here some more example of the errors of today:

Code: Select all

[Wed Mar 05 09:38:55 2014] [error] [client x.x.x.x] File does not exist: /var/www/portal, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=34009

[Wed Mar 05 09:49:33 2014] [error] [client x.x.x.x] File does not exist: /var/www/AVCP-SimogWeb, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=34090
[Wed Mar 05 09:49:33 2014] [error] [client x.x.x.x] File does not exist: /var/www/AVCP-SimogWeb, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=34090

[Wed Mar 05 10:00:49 2014] [error] [client x.x.x.x.] File does not exist: /var/www/portal, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=34172
[Wed Mar 05 10:02:03 2014] [error] [client x.x.x.x.] File does not exist: /var/www/js, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketPhone

[Wed Mar 05 10:20:42 2014] [error] [client x.x.x.x] File does not exist: /var/www/portal, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=34322

[Wed Mar 05 10:27:05 2014] [error] [client x.x.x.x] File does not exist: /var/www/portal, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=34368

[Wed Mar 05 10:36:22 2014] [error] [client x.x.x.x] File does not exist: /var/www/portal, referer: http://x.x.x.x/otrs/index.pl?Action=AgentTicketClose;TicketID=34419

[Wed Mar 05 11:00:25 2014] [error] [client x.x.x.x] File does not exist: /var/www/js, referer: http://x.x.x.x./otrs/index.pl?Action=AgentTicketPhone&Subaction=StoreNew&ExpandCustomerName=1&CustomerTicketCounterFromCustomer=1&CustomerSelected=1&SelectedCustomerUser=318372&CustomerKey_1=318372&CustomerTicketText_1=%22DSTGPP62A01C964P%20DESTRO%20GIUSEPPE%20SEGNALETICA%20STRADALE%20CONSELVANA%20DI%20DESTRO%20PAOLO%20E%20C.%20SNC%22%20%3CDESTRO%40ATLAVIA.IT%3E&Subject=&ChallengeToken=QKoylDU4PS6bwf5osRDj1A1GKIbayqzy

[Wed Mar 05 11:01:01 2014] [error] [client x.x.x.x] File does not exist: /var/www/favicon.ico

etc...

[crythias]

File does not exist: /var/www/portal

referer: http://x.x.x.x/otrs/index.pl?Action=Age ... etID=34009

This means: someone is on the referer page and clicks a link within (where? I don't know... but view source on it...) the link is probably like: "our portal" or something that has an anchor of href="/portal" or "portal"

[dolfiz]

Now I get what you and reneeb mean (I'm a little bit dumb sometimes). Yes, there are links of such kind on body emails and I suppose that apache resolves them locally trying to find something that locally does not exist.

So I can safely ignore those kind of errors, I guess, right?

[crythias]

So I can safely ignore those kind of errors, I guess, right?

Yes, ... sort of. OTOH, it'd be better to properly address the links so they're not broken.

[dolfiz]

it'd be better to properly address the links so they're not broken.

I really really don't care about it. I am concerned only about something that can undermine (is it the proper word?) the otrs logics.

We found some tickets that go through a Generic Agent but they do not experienced all the actions that the ga should have done. For example, the generic agent should have changed queue, state and the value of a field, instead only the queue has been changed. We started to analyze logs for this reason.