FAQ 2.0.8 for OTRS 3.0.22

[Mios]

Hi,

I am in process of upgrading OTRS 2.2.7 to 5 (+ ITSM, + FAQ).
Everything goes more or less good, but now I am stuck.

I upgraded from 2.4.15 to 3.0.9 and it went smooth. ITSM 1.3.2 -> 3.0.9 was also easy.
And then I updated FAQ from 1.6.6 to 2.0.8 and it just doesn't work. At all.

Entire OTRS seems to work without a problem. But when navigating to /public.pl or /faq.pl aor just using FAQ link in the agent menu, the website hangs. It returns Gateway Timeout (504) after very long time.
When executing /opt/otrs/bin/cgi-bin/public.pl the console also hungs. Just like there is infinite loop somewhere.
Syslog doesn't containt any relevant entries.

What can it be? Or how to start looking for it? I do not have any clues from OTRS about the cause.


I already run setPermission few times, because I though that upgrading ITSM opms may have broken something.
I do not use the 3.0+ version approach of unpacking files to another /opt/otrs/x,x,x and creating symlinks - instead I operate whole time in one dir and only copy over new tarballs. I though it could be the problem, so I swapped /opt/otrs (already updated) with clean 3.0.9 and run the "reinstall" command for FAQ. At start it gave strange error, but setPermissions resolved it. And it is also hanging. The only config files which I copied from the other instance were Config.pm, GenericAgent.pm and ZZZAuto.pm.
So, I do not think the cause is in some old, non-deleted files. Perhaps it is in my config files? But there is no "FAQ" occurence in them.

Maybe I could skip ot further updates and the problem would dissapear? But I am afraid about my data. Uniinstalling the plugin is also impossible, because it deletes the data, am I right?

A am using perl 5.10 if this can mean anything.

Thanks in advance.

[Mios]

Hi again.

I tried skipping this version and went directly for OTRS 3.1.21.
Upgrade went ok, OTRS was up and running. FAQ of course was the same - just hanging, 0 response.
Then I decided to leave ITSM not upgraded, and go straight for FAQ 2.1.4 (from not working 2.0.8 ).

Just after upgrade through web admin, everything was the same - OTRS working, FAQ - not. Then i run again SetPermissions.pl and restarted all services.
And now NOTHING works. I only get the login page at /index.pl. I put my credentials. Syslog gives entrey about succesfull auth, but the webpage hangs until 504 Gateway timeout. No errors, no logs. Nothing.

I tried cross checks - with completly clean OTRS 3.1.21 installation and with clean OTRS DB,
The results are:
upgraded system + upgraded DB -> not working (as described)
clean system (+ freshly installed FAQ) + clean DB -> working!
clean system (+ freshly installed FAQ) + old, upgraded DB - not working

I didn't yet run old system + fresh DB test. But it seems that the problem lies within the DB. This DB comes from version 2.2.7 with consecutive upgrades, and until OTRS 3.0 everything was fine.

Since the DB is the essential part of the upgrade (without it I would just install fresh 5.0), it has to work.
I suppose, that there might be some circular reference created by migration scripts (maybe circular object links?) which cause perl scripts to run in circle? While waiting for any script to timeout, linux top commands shows 100% CPU for this script and very little memory (like 2%).



Where to look?
Does anybody have an idea what can I check to find the bug and repair my DB?
Thanks

[reneeb]

Are there any entries in the webserver log?

[Mios]

Are there any entries in the webserver log?

There were none, except that the script timeout.

BUT! I think I resolved the issue. Since somebody may stumble upon it, i will describe it.

First - how to get any information about what is going on, where there is no log available? And you have 0 experience with perl (but you know anything about the code). Google it. Google gives hints for debugging perl here:

Code: Select all

http://docstore.mik.ua/orelly/weblinux2/modperl/ch21_07.htm

I used 21.7.4.1. Using the Perl trace, so I editet $OTRS_HOME/bin/cgi_bin/index.pl and added

Code: Select all

package Book::StartUp;
use Carp ( );
$SIG{'USR2'} = sub { 
    Carp::confess("caught SIGUSR2!");
};

before OTRS "use" clauses.
So, now index.pl script is aware of USR2 signal, and will print stacktrace when received.
I launched website. When hit "login" button, site is permanently logging in until timeout (dashboard was the cause, as it turned out - and specifically FAQ summary on the dashboard).

In the meantime I found this scrip PID through:

Code: Select all

ps aux | grep index.pl

and send this signal, effectively killing the process:

Code: Select all

kill -USR2 <found_pid>

This gave me beautiful stacktrace in apache2 error_log. The most nested function was in FAQ.pm :2033 line.
It was building CategoryTree for FAQ.

The issue

First thing that comes to my mind (and I already suspected some infinite loop, because hanging process, 100% CPU, etc.) - there is circular dependency in category definition. Namely, the tree is not a tree, but graph, and more - it has cycles.

Code: Select all

select * from faq_category;

Then copy-paste to excel (I had only 43 categories).
I found 2 entries which had id == parent_id. So, microscopic, but still - cycle in a graph.

Code: Select all

UPDATE faq_category SET parent_id = 0 WHERE parent_id = id;

SOLVES THE ISSUE.

I am not sure, but categories are a feature of FAQ in between OTRS 2.4 and 3.0 (so between FAQ 1.6.x and 2.0.8 ) I think. I do not know, where nor when this bug arrises, but is not corrected by any migration scripts / plugin upgrades.
Hope it helps somebody.