ERROR: invalid byte sequence for encoding "UTF8"

[coelhu]

Hello

This error appears when the client creates a new ticket in the screen after pressing Send

The ticket is created in otrs but all blank and the client does not receive mail confirmation

When debug is enabled (Frontend::DebugMode) this error does not happens, the ticket is created normally!

The log goes bellow, any help is appreciated.

Thanks

Code: Select all

ERROR: OTRS-CGI-50 Perl: 5.18.2 OS: linux Time: Wed Oct 22 16:06:48 2014

 Message: ERROR:  invalid byte sequence for encoding "UTF8": 0xe76f73, SQL: 'INSERT INTO article (ticket_id, article_type_id, article_sender_type_id, a_from, a_reply_to, a_to, a_cc, a_subject, a_message_id, a_message_id_md5, a_in_reply_to, a_references, a_body, a_content_type, content_path, valid_id, incoming_time, create_time, create_by, change_time, change_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 
        '2014-10-22 16:06:48'
    , ?, 
        '2014-10-22 16:06:48'
    , ?)'

 RemoteAddress: 172.20.10.114
 RequestURI: /customer.pl

 Traceback (2920): 
   Module: Kernel::System::Ticket::Article::ArticleCreate (OTRS 3.3.8) Line: 264
   Module: Kernel::Modules::CustomerTicketMessage::Run (OTRS 3.3.8) Line: 573
   Module: Kernel::System::Web::InterfaceCustomer::Run (OTRS 3.3.8) Line: 1000
   Module: ModPerl::ROOT::ModPerl::Registry::opt_otrs_bin_cgi_2dbin_customer_2epl::handler (unknown version) Line: 41
   Module: (eval) (v1.99) Line: 206
   Module: ModPerl::RegistryCooker::run (v1.99) Line: 206
   Module: ModPerl::RegistryCooker::default_handler (v1.99) Line: 172
   Module: ModPerl::Registry::handler (v1.99) Line: 31

[jojo]

OTRS version, database?

[coelhu]

OTRS version, database?

OTRS 3.3.8 - Postgres 9.1

[jojo]

might be an issue with a special character which can not be used in the database (I heard of a mysql bug for the same)

[coelhu]

might be an issue with a special character which can not be used in the database (I heard of a mysql bug for the same)

I will check it up and return later, Thanks

[coelhu]

Hello

The problem was that I have a queue called "Fila de Serviços" and the "ç" character was given the error, I changed it to "c" and it solved.

But this does not resolve the issue that PostgreSQL is not accepting "especial" characters like "ç" and this error could occur again in the future.

How do I fix this so that PostgreSQL accepts "ç"?

Is the problem with PostgreSQL or with OTRS?

UTF8 should support these characters:
http://en.wikipedia.org/wiki/UTF-8
Latin-1
0080
194
U+00E7 ç Latin Small Letter C with cedilla


Thanks

PostgreSQL version 9.1.12 encoding UTF8
OTRS 3.3.9

[amolina]

I am afraid that the error does not lie in PostgreSQL. The error is in ZZZAuto.pm (the way it's parameters are used, really).

Whenever you edit a parameter in the SysConfig OTRS interface, this parameter is correctly saved as UTF-8 encoded. But the rest of the international characters are stored in the form \x{yy}, i.e. it is stored as Perl source encoding. For example, the character ó (o-acute) is stored as \x{f3}.

You end with a ZZZAuto.pm file that has mixed UTF-8 encoded characters and Perl source code style characters. And for some reason that I have not been able to debug, having this Perl source code style coded characters (in my case \x{f3}) leads to your reported error.

[amolina]

Apparently, the encoded sequence of characters in ZZZAuto.pm is not decoded right.

In my case, I have a queue named "Tramitación" (Spanish name) as the default queue.

1. If the name is encoded as UTF-8 everything works fine;

2. If the name is encoded as "Tramitaci\x{f3}n" then OTRS will decode and reencode it as bad UTF-8. OTRS will read \x{f3} and instead of treating it as a Perl entity, getting the character and reencoding it as UTF-8, it gets it as the 0xf3 (hex code) character. In this case, the last two characters will be encoded as 0xf3 0x6e (0x6e is the UTF-8 code for the letter "n") and sent to PostgreSQL. But this is a wrong UTF-8 encoding pair, it should have been 0xc3 0xb3 0x6e (the acute o is 0xc3 0xb3 in UTF-8).

So OTRS is taking the sequences \x{yy} and passing them directly as hex values to an UTF-8 expecting server, instead of treating them as Perl source code encoded characters and reencoding them before sending the sequence to the sever.

But for some reason if Debug is enabled everything works fine.

[amolina]

Apparently, this weird behavior depends on the SysConfig screen used. For example, if I edit the default queue in Ticket -> Frontend::Customer::Ticket::ViewNew, all of ZZZAuto.pm parameters are encoded properly in UTF-8 except for the parameters in Ticket -> Core::PostMaster.

If I then go to change the default queue in Ticket -> Core::PostMaster, it is encoded properly but all of the other parameters in OTRS are reencoded in the wrong way.

So the wokaround for now is:

1. Change whatever you need to change in Ticket -> Core::PostMaster.

2. Change other parameters in other sections of SysConfig.

3. Edit ZZZAuto.pm carefully and change the \x{yy} codes that have been created for the parameters in Ticket -> Core::PostMaster to their correct UTF-8 counterpart characters.

4. If, for whatever reason, you need to change any parameter in Ticket -> Core::PostMaster using SysConfig, the go to any other area like Ticket -> Frontend::Customer::Ticket::ViewNew and save without changing anything. Then repeat step 3.