UTF-8 bug with Zope 2.10.6

[tom]

From Takeshi Yamamoto: I have found this rc1 is based on Zope 2.10.6 which has this bug.

See https://bugs.launchpad.net/zope2/+bug/160968 for detail (bug description included below)

I have attached the image file of how the result comes out when title was entered with Internet Explorer. (See screenshot here: http://educommons.com/trac-files/eduCommons-Takeshi-Bug.png) The first and third item has wrong characters displayed. My question is "Is there any chance to use Zope 2.10.8?", since it has no bug on this.

Full bug description here:

Default IUserPreferredCharsets' use of Zope 2's request problematic

The IUserPreferredCharsets implementation of Zope 3 found in zope.publisher.http.HTTPCharsets has the following condition in it to check if the HTTP_ACCEPT_CHARSET header is available:

header_present = 'HTTP_ACCEPT_CHARSET' in self.request

However, with Zope 2's request will return (the empty string) for any header that starts with 'HTTP_', see ZPublisher.HTTPRequest.HTTPRequest.get.

Ultimately, this results in the HTTPCharsets.getPreferredCharsets to return iso-8859-1?, where it should really return 'UTF-8'.

To understand this problem better, look at Products.Five.browser.decode.processInputs, which uses the negotiator to find out which charset to use to convert form variables. For browsers that do not send the 'HTTP_ACCEPT_CHARSET' header, this will result in wrongly encoded form values. To reproduce this, fill in Chinese characters to any Five formlib form with Internet Explorer 6.0. Since Firefox sends HTTP_ACCEPT_CHARSET, it's not a problem there.