AJAX Magazine: Prototype Encoding Issue

Another encoding issue for AJAX libraries after fixing something similar last year with AjaxAC and middle eastern encoding, yesterday I noticed that prototype have similar issue. The problem with prototype is the usage UTF-8 as default encoding even if your page have a different encoding, this will make the data exchanged between client and server incorrect, and in our cases the SQL executed in server side return no results.

I grabbed today the latest prototype 1.5.0_rc1 included with script.aculo.us, but be sure to get the latest one from their website because the script have changed and version remain the same. In the latest version you can notice a new parameters which have been added to the options "encoding" :

In this case be sure to set the appropriate encoding that your page is using for example windows-1252, windows-1256 ... etc. It should solve the problem if ... encodeURIComponent() wasn' t used for escaping special characters.

And that's the second issue for Prototype which is the usage of encodeURIComponent() which replace each instance of certain characters by one, two, or three escape sequences representing the UTF-8 encoding of the character. The problem here that all escaping functions replace non-ASCII characters with Unicode ones, and in this case even if you set the unicode by default to windows-1252 for example, a special character will be replaced by its equivalent in unicode.

The solution

There is two possible solutions, first find a replacement for the escaping method, and the second solution which is more easy to implement is using PHP iconv extension :

So even if Prototype send data in Unicode you can get it again in its original encoding and continue your process. Will face this bug anybody who work with middle eastern or asian encoding, even french, german, dutch and any encoding which using non-ASCII character. If you use already UTF-8 in your application and database, you should be safe, and won't face this issue. Let me know if someone have found similar issue for other AJAX libraries.

Excellent post. This is not an issue of Prototype library - the encoding attribute of Ajax.Request has nothing to do with the way encodeURIComponent() works, which may be surprising to some.

The server-side iconv solution seems indeed the simplest.

Posted by Mislav | November 24, 2006 4:03 PM | Reply

excelent post! thanks!

header('Content-type: text/html; charset=iso-8859-1');
//+
$texto = iconv("UTF-8", "iso-8859-1", $_POST['texto']);

that saved my life

Posted by KnF | January 17, 2008 10:33 PM | Reply

Thank you for your advice.

I tried many times to solve this problem, but it's not working. Still print "garbage data".

I used windows-1256, and I used this :

thank you

Posted by alhoymli | January 18, 2008 12:57 AM | Reply

I am using classic ASP and my pages and db are in windows-1251 encoding. Now I am trying to use AJAX and facing this problem. What am I suppose to do when there is no ICONV equivalent in ASP 3.0. I have been looking all over Internet for somebody who has already created such function without luck.

Posted by Georgi | March 1, 2008 9:13 PM | Reply

This is the method I call onboady load:

function getInitialTree(){
tree=new dhtmlXTreeObject('treeBox',"100%","100%",0);
tree.setImagePath("../../images/");
tree.enableIEImageFix(true);
tree.attachEvent("onClick",onNodeClick);
url = "/treecontroller?Action=GetDhtmlxTreeXML";
tree.setXMLAutoLoading(url);
tree.loadXML(url+"&framework.actionContext=20");
}

It gets a perfectly valid XML file encoded in iso-8889-1 (but could be anything else);

The problem I get is that the tree doesn't show on screen, instead I get a msg saying that the the XML is incorrect and when I hit "Generate Report" it shows me my XML file with the right encoding in its header, but the special french characters I use get all scrumbled. How is this possible and how can I solve it? I get the impression that whatever the encoding indicated in the header, the tree translates it into utf-8.

Btw, I know the file is ok, because I am able to see it in the browser when consulting the same url (IE). If I change the encoding to utf-8 in the XML file on the other hand, IE throws an exception.

Thanks.

Posted by Yves | March 17, 2008 5:18 PM | Reply

Georgi, try to put the following on the ASP page that you call with a ajax request :

Response.CodePage = 65001
Response.CharSet = "utf-8"

this did the trick for me

Posted, in reply to Georgi's comment, by reza | March 31, 2008 8:47 PM | Reply

This article is more than GREAT, you saved my time ... I was going to use the charset convert class.
For those who may interest in using a class to do this:
http://www.phpclasses.org/browse/package/1360.html

Best regards
Oras

Posted by Oras | June 9, 2008 6:04 PM | Reply

still i have the same problem
i am using protoform script and prototype 1.6.0.2 to validate and pass the data with(windows-1256) from html form to php file put the result is bad characters
, please any idea

this is the protoform site WWW.CSSREVOLT.COM

Posted by ramis hashim | September 30, 2008 10:35 PM | Reply