A recent upgrade of www.zappelin.nl brought this to light:

1. utf-8 present in HTML HEAD;
2. generated Rails app with all utf-8 defaults present;
3. mysql database still had latin1 as default character set;
4. same for the table;
5. using the old mysql driver gem.

The above combination led to strings still being stored as latin1 in the database.

MySQL documentation mentions charset conversions using ALTER TABLE. These did not work — the offending characters remained in their original character set. In the MySQL console these were printed fine: for instance ë was visible in the correct places. However in the Ruby world this was displayed as ë. Wuh?

I briefly considered dropping down to unix and using iconv on the database dump.

Luckily I was just too lazy today. Turns out you can convert the offending column to BLOB first and then to the correct character set. This worked! So if you have this problem the query below might fix things.

ALTER TABLE tbl_name MODIFY col1 BLOB;

ALTER TABLE tbl_name MODIFY col1 VARCHAR(255) CHARACTER SET utf8;

However, it still needed a bit of magic with regards to the default charset on the table. My best combination was to first convert the entire table to use latin1 by default, then run the conversion, then set the default to utf8. However, this didn’t work always.

ALTER TABLE tbl_name CONVERT TO CHARACTER SET latin1;

Keep in mind to always use the mysql2 gem when running under Ruby 1.9.2. Anything else is rubbish since this will not know how to deal with UTF-8 and you app will break and you will cry.

My wish for 2011: a definitive end to character set issues plz. This really is no fun.

Of course, per all the advice that’s out there, I needed to
run some nasty-ass updates directly in MySQL. Buh. Even in my utf-8 Terminal.app I could not display some weirdness properly. So it came down to hex() and unhex()’ing in MySQL. Joy!

Even if it’s only for myself, here is the code that was used:

update articles set body = replace(body, unhex('C3A23F3F'), "'") where body regexp unhex('C3A2');

and let’s not forget this beauty:

update articles set body = replace(body, unhex('C3A23FC29D'), "'") where body regexp unhex('C3A2');

both of which returned various combinations of ??? â?? and Japanese characters to a simple apostrophe.

And no, the character_set_server still isn’t utf-8. Most data is fine now though.

The best part was when the regexp engine of MySQL encountered the actual question marks as a result of the unhexing. This threw all parsing down the drain, since question marks need to be escaped in a regexp. But … as you’d have guessed … those weren’t actual question marks in the database. Just the same bit combinations. Double joy!