DOMDocument loadHTML and double UTF8 encode

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

i use DOMDocument loadHTML to parse pages from this forum (the forum is in utf8) my problem is some
pages are actually seen as utf8 like this one and some are not
  like this one
and so the second kind results in a double utf8 encoding

so i test if the page is doubly encoded and if yes i utf8_decode the
text value i want but there are some side effects, for exemple the euro
sign is not doubly encode so this one become crap when utf8_decoded ...
and i don't know if there are other signs like that

so i am lost (and pissed) any idea how i should manage all that ?

Re: DOMDocument loadHTML and double UTF8 encode

roger21 wrote:
Quoted text here. Click to load it

Strange, because when I load those pages, both are seen as being UTF-8
encoded by Mozilla Firefox 3.6.11 on my system (Linux Ubuntu Hardy 8.04
LTS), and everything seems to display correctly.

But, as I recently discovered, a lot will depend on whether the forum's
server is actually issuing a PHP header with a UTF-8 charset declaration
before displaying the pages instead of merely using an HTML meta tag. If
not, the page might still be displayed as some other character set, for
example ISO-8859-1 ... which wouldn't be so bad if the extended/accented
characters were correctly translated as HTML entities by the forum
software, and obviously they are not (i.e. the source for the pages
shows "á" in plain text instead of "á").

Of course, the forum member's client browser might not send messages
encoded in UTF-8, and the result can be garbage -- Russian text encoded
as Windows 1251, for example, and copied and pasted from an editor into
the browser will sometimes display as ISO-8859-1 extended characters in
some forums I have seen.

One thing you might want to try is to compare some original text with
the result of a utf8_encode(utf8_decode(orig_text)) combination, or
maybe vice-versa. If they are the same, then the page is being
interpreted as UTF-8; if not, it is most likely being interpreted as
ISO-8559-1 or some other character set.

Also, the user comments posted here might prove to be helpful:

Re: DOMDocument loadHTML and double UTF8 encode

Robert Hairgrove a écrit :
Quoted text here. Click to load it

thank you for your answer, when i say "seen" i mean by the loadhtml
function that translate everything in utf8 if it thinks it is not
already the case, of course both pages are in utf8 (by the headers and
the meta tags and the content therefore my double encoding problem) and
that's why i'm a bit upset (but i won't say the forum's pages are clean,
it is prety crapy over-all)

and my problem is when i utf8_decode my doubly encoded pages i have
characters issues that i don't have when the page is not over-encoded by
loadhtml (with the same chars)

and i don't whant to decode the text the first because i want to stay in
utf8 (and i will lose characters if i decode it first)

and i alredy checked the coments, they all seems to be related but i
tried most of them and it is either the same or worse (therefore i ask
here :p)

maybe i should try to understand why some page are doubly encoded, they
may have some crap that i could fix before giving it to loadhtml

Re: DOMDocument loadHTML and double UTF8 encode

roger21 a écrit :
Quoted text here. Click to load it

ok, one of the comments seems useful : the title tag is before the
charset meta tag, when i have a title with accents loadhtml over encode,
if i move the meta tags before the title tag that should be good
(according to the comment) i will try that

Site Timeline