The Loop

2:17 AM

@JackJohansson, sarcasm do not work here as well ;)

10 hours later…

12:04 PM

@TomJNowell The database is a combination of utf8mb4_general_ci and utf8mb4_unicode_ci .... the posts table happens to be unicode.

That's what it's looking like....

@TomJNowell I've modified the code to load the DomDocument like this:

$content = mb_convert_encoding( $content, 'utf-8', mb_detect_encoding( $content ) );
$content = mb_convert_encoding( $content, 'html-entities', 'utf-8' );

if ( $dom->loadHTML( $content ) ) {

so now it PREVENTS the weird encoding from happening, but it's not CORRECTING the already weirdly encoded content.

When I look in the database, I see this: â€&oelig;last updatedâ€

So maybe it is working, but I'd like to see it look like "last updated"

and also, is there a way to prevent the DomDocument from saving an entire HTML document into the post_content ? I just want to modify the post_content's HTML, not create an entire HTML document INSIDE my post_content

12:27 PM

I guess I'm going to just have to map these weird entities to the proper entities

12:37 PM

are you sure you're not using a non multibyte aware function somewhere to modify the string?

for the content that already has the issue, the content has been changed, you're going to have to go through and fix the issues manually

@TomJNowell I'm not sure what you mean by that statement.

usually via a search replace

@MichaelEcklund not all string manipulation functions are multi-byte aware

What does that mean.

e.g. a takes up a single byte

but a chinese character might take up 2

if you use a function that isn't aware of multibyte encoding, such as str_replace

it will slice that chinese character in half, and you get 2 characters

Basically I was just going to run something like this on the content (after its been modified):

12:39 PM

which show up as a second character that looks weird

function map_microsoft_entities_to_utf8_entities( String $content = null ) {

	return str_ireplace(
		array(
			'&acirc;&euro;&ldquo;',
			'&acirc;&euro;&trade;',
			'&acirc;&euro;&oelig;',
			'&acirc;&euro;'
		),
		array(
			'&ndash;',
			'&apos;',
			'&quot;',
			'&quot;'
		),
		$content
	);

}

Q: str_replace() on multibyte strings dangerous?

which is usually similar to what you're showing

there you go

str_ireplace

secure.php.net/manual/en/ref.mbstring.php

Given certain multibyte character sets, am I correct in assuming that the following doesn't do what it was intended to do? $string = str_replace('"', '\\"', $string); In particular, if the input was in a character set that might have a valid character like 0xbf5c, so an attacker can inject 0xb...

e.g. strlen vs mb_strlen

although I would have replaced your example with one that used an array and a loop

I'm mistrustful of functions that try to simplify the action of doing multiple things

that's what things like map reduce each etc are for

What are you saying?

There's a better way?

I don't think that tom is not right here. There must be an encoding mismatch somewhere

$replacements = ['&acirc;&euro;&ldquo;' => '$ndash;', etc ];
foreach( $replacements as $k => v ) {
    $content =str_replace( $k, $v, $content );
}

12:47 PM

str_replace should work on utf-8 string even without knowing they are multibyte

yes, my googling suggests str_replace is indeed multibyte aware

but most of the other string functions have multibyte versions

is it okay to use str_ireplace()?

strlen for sure needs to be multibyte aware

I mean what if it's in caps for some reason?

... if you want the number of chars and not the number of bytes

12:50 PM

so use str_replace(); and just hope it's all lowercase.

@MichaelEcklund I would guess that those characters come from you not knowing or handling correctly the encoding of your MS content

Thsi is a large multi author website. People copy/paste content from MS all the time.

IMHO your replace is multibyte safe if content is utf-8

my last question I guess would be, how can I make sure that there's not <html><head><body> ..... etc, being appended to my post content after modifying it with DomDocument?

it is sent to the server as utf-8, so there should not be any encoding mismatch unless it is the DB configuration that is wrong and the characters are added on insertion to DB. I understand very little about DB configuration so will not try to guess

12:55 PM

ooo nice

Apparently you just do this: $html->loadHTML($content, LIBXML_HTML_NOIMPLIED | LIBXML_HTML_NODEFDTD);

... when doing saveHTML() there will be no doctype, no <html>, and no <body>.

`LIBXML_HTML_NOIMPLIED` turns off the automatic adding of implied html/body elements
`LIBXML_HTML_NODEFDTD` prevents a default doctype being added when one is not found.

php.net/manual/en/libxml.constants.php

@TomJNowell @MarkKaplun Thanks for your help guys. Appreciate it.

@MichaelEcklund the editor should be handling copy paste of MS content long before WP sees it

Well... It's not lol.

They paste it in there. Then on save_post, I trigger the content modification via DomDocument HTML processing.

After processing the HTML, the content is encoded incorrectly.

This is a seemingly good workaround now.

are they posting this into the main content area, or somewhere else/custom?

It's a DIVI powered website. DIVI stores everything in post_content

1:11 PM

true, but does Divi handle MS Word copy pasta like the classic editor and gutenberg do?

it could just be a naive text box

1 hour later…

Sanzeeb Aryal

2:37 PM

I'm back!

3:12 PM

😱

2 hours later…

5:18 PM

how to you use Postman to make a post request to REST API?

Always returns false for me

Do you have to specify certain headers?

or is it not working because of nonces?

4 hours later…