@bp2017 -- The width of a boxed n requires explicit math, and suppresses the ability to apply spacing adjustments between the variable and the script. Try setting the n-scripts in the box and the result should have the same width as the i-scripts have. (But of course it won't look as good.)

@barbarabeeton, thank you. I've found this topic: tex.stackexchange.com/questions/565940/… and used \newcommand*{\zni}{\ensuremath{\mathmakebox[\widthof{n}]{i}}} to make it work.

@bp2017 now you fixed the used font sizes, but your kerning still is off. Take a close look, the two "is" still don't align. Your first attempt did use \textstyle for the superscript i, that's why it was so noticably wider.

@bp2017 why do you want to typeset it badly?

@Atex the amount of work I put into a forest answer is essentially inversely proportional to the number of votes it will get and the probability of its being accepted. but my two highest voted answers are both @UlrikeFischer's fault.

@DavidCarlisle is github.com/cfr42/nfssext/blob/devel/.github/workflows/… what you meant? it does work. it just seems circuitous.

@bp2017 if you want the "is" aligned, than align them (with a tabular or a tabbing environment). Or at least typeset all the math in a fixed size box...

@bp2017 what are you trying to acheve? Because whatever it is, doing it by mutilating the typesetting is likely not the right way.

@bp2017 \zni is a textstle math i set to the width of a textstyle math mode n 9assuming you have defined \widthof somewhere with that meaning) so it's wider than a superscript n (ah scrolling down i see you found this, but as @Skillmon says boxing breaks the kerning)

@cfr I'm not sure my comment was based on anything as explicit as usable code just a general concept that you can make later steps dependent on the return status of earlier ones. So if that works, yes that was my brilliant suggestion, and if it doesnt then no, I meant somethin else.

@DavidCarlisle in German, <z> is spelled /ts/, as technically, TikZ stands for "TikZ ist kein Zeichenprogramm"

@enkorvaks i took this tex.stackexchange.com/a/698189/49486 and this emacs.stackexchange.com/questions/53022/…

(not "spelled", but "pronounced")

@DavidCarlisle you saw tex.stackexchange.com/q/735555/2388 ?

@UlrikeFischer I hadn't but I have replied:-)

@Lupino yes I know but you asked how i pronouced it:-), I have heard some people call it "tick-zee" which sounds really weird to me as I would never pronouce z as zee (I leave that to @barbarabeeton)

@DavidCarlisle ;-)

@Lupino The othe answer to that question is mine, and has a link to my L3 mode on github. Have a look, and see what you think. It largely does syntax highlighting only, you will need AucTex if you use any other features for L3.

@cfr I see, and once again, the visual prowess of TikZ is displayed;)

@samcarter oh, showing off with some French :)

@mickep I would have asked @DavidCarlisle to translate, but he doesn't seem to be in chat right now :P

@samcarter He is writing secret emails.

@mickep Don't tell Paulo that there are secrets!

@samcarter Vous avez essayé de me contacter ?

@DavidCarlisle Ha!

@mickep , tu ne peux pas dire que tu as déjà vu ces exemples de multiplication invisibles, sans révéler le contenu secret de l'e-mail

@DavidCarlisle Oh, encore des secrets !

@samcarter non seulement secret, mais invisible

@DavidCarlisle Oh no

@DavidCarlisle How stable are the intent=... keys?

@DavidCarlisle I can do invisible secrets, too!

\documentclass{beamer}
\begin{document}
\begin{frame}
  \invisible{secret}
\end{frame}
\end{document}

@mickep that's a bit complicated. It's all in draft so unstable is the official answer, but we hope to start the REC process soon, but the idea is there are acore set of concepts that all AT should know (include times) these are listed at w3c.github.io/mathml-docs/intent-core-concepts these are supposed to be known but the system can speak them how it wants (that page has French versions for example)....

... but as an authoring tool you can use any concept and if the system doesn't know it it should read it as is so if you want <mo intent="this_looks_like_times_but_is_the_funky_product_defined_in_section_two">×</mo> then it will be read as that.

@mickep so once we move out of "editors draft" stage the core concepts should be pretty stable, but you can use any concept or as a community effort to try to use the same names for the same things there is an "open" list w3c.github.io/mathml-docs/intent-open-concepts that isn't stable at all

@DavidCarlisle Hmm, maybe something for ConTeXt to wait a bit about at the moment. I don't want someone to have a breakdown next time it gets changed.

@mickep yes at the moment it's in flux but firm enough to experiment with. In particular getting mathml (with or without intent) out of tagged pdf and to reach a screen reader that passes it to mathcat to read, needs a 2025 alpha release of NVDA (hopefully that will go live next month) so currently (for pdf not html) it doesn't matter what mathml you use as it isn't used anyway.

@DavidCarlisle I hope it will stabilize. ConTeXt already can add mathml, the question is just what "dialect" that should be used.

@DavidCarlisle It will be interesting to follow.

@DavidCarlisle Thanks for elaborating a bit in the mails.

@mickep well yes we have the same problem. Currently luamml generates mosty mathml-core except bits of mathml3 when it seemed reasonable, that means if you use our runners that generate tagged mathml and open in ngpdf with the mathjax turned off, some things don't work. I coudl of course open in ngpdf with mathjax turned on, then it looks better but it's harder to see what's going on as you don;t see mathml in the derived html inspector you see whatever mathjax made on the fly

@DavidCarlisle Hm, and what is your plan out of this problem?

@mickep get luamml just to use mathml core (I think) but we'll see. Currently it has essentially no options but in the end it may need some, and either by default or as an option staying within mathml-core certaiinly will help some wokflows

@DavidCarlisle Ok! As I said, it will be interesting to follow.

@Skillmon "typesetting it badly" can be interpreted in two ways: (1) the manner in which the code is written to achieve the alignment is wrong; (2) the alignment itself is strange looking ("wrong"). If you meant the first case, that's simply because I'm not good at TeX (the learning curve is too high). If you meant the second case: have you heard of monospaced fonts ;) Sometimes it is usefull to align distinct symbols by a fixed width (and/or height). But even for none of the above, why not? :)

I am (still) struggling with when to use \cs_new_protected:Nn, but now that I re-read a lot of ressources, I think it is quite straight-forward: As soon as there is a command in the definition that is not expandable (which can be looked up in l3interfaces for example), I should use it (except for some edge cases). Would that be right?

@bp2017 If you are going for a mono-spaced result, why not use a monospaced font? Its symbols would at least be designed for this.

@bp2017 try seeing it from this perspective: You're using a sophisticated system which is reknown to produce fine typography, and you're putting in effort to prevent that system from doing its job. If you wanted to show an animation in which some characters are changed to show that two formulas are (in principle) identical, I'd understand this. But if it's about displaying the two below each other: Don't align each individual character, align logical groups (here, align on thet iss)

@JasperHabicht yes. And the edge cases are very few, and you should know when you hit one.

@Skillmon And I take it that I should use \cs_new_protected: for internal functions as well, in order to have clean, understandable and maintainable code ...

@Skillmon I have one case I would consider edgy: \cs_new:Npn \__jsonparse_unicode_char:NNNN #1#2#3#4 { \symbol{ " \str_uppercase:n { #1 #2 #3 #4 } } { } } which ... I am not sure about. I think \symbol is not expandable, but I think it would be good to have \__jsonparse_unicode_char:NNNN expand to \symbol. So, I am thinking of not defining it as protected.

It is probably defined it a suboptimal way anyways ...

@JasperHabicht That would be appropriate - I'd likely have the \symbol separately (e.g. in a tl) as the function itself here is expandable, that's a non-expandable piece of 'payload'

@JosephWright Okay ... so, like define \tl_set_eq:NN \l__jsonparse_symbol_tl \symbol and then use \l__jsonparse_symbol_tl in the function?

@JasperHabicht Well \tl_set:Nn \l__jsonparse_symbol_tl { \symbol } but yes

@JasperHabicht as soon as a function can't work by expansion (and you're not in one of the exceptions) it should be defined protected, doesn't matter whether internal or not.

@JosephWright Oh, right

@Skillmon Exactly

@Skillmon Thanks! I think I slowly get it ... Or rather, I slowly get used to it

@JosephWright Nice, thanks!

@JosephWright symbol is a function, so I'd use \cs_new_protected:Npn \__jsonparse_symbol:w { \symbol }.

@Skillmon Yes, I just wanted to ask whether or not to use \cs_. And then, why not \cs_new_eq:? Would this have some bad implications?

Because with \cs_new_eq: I don't need to worry when omitting the arg spec ...

But well, it's weird anyways ...

@Skillmon No! It can't be protected, because this would not expand to \symbol then ...

Ah, I get it. Not _eq because we want it to have it expand once, which is the whole point of defining it outside the function. I feel this is kind of more a question of esthetics and maybe ideology then ...

@Skillmon Yeah, possible - it's always a value judgement

@Skillmon @JasperHabicht \symbol takes an ordinary argument, so why :w? Likely \cs_new_protected:Nn \__jsonparse_symbol:n { \tex_char:D #1\scan_stop: }

@JasperHabicht Note that \symbol is defined with \DeclareRobustCommand, so you don't want it to appear in some full expansion context.

@egreg True, but the function I define should expand everything as far as possible.

@JosephWright One day there will be an interface to \tex_char:D or \tex_Uchar:D

@JasperHabicht Expanding \symbol all the way takes nowhere…

@egreg No, I mean, I need \symbol{ " \str_uppercase:n { #1 #2 #3 #4 } } { } to expand. Of course, \symbol won't expand any further. So, I probably should define it in another (proper) way so that \str_uppercase is expanded first.

@JasperHabicht \exp_args:Ne \symbol{...}

@egreg Excactly

So, it should be \cs_new_protected:Npn \__jsonparse_unicode_char:NNNN #1#2#3#4 { \exp_args:Ne \symbol { " \str_uppercase:n { #1 #2 #3 #4 } } { } } which works as required.

Or maybe \exp_args:No rather

@JasperHabicht No, that wouldn't do, because it would try to expand ". But why this weird syntax? Note that Unicode characters may have up to six hexadecimal digits. Why not \__jsonparse_unicode_char:n?

@egreg \codepoint_generate:nn?

@egreg Because JSON only allows for 4 hex digits and the idea is to parse JSON. A valid JSON string could be abc\u0064ef which should then result in abcdef and not in abc擯.

This is also the reason for the {}

@JasperHabicht but why use \symbol in the middle of all that, rather than (an expl3 name for) \char or \Uchar ? also why 4 arguments, what about U+1D400 𝐀

@DavidCarlisle For the latter see my answer above. Yes, I could also not use \symbol but \tex_char:D but this is probably also a non-expandable token?

@JasperHabicht Ug do you need to use UTF-16 I guess I knew that once

@JasperHabicht if you are in luatex you can use \Uchar which is expandable

@DavidCarlisle No, I want to parse JSON. It only knows \uXXXX and that's it

@DavidCarlisle I am in whatever the user chooses. =) It should be engine-compatible.

@JasperHabicht but can't you use \u1234\u1234 to give the UTF-16 surrogate pair for the higher slots?

@DavidCarlisle Probably yes. But I am unsure how this is related?

@DavidCarlisle Or \codepoint_generate:nn, which works with everything :)

@JasperHabicht \uD835\uDC00 for bold A

@JosephWright it doesn't recombine utf-16

@DavidCarlisle Well no, but you could handle that in the numexpr argument

@DavidCarlisle But I still need to be able to somehow grab exactly four tokens after \u. There is no delimiter in a JSON string. So, this is why I use :NNNN.

@JasperHabicht because if parsing json and hit \uD835 you can't simply convert that as it's not a legal xml character you need to see it's the first half of a surrogate pair and combine it with the following utf-16 slot

@JasperHabicht yes agreed

@DavidCarlisle Okay, I see ... well but this is then probably another question. Thanks for pointing to this!

@JasperHabicht I have spent a long time in those higher Unicode panes :-)

@DavidCarlisle Good you found your way out to the lower panes =D

@DavidCarlisle Adventures in Unicode-land!

@DavidCarlisle That is where you have found your favorites?

@JasperHabicht 𝙰𝘳𝗲 𝕪𝒐𝕦 𝕤𝕦𝙧𝘦 ℐ 𝖉𝙞𝖽 𝘨𝗲𝑡 𝚘𝖚𝓉?

@DavidCarlisle Somehow I get a different output when I replace \symbolwith \char (or \tex_char:D)

@DavidCarlisle hmm ... thanks. it does work. (or rather it doesn't work in the right way.)

@JasperHabicht missing a trailing \relax (\scan_stop:` )? symbol is

\ifdefined\XeTeXversion
  \DeclareRobustCommand\symbol[1]{\Ucharcat#1 12\relax}
\else
  \DeclareRobustCommand\symbol[1]{\char#1\relax}
\fi

@JosephWright I'd forgotten we had to special case xetex :(

@DavidCarlisle Shouldn't that be ;?

@DavidCarlisle :)

@DavidCarlisle Yes, as well ... the problem is \str_uppercase it seems

@DavidCarlisle \codepoint_generate:nn knows all of this

@JasperHabicht Do you want a programmatic string?

@JosephWright yes, I was just looking up what symbol did

@DavidCarlisle :)

@JosephWright \exp_last_unbraced:NNe \tex_char:D " \str_uppercase:n { #1 #2 #3 #4 } \scan_stop: is not working

@JasperHabicht Er, you are passing four (probably) byte tokens not one Unicode codpoint

@JosephWright Oh, right

@JasperHabicht expl3 is all UTF-8 based

Ok, I think, it works now

@JasperHabicht What did you change?

@JosephWright NNf?

@JasperHabicht Ah, you've applied \exp_last_unbraced:NNe but you meant \exp_last_unbraced:NNNe as you've forgotten that " is a token - probably you could use \exp_last_unbraced:Ne \tex_char:D { \str_uppercase:n { "#1#2#3#4 } }\scan_stop:

@JosephWright But do I understand NNf correctly then? It takes a whole command including its arguments and expands that first?

@JosephWright Yes, This was wrong

@JosephWright Wow! What a long chat only to have one command conform with expl3 syntax =D

@JasperHabicht Well f-type expansion can be 'open ended', so here it keeps going and fins the argument to \str_uppercase:n - e-type expansion doesn't do that

@JosephWright Okay, but this would also be fine here, I think?

@JasperHabicht I do keep saying \codepoint_generate:n(n) :)

@JasperHabicht Yeah, but more by accident than design

@JosephWright Then, I will go with your \exp_last_unbraced:Ne variant

Now to surrogate pairs ...

=D

@DavidCarlisle Okay, it is pretty easy to implement a surrogate-pair-to-UTF-16 converter in TeX. What is not straight-forward is the question: how should I handle this during parsing ...

@JasperHabicht l3str-convert probably has that already

Check whether the hex value is in the low-surrogate range. If no: handle as before. If yes: store it and check whether \u follows. If not: dump stored stuff (even if this would result in invalid output). If yes: check whether hex value is in the high-surrogate range. If no: dump. If yes: calculate UTF-16 value from both hex values > return that

@JasperHabicht Looking at the entire thread, I think you don't want just to print a character, but to expand \uXXXX to a real character, so \codepoint_generate:nn or \codepoint_str_generate:n. If you parse \u you look for the next four tokens which should be hexadecimal digits. Then you pass them to \codepoint_str_generate:n and you get the real character.

@egreg Probably yes, now that I think about it.

@egreg good suggestion, if only @JosephWright had thought to suggest that

@DavidCarlisle Lucky you there are no ticks to steal inhere.

@egreg \codepoint_str_generate:n expects a hex value? Or do I need to mark it as hex using "?

@JasperHabicht no a number so you need to mark hex input

@DavidCarlisle Thanks!

@egreg yes, my bad. I was in a hurry and started of with a \cs_set_eq:NN to \tex_char:D` which I then incompletely changed.

Okay, I got it working. The parser now prints a\u0040c as "a@c" and a\uD801\uDC37c as "a𐐷c".

New functions: \jsonparse_unicode_if_low_surrogate:nTF, \jsonparse_unicode_if_high_surrogate:nTF and \jsonparse_unicode_combine_surrogates:nn (these might be useful, so I don't make them internal).

The last command is just \int_eval:n { ( #1 - "D800 ) * "0400 + ( #2 - "DC00 ) + "10000 } =)

\codepoint_str_generate:n yields different results than \char with PDFLaTeX ... for example \char"00D1 outputs Ñ, but \codepoint_str_generate:n{} outputs ÃŚ =/

I get the N tilde

\documentclass{article}

\begin{document}
\ExplSyntaxOn
\typeout{A:
  \codepoint_str_generate:n{"D1}
  }
\typeout{B:
  \char"D1\space
}
\ExplSyntaxOff
\end{document}

A:Ñ
B:\char "D1

@JasperHabicht oh if you typeset yes str functions generate catcode 12 tokens and you need them to be active to trigger inputenc processing

@DavidCarlisle Aaah, that's it

@JasperHabicht see test Y

\documentclass{article}
\usepackage[T1]{fontenc}
\begin{document}
\ExplSyntaxOn
\typeout{A:
  \codepoint_str_generate:n{"D1}
  }
\typeout{B:
  \char"D1\space
}


X \codepoint_str_generate:n{"D1}
\par
Y \codepoint_generate:nn{"D1}{13}

\ExplSyntaxOff

\end{document}

@JosephWright sorry, took me two days for the review, yesterday evening I was busy sleeping next to my two little ones

@DavidCarlisle But is it a good idea to generally set catcode to 13?

@JasperHabicht no it needs to be 12 (or 11) if less than 127 and 13 if greater (so you can do an expl3 version of \codepoint_generate:nn{#1}{\ifnum#1 > 127 13 \else 12\fi}

@DavidCarlisle Okay, yes. So 11 only for [a-zA-Z] I assume ...

@JasperHabicht but only if using an inputenc based input 8 bit tex, for luatex or xetex just access teh character directly

@DavidCarlisle okay

@Skillmon :)

@DavidCarlisle :)

@JasperHabicht if typesetting catcde 11 and 12 are essentially the same thing, either way it'll only work to the extent that the first 127 characters in T1 encoding are similar to unicode, unless you make all the non letters active and map T1 encoding

@JasperHabicht You want to use \codepoint_generate:nn as this will ignore arg 2 in pdfTeX if the char needs to be active

@JosephWright ooh I'd forgotten it did that

@DavidCarlisle I needed it for siunitx ;)

@JosephWright Uhm, but which catcode should I assign?

I just go with 10. If someone really enters \u0061 then ... well. I have a rescan function in the package anyways.

@JasperHabicht Something like I do in siunitx:

\cs_new:Npn \__siunitx_unit_non_latin:n #1
  { \codepoint_generate:nn {#1} { \char_value_catcode:n {#1} } }

i.e. the catcdoe that would be assigned if you typed in

@JosephWright I don't think @JasperHabicht wants that though. If a JSON file references { you want a catcode 12 { not a group start (same for & etc)

I will look into that again after I am done with rewriting the \u conversion function to be expandable

But let's see what will happen =D

I'll stick to catcode 10 first. I need to test the new functionality first

@JasperHabicht not 10 surely? that will make all characters act like space?

@DavidCarlisle No 12. Sorry ... other

@JasperHabicht better:-)