@JosephWright (anyone) am I missing something here or take to list?

@DavidCarlisle Hmm

\catcode`\_=12 %
\directlua{tex.sprint("[" .. token.get_next().cmdname .. "]")}\par
\directlua{tex.sprint("[" .. token.get_next().cmdname .. "]")}\UNDEFINED
\bye

@DavidCarlisle ^^^

@DavidCarlisle I think the issue is that \UNDEFINED doesn't link to a csname so Hans' approach is this data point is empty

@DavidCarlisle I guess Hans hasn't store the user input but rather a link to the csname definition, so with an undefined csname there's 'nothing to see here'

@DavidCarlisle It does say

> the associated control sequence (if applicable)

Note the part in brackets!

@StrongBad :)

@JosephWright morning/afternoon

@JosephWright yes but still seems odd \UNDEFINED is a control sequence token with name UNDEFINED even if there is not a control sequence of that name. and none of the other scanners really grab that in the same way you could grab it character by character looking at catcodes I suppose...

@DavidCarlisle I didn't say it was helpful, I said I have a feeling it's deliberate

@DavidCarlisle One for Hans, though: I can't be sure from the manual that I'm reading this right

@JosephWright yes I was assuming that's the answer I'd get if I asked on list, why i thought I'd ask here first:-)

@DavidCarlisle I think it's fair to say that the token library isn't meant to replace parsing stuff from the 'TeX side': not the way Hans works

@JosephWright summarize the manual as "stuff Hans needs is here. the end." :-)

@DavidCarlisle :)

@DavidCarlisle I see I was right ;)

@DavidCarlisle Weighing up whether to disagree with Hans: likely not worth it

@JosephWright I think "there is no token \UNDEFINED" is arguably wrong:-) (there is no command in the internal hash but \UNDEFINED is a token that can appear in token lists etc..

@DavidCarlisle That was the bit I was going to argue with, yes

@DavidCarlisle I guess Hans means something like 'There is no useful meaning of ...'

@DavidCarlisle Another interesting feature: try

\showtokens\expandafter{\directlua{tex.sprint("[" .. token.get_next().csname .. "]")}~}
\showtokens\expandafter{\directlua{tex.sprint("[" .. token.get_next().csname .. "]")}\~}

@DavidCarlisle That's of course predictable but still suggests one has to be cautious here

@JosephWright I replied on list...

@DavidCarlisle Hmm, your example suggests hic sunt leones in terms of undefined material! It might work, it might not, depending on the situation.

@egreg I wish. :)

@JosephWright if tex has scanned the undefined token it adds it to the hash, as that's what it means to be a token (from the inside) but if scan_token scans an undefined token it doesn't create a token in that sense, (given how often complained about \csname defining tokens as a side effect for similar reasons, hard to complain too much but it's pretty confusing behaviour

@DavidCarlisle Yes, I can see how this is happening

@DavidCarlisle But it would be rather clearer if the csname returned was simply \csstring of the grabbed token, ignoring the hash table entirely. (That is needed for the other properties, of course)

@JosephWright yes

@JosephWright vv

@DavidCarlisle: Microsoft released Paint 3D.

@PauloCereda WOW!

@PauloCereda Is it as useless as the 2D Paint? :)

@JosephWright :)

@wilx Think of aircrafts. :D

@wilx useless!!!???? what else could produce such fine artwork?

@DavidCarlisle :D

@JosephWright: this actually got my attention: github.com/microsoft/cntk

@DavidCarlisle Reading package manuals is often convenient. :P

@egreg it'll never catch on

@DavidCarlisle Was thinking about the tokenization business in the context of an expandable macro. I guess \firstofone is what is wanted here:

\firstofone{\directlua{texio.write_nl("[" .. token.scan_csname() .. "]")}\UNDEFINED}

@JosephWright yes firstofone or anything:-) in the stated uses case of a \detokenize variant it's probably not an issue as the things being scanned would probably already have been scanned by the argument ofthe detokenize macro

@egreg my answer is a generally applicable technique for finding clashing definitions, your version just works in this one special case

@DavidCarlisle Possibly: what you can't do is use it for verbatim-like material, or anything else where you don't want to grab first as a macro argument

Sing the bells, bells, bells, bells, bells, bells, bells, bells, bells of Notre Dame!

@JosephWright I think the current behaviour is sort of reasonable for token_get_next().csname as there is no token (although an error might have been expected rather than empty string) but for token.scan_csname() which returns a string it just seems pointless why doesn't it just return the string it just scanned?

@DavidCarlisle Yes, sounds reasonable (though there is still the business that it omits the escape char)

@DavidCarlisle Presumably they are the same thing internally

@JosephWright always been that way in tex:-) you can presumably distinguish them by the .command and/or .cmdname rather than .csname

\long\def\z#1{%
\directlua{
local t = token.get_next()
texio.write_nl("[" .. ((t.active and '') or '\string\\')..  t.csname .. "]")}#1%
}

\z\par
\z\UNDEFINED
\z~
\z\~

\bye

@JosephWright ^^

@DavidCarlisle Hadn't pursued all of the possible data :)

@barbarabeeton \binom.....

\documentclass{amsart}

\ifx\Ucharcat\undefined\else

\usepackage{unicode-math}
%\setmathfont{STIXMath_2.128e.otf}

\fi
\makeatletter
\def\genfracrule#1#2#3#4{%
\hbox{$\left#1\vcenter{\hrule \@width\z@
                       \@height \ifdim\fontdimen#2#3\tw@=\z@#4\fontdimen6#3\tw@\else\fontdimen#2#3\tw@\fi
                      }\right.$}}
\DeclareRobustCommand{\genfrac}[6]{{%
\@mathstyle{#4}%
\ifx @#1@\else
{\delimitershortfall\z@ \delimiterfactor\@m \mathsurround\z@ \nulldelimiterspace\z@
\mathchoice

I'm looking for some more help, regarding my question

I trimmed the code down to only focus on a column index and now want to condition on the column...

...here's the code:

\documentclass{article}

\usepackage{mathtools,collcell,eqparbox,xpatch}

% tex.stackexchange.com/a/65652/5764
\makeatletter
\let\old@tabarray\@tabarray
\def\@tabarray{\global\c@tabcol\z@\old@tabarray}

% tex.stackexchange.com/q/335981/5764
\xpatchcmd{\@xarraycr}
  {\cr}
  {\cr\noalign{\global\c@tabcol\z@}}{}{}
\xpatchcmd{\@xargarraycr}
  {\cr}
  {\cr\noalign{\global\c@tabcol\z@}}{}{}
\xpatchcmd{\@yargarraycr}
  {\cr}
  {\cr\noalign{\global\c@tabcol\z@}}{}{}

\newcounter{tabcol}

\newcounter{BMATRIX}

The attempt above is to condition on the column number so I only apply a macro for columns 2 and higher.

That's achieved by the counter value conditioning in \setmaxwd:

\newcommand{\setmaxwd}[1]{%
  \stepcounter{tabcol}
  \ifnum\value{tabcol}>1 % First column
    \eqmakebox[BM-\theBMATRIX][\BMalign]{$#1$}%
  \else
    #1
  \fi
}

However, it's obvious that \eqmakebox is doing something weird.

If I add the column index to the end of \setmaxwd, it doesn't work:

\newcommand{\setmaxwd}[1]{%
  \stepcounter{tabcol}
  \ifnum\value{tabcol}>1 % First column
    \eqmakebox[BM-\theBMATRIX][\BMalign]{$#1$}%
  \else
    #1
  \fi
  c\thetabcol
}

Replace the \eqmakebox construction with just #1 and everything works fine.

@egreg: ^^

Do people name fonts with weird names on purpose so that it's harder to use them on Xelatex?

@Alenanno yes

@Werner \BMalign does \global\c@tabcol=\z@

@Werner Sorry, misread

@egreg Okay, I couldn't find that...

@Werner I'm afraid that \eqmakebox calls @tabarray.

@egreg Ahhh, that makes sense that is causes an issue then.

@Werner Yes. I added \tracingmacros=1 in the code for \setmaxwd and got the surprise to see \@tabarray in the log.

@Werner Changing \@tabarray and \@xarraycr is not the way I'd do it anyway. Collect the body, do all measuring you need and then output the array.

@egreg I thought about doing it that way and switched after thinking that \eqmakebox is an easy-way-out for measuring boxes...

@egreg you would be proud of my last tikz-pgf answer

@DavidCarlisle e-TeX plan looks like it goes ahead

@JosephWright see you should have tested for \Uchar while you were on a roll

@DavidCarlisle See my starred comment on the right!

@DavidCarlisle For amsmath?

@DavidCarlisle Or for something else (above I see you test for \Ucharcat)

@JosephWright I meant top of latex.ltx :-)

@DavidCarlisle :)

@JosephWright the code above is for @barbarabeeton to do amsmath \genfrac in xetex (I think the xetex code is broken really, as far as i can see it only \abovewithdelims only works with delimiters specified by \delcode/\delimiter not \U... which means you can't do it with otf fonts where you don't have two slots

@barbarabeeton I'm sure you'll back my latest comment on the answer to tex.stackexchange.com/questions/332902/… (and yes, I know I'm writing about xparse)

@DavidCarlisle Tell Jonathan K?

@JosephWright need to cut down the example a bit first, lots of places it could be going wrong: need to get a non fontspec example:-)

@DavidCarlisle -- first quick test looks reasonable. will try harder, and report. thanks.

@barbarabeeton I think with cm fonts in pdftex it gives exactly the same results (that's where the magic numbers come from) in other fonts may differ a bit, but you can tweak fontparam 20 and 21 to get different effects

@JosephWright -- you'll notice that i did react to it. thanks for the suggestion.

@JosephWright Adding \Uchar or similar?

@barbarabeeton tex.stackexchange.com/questions/336070/…

@barbarabeeton ^^^ Can you have a look at this?

@egreg etex not Xetex, adding if not etex die \fi

@DavidCarlisle Yet it would be useful to have an expandable \char also with e-TeX

@egreg \alph (extended a bit)

@DavidCarlisle Boo

@egreg a 256 way \ifcase is more palatable than a 2^{21} way switch

@DavidCarlisle But it won't produce the character with the category code at the moment of the call.

@egreg another nested 16-ish way ifcase with the current catcode?

@egreg or ln -s luatex pdflatex and just don't tell anyone that pdftex has aquired a lua system?

@DavidCarlisle -- well, we've got a couple other fonts that might be possible to experiment with. and we've got a few test files with nasty examples. on my list to pursue tomorrow. thanks.

@egreg -- have looked and commented. unfortunately, the fact that a symbol is listed in one of the existing collections isn't quite enough for unicode. they want published evidence, preferably in a context which clearly indicates the meaning, either as a definition, or in a particularly obvious usage. the unicode committee has accepted the proposition that mathematicians will create new symbols, so acceptance of such symbols is expected .. as long as there is documentation. (cont'd)

(cont'd) what about a question (you can ask it!) about how to get a symbol into unicode. i can answer, with examples of documentation that has been accepted in the past. (or would that be off topic here?)

@barbarabeeton OT, I believe. It could be a page on the TUG site.

@egreg -- too bad. i don't know how many questions i've answered or left a comment on with that information. so i know quite a few people are interested. a page on the tug site, or an entry on the (to be moved and reconstituted? @StefanKottwitz ?) blog, seems like a reasonable idea. i'll work on that.

@egreg Question to dream about:

 What should a software return if a full 2x2 matrix and a 2x0 matrix is block diagonalized

 A 4x2 matrix with zero entries or a 2x2 matrix untouched?

This is what we are discussing on SciPy issue page about the behavior of block_diag command. I found the discussion interesting tough I have to admit I'm the latter proponent :)

@percusse Not sure I understand

You can define a matrix with no entries but a defined shape as in promoted here : 128.105.2.31/debooron/empty.pdf

But the question is what should be the algebraic rules when you use it together with regular matrices. A regular block_diag command is defined as direct sum of two matrices. But if ones of them is a 4x0 matrix and the other is 2x2 what should be the direct sum ? 6x2 or 2x2?

@percusse It's still 2 x 2

@egreg I also think that way I can't convince others about this so far :)

@percusse Let's see. an $m\times n$ matrix $A$ represents a map $F^n\to F^m$. If you consider a $p\times q$ matrix $B$, the block diagonal matrix with $A$ and $B$ at the corners represents a map $F^{n+q}\to F^{m+p}$, so in the case of $q=0$ we should get an $(m+p)\times n$ matrix.

@percusse The others are right, then.

But from the direct sum argument you have $S \oplus \emptyset = S$ so there should be no embedding

in terms of the span

There has to be a point where you have to invent or inject a choice about the behavior of empty matrices as is done with the article. But it didn't got picked up by the mainstream software