I’m banging my head against an expansion problem

how can I make it so that, in a macro definition, \othermacro{#1} the #1 is expanded before it is passed to othermacro, without using edef (since this goes into a csname later), and for plain tex/latex (not xetex/luatex)?

I can’t use expandafter because that instead “suspends the {”, whatever that means

\def\unicodedomino@last#1{%
 \expandafter\unicodedomino@l@st#1\empty\empty%
}%

\def\unicodedomino@l@st#1#2#3{%
 \ifx#2\empty%
[R:#1]
%  #1%
 \else%
[SKIP:#1:#2:#3]
  \unicodedomino@l@st#2#3\empty%
 \fi%
}%

Unfortunately, this is called as \unicodedomino@last{\string#2} from another macro, and its #2 is \lst@EC�, which results in [SKIP:\:l:s] [SKIP:l:s:] [R:s] t@EC� instead of the expected multiple skips until [R:�]

The basic idea here is to remove the \lst@EC if it’s there, but this seems to be difficult…

\expandafter\unicodedomino@l@st\expandafter{#1}\empty\empty% is even worse and prohibits splitting completely

oh, I got it… another \empty sentinel…

geilon…

this might even fix utf-8 with lstlistings in general

ofc not completely there yet…

wow, yes, it does

I’m amazed

well, mostly; character width is a concern ofc

should test with fullwidth chars too, but I don’t have them set up in latex

meh, the language option of lstlistings still fucks things up

oh no, it's just the languages reset the delimiters. outch.

in case someone wants to check this out: github.com/mirabilos/tex-unicodedomino/blob/master/…

(and yes, I plan to upload to CTAN eventually… I just never have the time…)

\end{chat}% goodnight

@mirabilos \ifx#2\empty% is probably not what you want to do.

@mirabilos don't use \makeatletter/\makeatother in a .sty file (that used to kill latex 2.09 completely, in 2e we do guard against that so it's harmless but doesn't do anything useful)

@DavidCarlisle Interestingly, listings and UTF-8 came up in the 'TUG2018 team postmortem' [i.e. a walk to Museu de Arte Contemporanea Niteroi :)]

@JosephWright I gave some code to the minted maintainer once, that did something partially sensible with utf8, probably listings ought to be similar.

@DavidCarlisle One of the topics we discussed was the lack of a maintainer for listings ...

Wow, l3build update from last night already in TL

@JosephWright oh I hadn't realised that, has Jobst Hooffmann dropped out?

@DavidCarlisle No updates since 2015 ...

@DavidCarlisle Perhaps one for the team list

@JosephWright well i maintain lots of packages that have not had updates in that timeframe, so I don't take that on its own as sign of lack of maintainer (other people might though:-)

@DavidCarlisle Sure, but the long-standing need to sort UTF-8 support in some meaningful way is less good sign

@JosephWright we should probably provide the necessary hooks in the kernel....

@DavidCarlisle You sound like FMi ;)

@JosephWright eek I'll have to fix that

@JosephWright I saw him last year at the Dante meeting and there he had plans for listings but was in difficult private situation (illness). @DavidCarlisle

@UlrikeFischer Ah, right

@UlrikeFischer Might drop you, @DavidCarlisle and Frank a mail

Morning @egreg

@JosephWright Good morning!

@egreg Awaiting your answer to tex.stackexchange.com/questions/444653/…

@JosephWright Writing it. ;-)

@egreg how long does it take to write don't do that ?

Drat, one issue still in l3build

@JosephWright as I just get the second time the error ".exlive/2018/texmf-dist/scripts/l3build/l3build-check.lua:595: attempt to concatenate global 'tlpext' (a nil value)" (because I have a lvt in "developement" the testfiles folder which hasn't a tlg yet) is there an option "ignore test files without reference files"?

@JosephWright I could naturally use another extension for the file until it is ready.

@UlrikeFischer I'll look at that: we used to work out tests by available .tlg files in days of yore (the .lve business makes that more tricky)

@UlrikeFischer I have one issue to fix for CTAN releases of bundles: missed something in my reworking

@UlrikeFischer Should be done later today

@UlrikeFischer I've fixed the out-and-out code error: I'll add a 'skip this test' switch

@egreg sorry, I could have warned you about that OP... chat.stackexchange.com/transcript/message/35371022#35371022

-_-

it appears the OP doesn't have a home page

or consistent username

through chat

@marmot @UlrikeFischer Ducks are already in use as integral replacement: tex.stackexchange.com/questions/317265/… :)

@crypto If you wait until he appears online again, you'll have to wait a very long time :) If you need ideas for profile pictures, have a look at tex.stackexchange.com/questions/387047/…

@DavidCarlisle Uh, the mind boggles. How appropriate, then, that the chat software clips the username to “AbstractDiss”.

@CarLaTeX ad TUG: You know, it's difficult. The point is that in the past (think 20 years ago), user groups were basically the only way how to get to information. Nowadays, you download the whole TeXLive from the internet in couple minutes, you get all support you need on the internet, etc. So what happens is that only very few people gather in the user groups, and "user groups" become more of "developer groups" anyway. So if this is what you mean by "cult", then you are completely right.

@yo' Totally agree

@JosephWright well, I'm just basically repeating what was said at the AGM :-)

@yo' Coupled with that, although TeX is a programming language, most users don't actually do (or want to do) any programming

@yo' Yup: sounds familiar ;)

@yo' DANTE of course is an exception

@JosephWright tug could become a Word reseller and make some money selling software that works the way people expect.

@JosephWright that's an important point to make. I don't know how they manage though.

@DavidCarlisle That's not a million miles away from something said at TUG2018 [@yo' :)]

@yo' They reached a critical mass long ago enough to have self-sustaining 'Stammtisch' events

@JosephWright I have to confess I don't remember that one.

@yo' The point about Sage, etc., which started 'free' and now are chargable

@JosephWright ah yes, this one :-) Well, Sage is just on the way, but Wolfram made it though a long time ago :D

@yo' My notes say that was contributed by 'Tom' ;)

@yo' (Other examples given. ED: What were their older license/copyright situations.)

@JosephWright indeed it was. I thought you're making another point, which was made by Kaveh at TUG15 in Darmstadt, it was something like: Why isn't LyX or a similar product the default "WordPad" in linux distros?

@JosephWright sage (sagemath) is still GPL isn't it?

@DavidCarlisle Don't know: never used it

@JosephWright wikipedia says it is:-)

@DavidCarlisle FWIK yes, but AFAIK there is already a SageMath, Inc. behind, or something like that

@yo' yes there is a company behind it (there are lots of bigish companies behind linux distribs as well:-)

@DavidCarlisle Sure, but the suggestion at TUG2018 was paying for access, which isn't true in any case where the software is GPL. I thought of QtiPlot, where binaries for Windows are charged for: most users can't compile their own, so it works ...

@DavidCarlisle William Stein, June 11, 2016, wstein.org/talks/2016-06-sage-bp/bp.pdf :

@yo' yes but the software is still GPL (with source available on github) presumably the company charges for consulting or extra support or whatever.

@DavidCarlisle So like Linux then

@DavidCarlisle I'm pretty sure this is a-changing.

@yo' Depends on the copyright situation ...

@yo' it is (by design) pretty hard to change GPL software not to be GPL (if you have ever accepted any contribution from anyone)

@DavidCarlisle Unless you have copyright transfer as a requirement

@DavidCarlisle but it's pretty easy to rename Sage to "Magmage" and start developing as paid. There are ways, and anyone believing there are not is stupid not enough experienced.

@DavidCarlisle That's true for any license, surely: you need permission of copyright owners

Or, well, you end as Debain where "open license" (whatever that means) is a religion for which people are willing to suffer. Or are not, which means that texlive may easily be out...

@yo' Ah, the famous 1600 emails

@JosephWright well no, mozilla licence (said and was not popular for saying it) that any contributions could always be used by mozilla corp for any use they wished (more or less)

@DavidCarlisle well, in that case the permission was simply given by submitting the code :-)

@DavidCarlisle Was it ever tested in court? I wonder if that would stand up, at least for contributions without signed paperwork

@JosephWright don't know:-)

@yo' The sagemath website still lists its "mission" as generating free open source software.

@DavidCarlisle I have to confess I didn't follow the Wolfram journey altogether, but from what I was told it looks very familiar to this. Also, even you include "still" in your sentence, just in case :-)

@yo' the mathematica code was never GPL surely

the Internet will eventually choke the legal system with a back log of copy right cases

@DavidCarlisle no, probably not.

@user2236 that's what happens now with patents in the US, and it's basically the same thing.

@yo' not so easy to get a company funded on that basis, you may be able to hide gpl code from the outside word but you would find it hard to get any investment in a company if its only product was built on that basis. Licence restrictions have an effect even if things are not dragged before the courts.

@DavidCarlisle I don't say it's easy. My point on TUG AGM was basically about showing that it's a trend for "free" software to become "non-free" because it's not sustainable.

@yo' but I was asking for an example of that, and I don't think sagemath is one.

@yo' I thought it was a little more nuanced: the 'big' example is Linux, and that is legal free but well-supported by commercial entities. But there we have lots of customers willing to pay ...

@DavidCarlisle well, then Wolfram is, AFAIK

@yo' not really, it was sort of loosely based on the earlier university smp project but not really the same (and GPL was explicitly written to make such commercial forks harder, so it is less likely that even that happens in future)

@JosephWright which was always RMS's vision of how software should be financed, give the code away and charge for support.

@DavidCarlisle yeah, I dunno. Maybe I don't have a point, in the end...

@yo' I think the current trend is for opensource projects to be backed by big companies, Microsoft in particular has been releasing a lot of free software (and a lot of that GPL) in recent years. Something that would not have been imagined a few years ago. What harms TUG is not that tex is free, it's that stackexchange (and before that c.t.t.) is free.

@DavidCarlisle Or rather that there are not (many) companies who want contract support at a level that can employ people

@JosephWright something that comes up from time to time at the day job as well:-)

@DavidCarlisle I'm sure!

@JosephWright well, if I ever get freelance, I will certainly offer TeX support and publishing services as well.

@JosephWright It seems I can remove offensive comments (from the web interface).

@yo' Oh sure, but that's a pretty small market I think

@yo' I tried that for 9 months, I hated it:-) (apparently you have to ask for money and things, took all the fun out of answering tex questions)

@egreg Quite possibly

@DavidCarlisle but did you hate freelancing or TeX-related freelancing? :-)

@JosephWright indeed. It would very likely be a small part of the portfolio.

@yo' both (mostly the former I suppose)

@DavidCarlisle ok, I thought so actually :-)

@DavidCarlisle thanks, what do you suggest instead? (I also think we need 2e already due to use of detokenize, but my Teχ skills are very… beginner)

@DavidCarlisle hm well, it works… but I’m always open to bugfixes from people like you ☺

though I think it’s \ifx\empty#2\empty now

@mirabilos it depends what you want to test \ifx#2\empty means if the first two tokens in #2 are equal, execute the remaining tokens in #2 followed by \empty, if #2 has just one token, compare it to \empty, if #2 is empty compare \empty with the following (unshown) token in the code. That is well defined and syntactically valid, but (I guess?) was not the intended test.

@mirabilos \ifx\empty#2\empty now is slightly more reasonable, but treats #2 being \empty and #2 being empty the same, and still does something weird if #2 has more than one token.

@mirabilos \detokenize is an e-tex primitive so it is available or not depending which tex engine you use (all current one in practice have it) it does not depend on the version of the latex format.

@DavidCarlisle hmm good question ;_) I want it to work, for this specific use case

it needs Debian stretch (TL2016) now anyway, I used to have this compatibility code in it to make it work on Debian jessie (TL2014), but that’s old enough to drop, now (plus that code was copied from a newer utf8.sty so it’d be problematic, licence-wise)

in this line #2 is either one character or \empty so this should work

(I believe, anyway…)

@mirabilos e-TeX was finalised in 1999 ...

☺

@mirabilos "work" means do what you expect, as I say it works as it is, but it does what it is defined to do, only you can say if that is the same thing,

as I said, I’m somewhat of a novice

@mirabilos ???? utf8.enc has been in latex for a lot longer than that.

@DavidCarlisle well, depends on what part of the code, I have this construct multiple times; in unicodedomino@last it’s just supposed to return the last character

@DavidCarlisle yes but \decode@UTFviii is new

and I use that in my cod

e

@mirabilos yes sorry I just realised that's what you meant, yes I wrote that for an answer here originally, and added it to core latex around that time.

@mirabilos in that case the \empty after #2 isn't doing anything useful.

@JosephWright I think you know what kind of comment I'm referring to. :-(

@egreg Yes

@DavidCarlisle yes, in that line I had not added it

I worked on this for about 10 hours with only an hour or so of food break, till 5 in the morning, so my memory of “yesterday” is a bit full

we talked about this project of mine in here a year or two ago actually; it renders unicode chars that are not defined as

@mirabilos hmpf that describes a typical tex coding session for me too. Then wait and see how you feel 25 years later and someone asks how it works...

+-----+           +-------+
| 1 2 |      or   | 1 2 3 |
| 3 4 |           | 4 5 6 |
+-----+           +-------+

@DavidCarlisle I completely understand (although I must admit I’m currently not doing anything I did 25 years ago wrt. coding), so hats off

well, this broke in lstlistings, and by fixing that I seem to accidentally have made available UTF-8 for lstlistings without the need for literate (needs to be tested of course)

(and the “broke in lstlistings” was a side effect of me not knowing that loading a language resets the moredelims, and I had used fancy chars like 「 and 」 for moredelims)

fun thing, neither ‘’ nor “” work as delims…

@mirabilos don't the delimiters need to be a single character token?

@DavidCarlisle I meant something like moredelim={[is][\color{blue}]{‘}{’}}, i.e. one left, one right

@mirabilos OK, it's been a while since I looked at the listings code

yeah, and moredelim={[is][\color{red}]{【}{】}} too, in this file

perhaps ’ is already expanded on the Teχ side before…

whereas I did not newunicodechar 【

nevermind, I got it to work well enough

hmm \ifnum#1#2=0 doesn’t work, nor \ifnum1#1#2=100…

@mirabilos note you don't need to use newunicodechar (that's an @egreg extension:-) the standard interface would be \DeclareUnicodeCharacter

@mirabilos they will work if #1 and #2 are both sequences of digits, and give an error otherwise

@DavidCarlisle ah ok ;) but the other package of mine (unicodepoints.sty) defines a \renewunicodechar (that skips the warning if it was already defined), so it works out

hmm both are supposed to be numbers between 0 and 15, inclusive

but I can add debugging, I guess

oh, no, I see, numbers between 0 and F… oops ;)

@mirabilos declareUnicodeCharacter doesn't warn either

well let’s make it #ifnum"#1#2=0 then

ah

hm, perhaps I wanted the warning in the normal case… dunno

I probably shouldn’t assign to \temp here, should I?

@mirabilos I don't know where "here" is but \uniqueprefixformypackag@temp sounds safer

@DavidCarlisle yeah, thought so, thanks

nice, almost all of literate is now no longer needed, U+2011 ‑, U+2013 –, and the ellipsis … are still necessary though

and without literate, they even show up in the \lst@BeginAlsoWriteFile output

@mirabilos odd, what's special about dashes?

I think I have defined them to something special, let me look

\renewunicodechar{‑}{\unicodepoints@nbdash}%

hm, or it might be that the font does not have them

(using inconsolatazi4)

but not needing literate for ä ö ü etc. will already make my listings much more useful ;)

weird… out of all of them (2010-2015), the U+2010 one is the only one that works, but it shifts the position of the text, so I guess I better literate them

it’s really too bad that literate makes the chars not show up in the listing file

but I kinda have a tutorial paper to prepare and invested way too much time into this already…

hmm, ™ also shifts slightly

oh ouch, no, they all shift the text slightly to the right

damn, so, not a proper fix

@mirabilos Stray space in your code?

no, it’s more of a rendering issue… if I write “mäh” at the beginning of the line, the entire word is moved slightly to the right; at the same time, it somehow appears to use less space, perhaps the raw UTF-8 char is not counted width-wise

I’d (or someone else…) have to dig into listings interna for that

(and making literate chars show up in files)

but it’s good enough, for now I’ll continue using literate, so I have no regression

I just thought I had accidentally fixed it ;)

oooooh, off-by-one in utf8.def, octet 0xF4 is a valid lead octet

@JosephWright Please, are you able to resurrect an auto-deleted comment? I made one here and it got deleted when the question was duped... tex.stackexchange.com/questions/444718/…

@yo' No, can only undelete comments that I remove, and then only for a short time

@yo' I'm intrigued by tex.stackexchange.com/questions/444718/…: decisions about length are for the editorial office, in my experience: the typesetter just does 'what they are asked' by the editorial office

I’ve pushed a newer version, so the links above linking directly to lines are no longer valid, FWIW

@mirabilos regarding the comment "% patch up bug in utf8.def" and the following code in your style: if you find a bug in the latex kernel you should make a bug report instead of redefining kernel commands.

@UlrikeFischer yes, of course… but later…

I need the fixup anyway because I have to support Debian stable with my module

(and it needs another patch, the UTFviii@checkseq is too lenient)

this needs revisiting anyway… I currently have at least three different ways of converting a number to hex…

@mirabilos it's very odd to refer to the version of an application by the version of the operating system on which you happen to run it. You can run a current texlive 2018 on an old debian.

@DavidCarlisle I was thinking much the same

@DavidCarlisle Or indeed just LaTeX itself

@DavidCarlisle Or latexrelease.sty

@JosephWright details details:-)

@JosephWright well, if the journal had a "IEEE style" strict restrictions on the length, I would simply inform the communicating editor that the article actually exceeds the length and stop working on it until they tell me what to do.

@DavidCarlisle you “can run”, in theory, yes, but Debian users normally only run the version as shipped by Debian

meh. since I can’t concentrate on the main task, I may as well report the bug and my patches upstream

/me looks up whether there are any contribution guidelines

and licence foo

@mirabilos upstream for inputenc? github issue is best or just say what it is here:-)

@mirabilos checkseq is only used for printing error messages so I just made it check enough that you didn't get any arithmetic errors while decoding, it doesn't do a full check, it's true.

@DavidCarlisle hm yes, but to ensure all chars outside the valid range and overlong encodings are caught it needs improvement, I’m doing that now

thought mailing list… I was preparing a patch for utf8ienc.dtx from svn at the moment

@mirabilos the master is on github not svn

@mirabilos You are welcome to mail the team list, of course

@mirabilos here github.com/latex3/latex2e/tree/master/base

@DavidCarlisle oh okay, that doesn’t match the docs on the texlive site then

thanks, will propose a patch

@mirabilos texlive is in svn but isn't the master source for latex, they get it from CTAN after we make a release from git to the CTAN network

I just realised I can get by with less macros by using more arguments, which seems to be a good thing

ah okay, thanks then

@mirabilos to be honest it's more important to report the issue than suggest a fix. there are other considerations when changing internal macros, such as whether any packages are depending on the current definition (we usually end up grepping the whole texlive tree for usage)

@mirabilos github.com/latex3/latex2e/blob/master/CONTRIBUTING.md

@DavidCarlisle yet, I have a fix (and other useful patches)

grepping everything is good, though

@mirabilos Entirely necessary, and still misses stuff :)

I’d be slightly changing \UTFviii@checkseq and removing \UTFviii@check@continue

(replaced by three separate continue functions)

@mirabilos happy to see the diff but we almost never accept PR on the 2e sources (actually I am not sure if there has been a PR:-) but code suggestions are always welcome, as it says in the contributing guide though a test file in an issue is the most important thing. (then you can make a PR referencing the issue)

a test is tricky

I just ran a series of them, but the test succeeds if the pdflatex build fails…

well for some of them

@mirabilos the point is to clarify to the team the issue being solved, not to write an automated test

@DavidCarlisle ok, sure then

@mirabilos As @DavidCarlisle says, the team tend to work on the basis we write the patches: we just need clear issues to fix :)

@mirabilos Or rather, @DavidCarlisle and Frank Mittelbach fix them: I just do expl3 and associated tasks (and take the minutes, and ...

@DavidCarlisle Oh, BTW, did I say that somehow I ended up as 'secretary' for the TUG AGM (with help from @yo')

@JosephWright true you never touched the ltluatex support in 2e

@DavidCarlisle :)

@DavidCarlisle Well with Frank's new view on 2e changes vs a new format, things are a bit different

@JosephWright yes we can dump it all on you

@DavidCarlisle github.com/latex3/latex2e/commit/… :)

@samcarter I just realized you have moved. Please tell, do you like it in "Universe"? ;-)

@marmot I like it very much - it is the best Universe I have ever been to!

@samcarter In the whole Multiverse? ;-)

@JosephWright @UlrikeFischer @WillRobertson @PauloCereda I'm probably the last non-Brazillian at TUG2018 to leave Brazil tomorrow. It was a fun time :-)

@yo' Cool :)

@yo' It was a lot of fun

@yo' Did you stay for ICM? :)

@JosephWright how do I actually show a " in a message, and pass it around so it does not get interpreted?

hm, just writing " seems to be enough…

@mirabilos Exactly :)

@mirabilos It's only special in numbers, though you might want a defensive \string to deal with the \active case

hm, all occurrences are in the PackageErrors, so…

@JosephWright “The kernel requires e-TeX nowadays [and was meant to for many years :)].” does this mean I should remove the non-numexpr cases? Or should that rather be done separately (before or after these PRs) for consistency?

@mirabilos I mean you can assume \numexpr exists: if it doesn't, the hyphenation patterns will fail anyway :(

oh ouch ;-)

oh, and address the travis failures (these never occurred for me, but it’s good to have tests ;)

@mirabilos Yup :)

@mirabilos More-or-less the first thing the team did when they took over LaTeX (~1992) was set up tests (predates me of course ...)

nice

aaah, I forgot a macro when splitting up patches

@mirabilos You know that one patch would have probably been OK :)

damn, this is needed for both PR61 and PR62

ok, I’ll merge them

@mirabilos thanks. some of the cosmetic ones I knew about but was probably worried about space or time (neither of which is so much of a concern these days) But the F4 just seems a flat out bug, Too tired now but I'll have to go back and see if that was always wrong or if it got broke while the code was re-arranged, I fear it was always wrong, I'll need to find someone to blame:-)

@DavidCarlisle sure, no problem

@DavidCarlisle github.com/latex3/latex2e/commit/…

I just do a lot with UTF-8, and therefore I happen to find such things

I have one more to go after #60 is in

@JosephWright that helps to show the error was not in my original code, and hasn't been in latex long (which is good) but it doesn't help with the issue of finding someone who isn't me to blame (@mirabilos)

@DavidCarlisle :)

@mirabilos I'll make a quick reply on list so the rest of the team know I have seen it but I may not be able to get to this for a couple of weeks (one of the other team members may handle it though, @JosephWright)

@DavidCarlisle Clearly outside of my area ... one for Frank

@DavidCarlisle I have Bruno's stuff to worry about, as he's busy (we need to do the peek stuff)

@JosephWright I can do it, but it may have to be a couple of weeks unless I can get it done in next day or so.

@DavidCarlisle that’s just fine, thank you for considering it at all

@DavidCarlisle Should be OK: I don't think this needs an urgent fix

@JosephWright well true not many people are using plane 16 characters in latex, but it's wrong and got my name attached so I'd like to fix it:-)

sure; as I said I’ll need to carry it locally for a while (about two years) anyway, but I’ve already separated my module into stuff like this and the “core” functionality of making those domino pieces for unknown codepoints

@DavidCarlisle you also introduced a “teh” misspelling of “the” in that commit ;-)

@mirabilos that I didn't even need to check the git log to blame myself, I have been typing the as teh consistently for the last 50-something years (no idea why, it just comes out that way:-)

@DavidCarlisle our boss at a charity also does that, and I’m his “^T bot” (Ctrl-T being the key combination in Emacs command line editing mode on the shell that swaps two letters)

@samcarter The post by Mark Wibrow is really nice but they are too cute for my taste. Was looking for something funky, like @Jake's owl or just the brows would make a nice profile pic mysoti.com/img/user/timmy/product/web/1217218/…