Anyone know how to tell an InputStream it's been read from -___- Turns out there's a subtle but reallllly annoying bug in reading from a stream when StreamPosition[s]==0:

test = "([ test ([";
stm = StringToStream[test];

Table[
 {
  Read[stm, Record, RecordSeparators -> {"(", "["}, NullRecords -> True],
  StreamPosition[stm]
  },
 3
 ]

{{"", 0}, {"", 1}, {" test ", 8}}

That first Read call did advance the stream, but it's saying it didn't.

But if you SetStreamPosition[stm, 0] you'll actually re-read

Which means that there is literally no way to get to that first "[" unless you already know you've read from the stream once.

But you can't go past it and then jump back to it because instead you'll advance to that first ( token.

And now Mathematica just crashed for no reason when I searched in the documentation for "stream" -_________________-

At least we have Language`ExpressionStore to allow us to work around WRI's bugs in a leak-free way.

Getting ready to release this parser stuff. Probably this weekend or so.

Here's a sample of what it can currently do:

I defined a language grammar, that grammar can generate a parser, the parser can parse into an AST, and then I have a very general function to walk the AST much like DepthFirstScan.

I used this to implement a function that trims extraneous whitespace and also one that converts the AST into a RowBox/StyleBox construct.

Lots of control over how that can be styled.

The current AST structure isn't super sophisticated, so I'm gonna write utilities on the walk function to merge nodes that should be together (like that printfort and that ( x ) which are currently two children of ;).

There's probably still a good amount of space to optimize, but even large code samples can be handled quite quickly. That parser is generated up front, so it can be embedded into a Stylesheet and used as a way to auto-style code in other languages. It would be possible to simply update an AST rather than build a new one if that can be detected.

But that's where it makes more sense just to use multiple cells, I think.

If you want to write your own parser framework, that parser object stores a lexer object you can call into as you will to break the source code into tokens in a pretty efficient fashion.

@Kuba remembered out how to tweak the appearance of Show Expression cells. You do it via the "CellExpression" style.

@MichaelE2 I think your logic is not too far off, Unevaluated just seems to be completely invisible to the pattern matcher:

Verbatim[Unevaluated][foo[x_]] ^:= bar[x]

Unevaluated[foo[x]]
(* Unevaluated[foo[x]] *)

Attributes[foobar] = {HoldAll};
foobar[Verbatim[Unevaluated][x_]] := bar[x]

foobar[Unevaluated[x]]
(* foobar[Unevaluated[x]] *)

Generated from an AST and formatted nicely in the FE (using every imaginable stylesheet tweak)

Notice how it seems impossible to match Unevaluated[...] using DownValues and UpValues. The same is not the case for the normal pattern matcher:

foobar[Unevaluated[x]] /. DownValues@foobar
(* bar[x] *)

@b3m2a1 Can you do the timing for say 100 lines of dense C code?

@b3m2a1 It seems we've created a race condition ;) But that does look very fancy

Wait, that's not C..

@halirutan I'm gonna do more profiling tomorrow or when I get the chance. It scales quite well with the number of tokens. There's almost no backtracking and I don't parse character by character.

On the other hand it's not optimized in the slightest, yet.

@halirutan Nope definitely isn't. It's some random code conforming to this grammar:

EBNFGrammar[
 {
  (* blocks *)
  "curly" :> Sequence["{", ___, "}"],
  "paren" :> Sequence["(", ___, ")"],
  "bracket" :> Sequence["[", ___, "]"],
  "comment" :> Sequence["(*", ___, "*)"],
  "stmt" :> Sequence[___, ";"],
  (* strings *)
  "str" :> Sequence["'", String, "'"],
  "comm" :> Sequence["#", String, "\n"],
  (* operators *)
  {"assign", "Precedence" -> 0} :> Sequence[_, "+=" | "=", _],
  {"comp", "Precedence" -> 1} :> Sequence[_, "<" | ">" | "<=" | ">=", _],
  {"add", "Precedence" -> 2} :> Sequence[_, "+" | "-", _],

@b3m2a1 Alright

But I've taken that chunk and nested it and copied and pasted, etc. like 50 times and it's still linear in the number of tokens that you need to parse.

Which means parsing a huge source code file will likely be slow, but working with smaller portions should be fast.

In the end, I should be able to leverage FE trick to not have to rebuild the entire AST on edits but instead just work with the part I want.

I'll also need to allow it to gracefully handle syntax errors, too.

Currently it just complains and aborts when it hits one.

@b3m2a1 The partial rebuilding of the AST for portions of the code and the fact that you need an error ignoring parser are two key ingredients.

Yup. For now I'm happy to have a basic AST builder though. Those will come as I integrate this more tightly into the FE.

@ChrisK, yes, i was trying to just use just the right equation and you need to multiply by the left two to cancel the derivatives of the eigenvectors. Thanks for your help (and answer in the first place)

I would be interested to see an answer to this question: mathematica.stackexchange.com/questions/197985/… From five minutes of poking around I can't see a way to do it, but I don't know much

@b3m2a1 thanks for that tip

@b3m2a1 Looks fantastic. I don't suppose you've ever played with Racket?

Perhaps a mod could help out this user in merging their account: mathematica.stackexchange.com/questions/197867/…

smbc-comics.com/comic/fundamental-2

@b3m2a1 smbc-comics.com/?id=2861

@CarlLange Never have but it looks interesting. I was thinking more of implementing something that looks vaguely like EBNF syntax as given by Wikipedia.

@CarlLange One of the PhD students in the physics department here has that one on his door :)