VBA Rubberducking

Mat's Mug

04:00

when you push, the commits get to the remote repository

IvenBach

So I can save (Ctrl+S) but that still doesn't mean they are committed?

Mat's Mug

exactly

IvenBach

Ok. Understood.

Thanks for that.

Mat's Mug

ideally you make small, targeted commits

IvenBach

VBALexer.g4 open

Mat's Mug

04:01

every commit requires a comment/description

IvenBach

and it looks all foreign to me

Mat's Mug

it's ANTLR's language

this is the lexer grammar

IvenBach

This may take me a while but I'll try.

Mat's Mug

the difference between a lexer and a parser it how they work with tokens

the lexer creates them out of fragments; the parser uses them in parser rules

the syntax is pretty simple:

IvenBach

what is a token?

Mat's Mug

04:04

we have a string that contains the code of a given module, we got that from the VBIDE API

if the best we can do is searches in that string, we're doomed ;-)

IvenBach

quora.com/What-is-the-difference-between-a-lexer-and-a-parser

Mat's Mug

so what we do instead is, we tokenize it

IvenBach

Is that a good explanation?

Comintern

:thumbsup:

IvenBach

Basically: 'Parsers work at the grammatical level, lexers work at the word level.'

Mat's Mug

04:06

yeah. the lexer grammar defines a set of lexer rules

ThunderFrame

IIUC, "End" is a keyword, but depending on context, it can be an "End" statement token, or an "End" Property token, or an "End" If token etc.

IvenBach

Ok.

Mat's Mug

RULENAME : {match pattern}

actually, scroll all the way to the bottom

the smallest parse unit in the grammar is a fragment

IvenBach

:boom: Oh man.... It just hit the fan and got real messy in here!

Mat's Mug

the lexer rules use these fragments

 fragment A:('a'|'A');

fragment JANUARY : J A N U A R Y;

there's a rule for everything you could possibly see in a VBA program

NEWLINE : '\r' '\n' | [\r\n\u2028\u2029];
SINGLEQUOTE : '\'';
UNDERSCORE : '_';
WS : [ \t];

well, one we can parse anyway

a WS lexer token is either a space or a tab

IvenBach

04:10

\r \n are the unicode characters?

WS = WhiteSpace

Mat's Mug

yeah

IvenBach

'\' denotes the escape character?

Mat's Mug

as in regex

@IvenBach exactly

IvenBach

so WS : [<space>\t];

Mat's Mug

as far as Rubberduck gives a duck, yeah

IvenBach

04:12

In laymans terms. WS is taken to mean either a <space> or <tab character>

Mat's Mug

yep

then you scroll up and you get the whole list of VB keywords and operators

IvenBach

NEWLINE is taken to mean <return> <newline> or either [<return>,<newline>,<u2028>,<u2029>]

Mat's Mug

yep

IvenBach

Think I'm understanding

Mat's Mug

lexer rules are always SCREAM_CASE

IvenBach

04:14

So anything thats a fragment can be to be something that can later be built upon?

Mat's Mug

fragments are always all_lowercase

exactly

as are lexer tokens themselves

EXIT_DO : E X I T (WS | LINE_CONTINUATION)+ D O;
EXIT_FOR : E X I T (WS | LINE_CONTINUATION)+ F O R;
EXIT_FUNCTION : E X I T (WS | LINE_CONTINUATION)+ F U N C T I O N;
EXIT_PROPERTY : E X I T (WS | LINE_CONTINUATION)+ P R O P E R T Y;
EXIT_SUB : E X I T (WS | LINE_CONTINUATION)+ S U B;

I never thought one could put a line continuation there

IvenBach

EXIT_SUB : E X I T (WS | LINE_CONTINUATION)+ S U B; the <+> symbol at the end denotes 1 or more?

Mat's Mug

exactly

* denotes any number

IvenBach

HAHAHA. Who would put a LINE_CONTINUATION character between Exit and Sub

Mat's Mug

IKR!

IvenBach

04:16

Now I understand you you @Comintern @ThunderFrame were talking about.

Mat's Mug

// Apparently END_ENUM and END_TYPE don't allow line continuations (in the VB editor)
END_ENUM : E N D WS+ E N U M;

IvenBach

goodreads.com/quotes/…

Comintern

Sub Foo()
    Exit _
 _
 _
 _
 _
 _
    Sub
    Debug.Print "Does not print."
End Sub

IvenBach

Or for that matter a single individual, myself included.

Mat's Mug

so far so good?

IvenBach

04:18

I think so.

Mat's Mug

that's all there is to the lexer really

bring up VBAParser.g4

IvenBach

For END_ENUM : <Stuff in here>

does everything have to be separated by a space? Always?

Mat's Mug

yes - E N D is the E fragment, the N fragment, the D fragment

Comintern

The 'E N D ...` are fragments.

They're specified that way so they can be case-insensitive.

Mat's Mug

^

Comintern

04:19

fragment E:('e'|'E');

IvenBach

and the fragments (e|E) (n|N) (d|D) constitute END

Mat's Mug

exactly

IvenBach

which itself is a fragment that 'could' be further built upon.

See you are explaining this fine.
It's the monkey you're teaching that has to take it slowly.

Mat's Mug

propertyGetStmt :
    (visibility whiteSpace)? (STATIC whiteSpace)? PROPERTY_GET whiteSpace functionName (whiteSpace? argList)? (whiteSpace asTypeClause)? endOfStatement
    block
    END_PROPERTY
;

that's a parser rule

parser rules are camelCase (aka javaCase)

IvenBach

It's also <thing> : <rules> syntax?

Mat's Mug

04:21

yep

IvenBach

Ok

Mat's Mug

this is how Rubberduck sees a module:

module :
    endOfStatement
    moduleAttributes
    moduleHeader?
    moduleAttributes
    moduleConfig?
    moduleAttributes
    moduleDeclarations
    moduleAttributes
    moduleBody
    moduleAttributes
    // A module can consist of WS as well as line continuations only.
    whiteSpace?
;

actually, this:

startRule : module EOF;

FreeMan

@Mat'sMug I was catching up until right about here. I've got 5484 outgoing commits showing in VS Team Explorer. It won't let me pull, the only thing I can do is push to Origin|Rubberduck-VBA|Upstream

Mat's Mug

@FreeMan you're F'd

Comintern

^

Did you clone main?

FreeMan

04:23

I just did the merge from upstream yesterday/Sunday to pick up all the .12 changes...

IvenBach

Give me a bit for the RD seeing a module part

Mat's Mug

sure

FreeMan

I made the couple of changes for 2803 in NonReturningFunctionInspection and wanted to keep those, so I did a commit on them, and that's what it's telling me now.

:(

Mat's Mug

github won't let you push to rubberduck-vba/Rubberduck/next

Comintern

@FreeMan - It looks like you forked master instead of next.

> This branch is 5517 commits behind rubberduck-vba:master.

Mat's Mug

04:26

huh, you fork a repo, not a branch?

FreeMan

crap. Yup...

just saw that...

Comintern

Go to your git.

Change the branch dropdown to next.

Mat's Mug

grabs popcorn

FreeMan

GitHub, WinGit or TeamExp

Comintern

github.com/daFreeMan/Rubberduck

FreeMan

04:27

create a PR for this?

Comintern

No.

IvenBach

@Mat'sMug This is going to take me a little longer to digest. Working on it.

Comintern

When you switch to next, it should show a commit or 2 ahead of next

FreeMan

This branch is 2 commits ahead of rubberduck-vba:next.

Comintern

Perfect.

FreeMan

04:28

yup, I see that.

Comintern

Delete your entire local directory

2

Then hit the green [Clone or download] button on github and copy the URL.

FreeMan

including the .git and other . directories?

Comintern

Yup. Including the directory itself.

FreeMan

closes VS first

Comintern

^ And any open VBE hosts.

FreeMan

04:30

^none at the moment.

deleted & URL in clipboard

Comintern

OK, open VS as admin.

IvenBach

<Token>? means 0 or 1 occurences?

Comintern

When it loads, go to team explorer.

FreeMan

then Add?

Comintern

Clone

Then paste the url.

FreeMan

04:31

that makes more sense.

Cloning

Mat's Mug

@IvenBach isn't Antlr awesome? You can use it to define a formal grammar for any language you want.. even make a new one just for your application

Comintern

Make absolutely sure you clone to the same directory path you had before.

FreeMan

Yup

and... done

Comintern

If you don't, it screws up your registry.

OK, now open the solution.

FreeMan

double click & open

IvenBach

04:32

@Mat'sMug I'm unable to determine if there is awesome in what I'm looking at. Give me a few more minutes to get back to you on that.

Comintern

Right-click on Rubberduck then properties.

Mat's Mug

you're looking at the parser grammar?

FreeMan

presume I don't need to install the 3rd-party Git command prompt tools, since I've already got wingit installed?

Comintern

The only thing that was blown away was your startup project.

IvenBach

@Mat'sMug Right now and traipsing through each one to get an idea.

Comintern

04:33

@FreeMan Those follow VS, not the solution.

IvenBach

I just can't find EOF anywhere. I know it stands for End Of File

FreeMan

this is the first time I've opened TE and it keeps prompting me to install them. I don't need another set of command line tools, I shouldn't think

Rubberduck, right-click, properties

Comintern

Nah, don't worry about it. If you have wingit installed, you can do that anytime.

OK, in properties, go to Debug.

FreeMan

start external program -> Excel.exe?

Mat's Mug

@IvenBach correct. ANTLR see that and knows it can return from the startRule

Comintern

04:35

@FreeMan Bingo.

Hit save after that, and then F6 to build.

You should be good to go.

IvenBach

I've got a general loose idea of what it's doing. Not as good as the lexer since that was much simpler but I think I'm ok to proceed. Bear with me if I don't get it all.

Mat's Mug

the startRule is the single parser rule we need to call to process an entire code file

startRule : module EOF;

FreeMan

huh... Mine says Ctrl-Shift-B to build

IvenBach

Ok

FreeMan

========== Build: 2 succeeded, 0 failed, 0 up-to-date, 1 skipped ==========

Mat's Mug

04:37

each parser rule, once processed by Antlr, becomes a ParserRuleContext class

and within a parser rule, every time another parser rule is referenced, it becomes a method of that class

FreeMan

thanks again, @Comintern.

Comintern

NP, I could do that in my sleep at this point.

Mat's Mug

so StartRuleContext.module() returns a ModuleContext

module :
    endOfStatement
    moduleAttributes
    moduleHeader?
    moduleAttributes
    moduleConfig?
    moduleAttributes
    moduleDeclarations
    moduleAttributes
    moduleBody
    moduleAttributes
    // A module can consist of WS as well as line continuations only.
    whiteSpace?
;

so the module parser rule becomes a ModuleContext object

that ModuleContext object has a method for each of these referenced rules

let's go do moduleBody

moduleBody :
    whiteSpace?
    (moduleBodyElement endOfStatement)*;

moduleBodyElement :
    functionStmt
    | propertyGetStmt
    | propertySetStmt
    | propertyLetStmt
    | subStmt
;

a module's body is made of whitespace and as many moduleBodyElement+endOfStatement as we can take

IvenBach

the ParserRuleContext used by Rubberduck.Common.CodeModuleExtensions.GetStmtContext which it instantiates?

Mat's Mug

yep

Comintern

04:44

@IvenBach I wouldn't worry about the code at this point.

IvenBach

@Comintern Don't hide the magic from me!

Mat's Mug

lol

we'll get there :)

IvenBach

@Mat'sMug Getting this understood right now.

Comintern

The parser generates the objects, and ANTLR generates the code. If you thought our code was sausage, you should peek in VBParser.cs sometime...

Mat's Mug

^

not too long, it's a HeadacheFactory

@Comintern he meant the extension method though

(there's a GetStmtContext method generated by Antlr in the VBAParser.cs class that Antlr creates from that grammar)

FreeMan

04:47

gonna stop while I'm ahead. Will take another crack at #2803 tomorrow.

IvenBach

@Mat'sMug I've somewhat understood it. Enough to take another step.

Comintern

Yep. The GetStmtContext method is using objects generated by the parser.

Mat's Mug

@IvenBach ok. Ctrl+F and search for functionStmt :

Comintern

@FreeMan That one might be a bit more in depth than I thought at first glance - it's something deep in the bowels of the resolver.

Mat's Mug

functionStmt :
    (visibility whiteSpace)? (STATIC whiteSpace)? FUNCTION whiteSpace? functionName (whiteSpace? argList)? (whiteSpace? asTypeClause)? endOfStatement
    block
    END_FUNCTION
;
functionName : identifier;

now we're deep enough in the tree that we're starting to see lexer tokens (they're ALL_CAPS)

the rest are other [lower-level] parser rules

IvenBach

04:49

(visibility whiteSpace)? means what?

Mat's Mug

Ctrl+F whiteSpace :

whiteSpace : (WS | LINE_CONTINUATION)+;

Comintern

If you see a non-lexer identifier, it's another parser rule.

Mat's Mug

^

Comintern

visibility : PRIVATE | PUBLIC | FRIEND | GLOBAL;

FreeMan

@Comintern I couldn't even find what I was looking at yesterday. Def time to crash. see ya in the morning.

IvenBach

04:50

visibility <-- Wouldn't it have been better to use accessModifier ?

Mat's Mug

all parser rules either refer to other parser rules, or to lexer tokens

Comintern

@FreeMan Give me a ping when you're working on it - we can team debug it if you want.

@IvenBach The terminology comes from the VBA language specification for the most part.

IvenBach

Ok

Mat's Mug

also we can't just go and rename parser rules on a whim - just renaming VisibilityContext would break the parse tree listeners handling these rules

but let's not get into listeners and tree walkers just yet

arg : (OPTIONAL whiteSpace)? ((BYVAL | BYREF) whiteSpace)? (PARAMARRAY whiteSpace)? unrestrictedIdentifier (whiteSpace? LPAREN whiteSpace? RPAREN)? (whiteSpace? asTypeClause)? (whiteSpace? argDefaultValue)?;

IvenBach

@Mat'sMug We gotta stay focused on making the sausage and eating it after.

Mat's Mug

04:55

lol

so parser rules are really just abstraction layers: the more parser rules, the more expressive the grammar becomes

IvenBach

OK

Mat's Mug

when you're done drilling through parser rules, down at the bottom of every branch of that parse tree, is lexer tokens

the fun part is that parser rules can be recursive

// 5.6 Expressions

the # stuff at the right isn't a comment; these identifiers become derived types

AddOpContext : ExpressionContext

IvenBach

Starting to not follow...

Mat's Mug

| expression whiteSpace? (PLUS | MINUS) whiteSpace? expression                                  # addOp

IvenBach

derived types? Kind of like Base class and derived class?

Mat's Mug

04:59

exactly

Comintern

^

IvenBach

so <addOP> = expression whiteSpace? (PLUS | MINUS) whiteSpace? expression

Mat's Mug

yeah

Transcript for