REFRESH!

[Minesweeper] 105 Games Played. 65 Bombs Used. 13120 Moves Performed. 15 New Users

[rubberduck-vba/Rubberduck] 1 issue comments

[Rubberduck] 2 Synchronizations

So, I had ah-ha moment: the proxies have the JsonRpcMethod attributes, thus have the protocol-defined method signatures... which may or may not be practical from the caller's standpoint (e.g. I want to LogTrace(LogLevel, string), not LogTrace(TraceMessage))... So there's a Rubberduck.RPC.Proxy.SharedServices.Console.ConsoleService class that wraps the proxy and prepares the request parameters and gets to convert a response (i.e. a LSP model class, or some InternalApi.Model type)

..before returning to the caller

so we'll be injecting services instead of proxies 🥳

Just curious about the impact of the new RD3 architecture on the existing RD test suite. For testing, will the LSP and DB be in separate processes for testing as well?...and, regardless of where they reside, would they be spun-up/down for each test?

hm that's a great question!

but no, unit tests wouldn't spin a server process :)

but ...yeah it destroys much of the whole "mock the entire VBE" approach

tbh I knew the test suite was going to be a massive issue, ...so I stuck my thumbs into my ears and started singing "lalalalalala" whenever the thought of unit testing RD3 came to mind

not because I don't want to think about it... it's just... a "later" problem for now

in theory it should make everything simpler to test.. I think.

to test the LSP implementation we'll have to mock the LocalDb proxies

(and the LspClient proxies for LSP->client requests/notifications)

VBE mocks will be mostly useful for testing the code that depends on client->LSP proxies

Ah, OK thanks...so it's a substantial impact. But also sounds like it is probably not worth spending much mental time on until the RD3 design/implementation is a lot further along. Gotta say, the progress on RD3 is really quite amazing. :)

Thanks! The ball boulder is finally moving!

Max is right though, there isn't much of RD in there yet.

Actually, once the architecture solidifies (stuff is still flying around everywhere as my understanding progresses) the add-in and UI can very well progress concurrently with the LSP implementation, since the add-in/UI really only needs to call the server proxies and pretend the response isn't anything special.

damn, this is probably one of the first times in ~20 years I ever need to write get/set logic beyond OnPropertyChange()

    JsonRpcConsoleVerbosity _trace;
    public JsonRpcConsoleVerbosity Trace
    {
        get => _trace;
        set
        {
            if (value != _trace && _traceLevels.TryGetValue(value, out var newTraceValue))
            {
                _trace = value;
                var parameters = new SetTraceParams { Value = newTraceValue };
                var request = ServerProxy.CreateRequest(JsonRpcMethods.SetTrace, parameters);
                ServerProxy.Notify(request);
            }
        }

[rubberduck-vba/Rubberduck] cwest28 starred us

@MathieuGuindon Does this mean that if I set logging to Trace from the Access client, that the Excel client will automatically switch from Warning to Trace, or is that some other level of verbosity I'm not thinking of?

@FreeMan trace level is just Off, Message, or Verbose; basically when we log messages we can say "something happened" in the message and then the verbose message might include additional information like line numbers or tokens or whatnot - these only get logged when trace is set to Verbose. Since it's a server setting (per protocol), that means yes, requests from all clients get the same treatment.

So you're not talking about the log file I'm used to seeing & including in bug reports, then?

Kind of; the log file is pretty much obsolete in RD3, but yes trace logs are the logs that would end up in that file with 2.x

The log level is kind of a filter on what gets written there

(error/warn/info/debug/trace)

it's a different setting (not LSP-defined) than the trace level

I've added a [LspCompliant] attribute to mark members defined by LSP that are intended to follow the LSP specification

So trace logs are sent client to server via a $/logTrace request. Then there's server to client window/logMessage requests for what I think is another type of logs that the client is responsible for.

I'm still a bit lost, but I'll trust you on this. I'm sure it'll make sense once I see it.

I presume that I'd be able to point any SQL front end tool (like SSMS) at the SQLlight DB file and rummage about in the db myself, just to see what's there?

With a SQLite client, yes. I think SSMS is SQL Server only. Have you tried Azure Data Studio?

That said the LocalDb client app is going to have some level of data viewing and exporting capabilities, so one doesn't need to know how to query SQLite to see what's in there.

Now I wonder if Excel can import directly from the .db

(and can Access use it as a source?)

that could lead to some interesting meta-programming feats

VBA reflection via SQLite lol

Hm, if then there could be a COM client library that does exactly that

10 points!

:D

Note, a Mug earns more points than a physicist or mathematician

@MathieuGuindon Theoretically, yes - there's an ODBC driver for SQLite. However last time I experimented with it, it was crap. IDK if they improved on it since. It's a third party sooo.... I know there are tools that you can download to view & run queries on the SQLite database but not sure if that let you do it for a file in use by other processes.

(having never used SQLight myself, I have to ask) based on @this question, can SQLight support access from multiple users/processes? I'd presume so, but it's worth asking...

I am not familiar with this "SQLight" thingee.

But SQLite does theoretically support multithreading. But even so it's only for show because ultimately there is only one locking level and that's serializing the data access.

They have a FAQ and it's the question #5

The main difference is that SQLite is basically just a file, there's no "server" per se, it's intended to run in-process

So no, it doesn't do multiple concurrent accesses

(that's why we need a LocalDb server process 😉)

@FreeMan lol!

FWIW, this side remark makes me think they've managed to implement infinite improbability drive:

> Under Win95/98/ME which lacks support for reader/writer locks, a probabilistic simulation is used instead.

Yikes

@MathieuGuindon Ahhh, gotcha! The localDb server process is the one & only thread that accesses the db - to make it multi-user/process you're writing a client that handles requests from everyone else. lightbulb

Yes!

probably should add a note to RD3 doc --- even if one got it running on 95/98/ME, they should expect lot of interesting results.

95/98/ME wouldn't run .net 4.8 anyway, no?

Probably. I'm sure there are lot of other obstacles

but even if they managed to get SQLite running on it... #HaveFun

We'll build it in .net6 just to make sure it doesn't 😂

(tbh I don't care much about the target runtime for now)

Agree. That's a later problem.

@this "probably should add a note to the RD3 doc --- if one gets it running on 95/98/ME, WHY???? Also, there are serious security/performance issues and you need to upgrade a decade ago" FTFY

@MathieuGuindon yay me!

@this something to do with a lightbulb, probably!

@FreeMan but, but, I want to enable the infinite improbability drive feature in SQLite and it's only available in 95/98/ME! Is that not reason enough?!

@MathieuGuindon LOL

@this You'll have to have a chat with Hotblack Desiato about borrowing The Heart of Gold, but he's spending a year dead for tax reasons.

:)

I knew exactly what you meant, but it's been years since I've read the books, so I had to go look up the names.

did a few tests, and it appears the [JsonRpcMethod] attributes have to be on the concrete classes after all (unless JsonRpc uses extensive reflection to pick up implemented interfaces, that is... but I doubt it)

is the JsonRpc nuget package the only choice?

looks like there are several to choose from. Whether it'll be better would have to be researched...

Hm, StreamJsonRpc would be the go-to, it seems

and I think you want this, too

Oh my. Damn right!

and we're back to interfaces. Much easier to grok

I'll take the time to read everything about it after work, I very much like the idea of using a Microsoft library for this

Weird

> A method's return type may also be void, in which case the method sends a notification to the RPC server and does not wait for a response.

But the previous section says (point 2)

> Methods are sent as ordinary requests (not notifications).

yes, confusing -- I think what they mean is that a void method is a "fire-and-forget"

so it's not a true notification in the sense that the server is telling client something happened.

Ok, handling responses is going to be a little bit trickier than the super-friendly API I wrote, but it's going to work. Actually with a service layer calling the proxies we'll get the friendly API anyway.

@this I think the docs are just missing the words "with a return type", meaning not-void methods are sent as ordinary requests

could a server send a notification and get back a response from the client?

hmm, i guess technically, that's a request from the server-as-a-client.

yeah no, by definition a notification is fire&forget

but yeah, server to client notifications are sent to a "client process" proxy

where the "client process" is in the role of a RPC server

it really helps to scope the terms "client" and "server" to a single request, and forget about client and server apps

that implies that the notification sent as event cannot be used to receive any data via the event parameters

that would be a request

actually with that library cancellation works by making the last parameter a CancellationToken

otherwise whenever there needs to be a return value, it's always a request+reaponse

Right. It does implies that if we have a pattern where we have a notification from X, then respond to it using another notification or request back to X, that would be a code smell

More likely, we want to ensure that a notification is sufficient in itself (e.g. provides all the data without the X asking for more) or at least the X should queue that data, and have another task/thread to then act upon on it and sending requests for more data as needed. (at least I think so?)

I think that makes sense, yeah

Notifications are protocol-defined though, not worried about them being insufficient

Some responses can prompt follow-up requests though, especially if the client signals that it supports partial results

The "sufficient" part would depend on how the data is implemented in the notification (in this case, as an event argument).

i.e. you get an array of CompletionItem, but they don't have a description tooltip until the user hovers one and the client requests the resolution of that particular CompletionItem

right. that's a good thing generally.

indeed!

the StreamJsonRpc has an IAsyncEnumerable which would help make it simple, hopefully.

IIUC that's useful for streaming a response (and/or request?)... basically we could have the client stream the text document content to the server instead of sending it the whole string.

swaps LSP server for Netflix, starts streaming

That's great because the AvalonEdit TextDocument can do exactly that

Not sure if very beneficial though: server needs the whole document to begin parsing it anyway

that's awesome!

let parsing part run in background

Or can we make a token stream from... a text stream?

load the text immediately

@this that one's a given ;-)

IIRC the Antlr lexer only works off a string.. if we could make it accept the text stream from the document, we could be lexing tokens as the text document arrives server-side, and then start parsing immediately... that shaves a whole whopping.... ...uh, one, maybe two milliseconds?

IDK if it's worth it tbh

We're never going to receive more than 10K*1024 characters

and 99.999% is going to be much, much smaller payloads

1) make it work

it's still good that we're identifying areas where perf could possibly be improved though

I was thinking more about the loading of text into the avalonedit textdocument from the files as a stream. We definitely want to minimize any changes to the parsing pieces as much as we want, though.

(for the initial release. as you say, make it work)

oh.. that would be an implementation detail of the client, no?

yeah. not sure if we want and can have the AvalonEdit to run async from the VBIDE host.

naaah

probably overthinking. just didn't like how VBIDE has its own mind that might make everything feel slow

we'll think of something for the workspace initialization sequence

we probably don't even need to interact with the VBIDE at all, other than syncing the code

yeah...

that syncing is also a client implementation detail

thinkinga bout it more, sync probably can happen fairly infrequently

indeed!

mainly because we don't want to pollute VBIDE's extremely limited undo stack

in my mind we start the add-in, connect to the server and initialize LSP: when we initialize the workspace, we load the documents. and then we sync on close, and we sync on save (/compile)

on document close, I mean

if we're now importing document, it's no longer possible to have dirty documents for worse or better.

like, when you close an editor tab, the content goes to the VBE

yep, if you run the RDE and make changes in the VBE, that's on you

except perhaps in a debugger session... we'll want to re-sync when the IDE enters edit mode, if we can pick that up

yes, but I mean that the workflow changes

totally

in VBIDE, it's possible to have 3 documents edited and dirty, compiled and go into debugging without saving anything.

true, and that's a bloody mess :D

with RDE, that wouldn't be possible anymore and as soon as we start debugging, those 3 documents edited in the RD3 editor would have to be imported (effectively overwriting) before going into debugging.

yep

but I think we can handle this with our unlimited undo stack to restore the state if we want.

have you read the LSP spec yet?

only glanced at it but not really dug into it.

each document edit increments the document version, so the server can very well keep more than just the current version around

(and push it to the client)

I see that does simplify things

the biggest challenge is going to be to not treat the LSP server as unlimited out-of-process memory :D

ok maybe not the biggest

perhaps we should build the LSP server for x86, that way we limit ourselves to what, 2GB?

I need to add process memory info somewhere in the server console apps

[rubberduck-vba/Rubberduck] 1703 stars vs. [decalage2/oletools] 2312 stars