« first day (1327 days earlier)      last day (1853 days later) » 
00:00 - 21:0021:00 - 00:00

9:00 PM
Database + a LOB is when you need a rigorous data entry / import process to keep data clean/valid.
 
LOB?
 
line of business
 
what's that actually defined as?
I've been doing minor updates they've requested. I want to migrate this out of a Excel into its own dedicated program.
 
can literally be anything - could be a web application, a C# desktop application, an Access application, heck, even an Excel XLSM (which Mat did) but basically it provides interface for users to carry out their tasks --- salespeople entering their sales order, shipping people deducting shipped stuff from the inventory, that kind of thing.
it's very open ended definition but you also can think of it as "mission-critical" since it tells everyone what's what at the business.
 
I'm running 2 workbooks with 500k records each through 2 clusters.
 
9:03 PM
you do exactly what?
 
4 mins ago, by IvenBach
#IDoWhatImTold
 
like, a what-if analysis or data cleanup or make reports out of raw data?
 
I got the spreadsheet to go from sequential calculation to use HPC (High Performance Computing) for Excel which lets the calculation be offloaded to a computing cluster.
 
@Hosch250 to follow on your thoughts
 
@this I know the cluster is a temporary band-aid since it's not viable long-term.
 
9:06 PM
@Hosch250 waterfall faster is agile. youtu.be/k_ndH7B-IS4
 
interesting. Do you repeat the calculations with different inputs?
 
@PeterMTaylor I can believe it. But I do waterfall so fast we turn it around it in almost-Agile times.
 
I kept thinking it’s an impostor syndrome lol @Hosch250
But point taken
 
@PeterMTaylor I just want a big pencil now.
@this Yep, calculations are run with varied inputs.
 
Tricky to write a shopping list I think @IvenBach
 
9:13 PM
yeah, definitely better w/ a database - you can set up for headless automation that way.
 
@this In one column for Field Foo you have A,B,C,D. Another is Bar 1,2,3,4. Combine them together and you have 16 combinations. Add another field/column Duck with Cat,Dog you get 32 combination... etc...
@this I'm very interested in learning about this.
Not specifically about you but about properly setting stuff up.
/badjoke
@PeterMTaylor Easy to poke distractions out of your office though.
 
From your colleague you mean?
 
without knowing more i can't really advise any good recommendation beyond the fact that if you do need to be able to run your calculations without watching paint dry, you likely need a SQL Server (which can be used in headless automation)... Depending on your security policies, you might be able to use SQL Azure, which in turn enable you to use other stuff in Azure that would help with running calculations and sending you the results.
 
I don't know how to accurately describe what I'm doing either.
 
You'd best first.
Otherwise this will only end up in failure
because a good LOB application really does depend on having the process clearly and well defined
if they're just flying by the seat of their pants....
 
9:28 PM
I have to sit and think how to accurately describe what is being done. I haven't done that yet.
 
if EXCEL is used primarily as datastore and you're running long-running VBA procedures to extract any meaningful data, you should probably change to a Database
if EXCEL is primarily used to generate BI and other soft data-requirements, you can still go and use a database, just be aware that generating nice charts from databases is a bit more programmery than selecting a range and saying "generate chart"
 
meh. generating a chart off a db on a SSRS report is pretty much selecting a bunch of fields and saying "generate chart" :)
 
Excluding the parameterized run there is one table that should be in a DB.
I'm all for knowing what other options are out there to make a better informed decision on what to do.
Continuing with how it's been done won't cut it long-term and I know it.
 
hmmm ... either way I'm back to banging my head against OpenGL, hoping it budges
 
If I do move to SQL I'll make sure to harass @puzzlepiece87 for what info he already knows.
 
9:42 PM
I'm available whenever you need help :)
Not an expert but at least an intermediate, I hope.
 
@Vogel612 Yep agreed. That's why it pays to have your requirements clear.
 
I don't know very much of what I don't know for SQL - the main thing I know I don't know is the intricacies of why using (nolock) all the time is a bad idea.
@Mat'sMug Does github have an easy way for me to search all the code in a repository for a specific phrase, without opening or going to each file's page individually?
 
because nolock says I can rip out the page from the book you are reading at this very moment.
 
^^ that
oh and because nolock violates the A and I from ACID
 
Ask and you shall receive.
 
9:45 PM
more generally, it's abused as bandaid for something else
because SQL Server by default uses pessimistic locking model, if you have too many readers, it's easy to get blocked
 
@this Right, that's how my team uses it. We use it because we're generally okay with dirty reads and don't want to hold up more important database actions.
 
and people go, "oh I'll make it nolock"
 
and if you're not careful, you can kick the consistency of transactions right into the nuts when they're executed with nolock and modify the same records
which can lead to unnecessary rollbacks..
 
Exactly, people do it to try to avoid blocking each other
 
but not even realizing that the correct action was to add index so that readers would no longer block other readers or writer.
 
9:46 PM
Ah, so add the index column to the query instead?
 
A covering index is worth much more than hint liberally sprinkled over everywhere.
covering
 
Or add an index to the table?
 
@puzzlepiece87 that....
 
Ah, all of our tables are already indexed.
 
by definition, an index can be only added to a table (I suppose you can add it to a view but there's lot of limitations)
but are they covering?
 
9:47 PM
if you do slow reads, that's generally because you have a table_scan instead of an index_scan
 
that's the important thing.
to put it in concrete terms -
 
@puzzlepiece87 you mean aside from the seach box at the top?
 
@this Not sure, I don't know what a covering index is. I'll research.
 
suppose you have SELECT .... FROM someTable WHERE colA = 123 AND colb = 'abc';
and you created 2 indices --- CREATE INDEX IX_One ON someTable (colA); and CREATE INDEX IX_Two ON someTable (colB);
 
@Mat'sMug Got it, you can search by "this repository". Thank you!
 
9:49 PM
that's not covering because the engine must compare the 2 predicates, so it will end up NOT using the indices at all.
 
@puzzlepiece87 hit backspace and you'll search the whole of github
 
better: CREATE INDEX IX_dat ON someTable (colA, colB);
 
@Mat'sMug I may have one more question in a bit - I'm trying to implement ADODB "try again" functionality because one of our databases has been having connection issues. I'm searching through your VBTOOLs to see how you did it for my pre-question homework.
 
now this "covers" because it matches the query's WHERE, so engine can use it to get good selectivity and therefore faster (and non-blocking) reads
 
@Mat'sMug Just made that mistake
 
9:50 PM
@puzzlepiece87 I don't have a readymade solution for this
 
FWIW, I wouldn't manage transactions client-side.
 
that's a job for a circuit breaker pattern
 
I'd rather do transactions server-side.
 
@this sometimes you have to
 
naw. I don't believe that.
got data here but server's there? dump it in a staging table.
then run a transaction there
 
9:51 PM
why would ADODB.Connection expose a BeginTrans member then
 
no client-side transaction need.
 
@this Understood, though I guess there's not much I can do if someone wants to query for unindexed columns in their where clause.
 
the best justification, IMO is when you are orchestrating other "transactions" (let's say another database that can't be linked directly), but even so, staging tables still is much better solution.
or SSIS if it's a regular thing and can be automaed.
 
@Mat'sMug @this I'm guessing you'd both severely frown upon something like "if continually unsuccessful, try again every second for 30 seconds, then alert the user"
 
I wrote a client-side transaction no later than last week. #DoingItWrong
 
9:54 PM
retrying exactly what?
there are processes that you just have to retry.
 
To connect to the server. Our error is ORA-12170: TNS:Connect timeout occurred
 
@Mat'sMug I've done that myself but every time, I get burned and found that implementation became much simpler once it's all server-side.
 
if your timeout is 30seconds, you're continually retrying
 
All that's needed to fix it is to try again, so I'm trying to have my queryer try again automatically rather than needing the user to push Debug, Continue.
 
9:56 PM
@this can't say it's perfect code, but it works and doesn't create half-records
 
@Mat'sMug My timeout is 0, which on Oracle I think means a single try.
For SQL Server it's unlimited tries I think.
 
in any case, it's useless to continually retry connecting to a server that's offline
 
I'm wondering if it's really connecting
 
The server's not offline, the connection is just momentarily lost.
 
I don't work with Oracle so.
 
9:57 PM
And I'm using lost inappropriately
 
are you caching the connection?
 
The connection is momentarily not present.
I am not sure and therefore will say no.
 
rephrase- are you always newing up your ADODB.Connection when you run your query?
 
(hopes for a yes)
 
yes
woo, made @Mat'sMug happy!
 
9:59 PM
@Mat'sMug TYVM.
 
good and do you know for the fact that the query itself does not take > 30 seconds?
again, I don't know Oracle but w/ SQL server, there'd be a different error message, though.
 
@this You can probably set the timeout.
 
> Handle faults that might take a variable amount of time to recover from, when connecting to a remote service or resource.
 
Private Function OpenConnection(ByVal QueryMetaParameters As classQueryMetaParameters) As ADODB.Connection

    Dim cnn As ADODB.Connection
    Set cnn = New ADODB.Connection
    cnn.ConnectionTimeout = 0
    Dim strConn As String
    With QueryMetaParameters
        If .strSQLApp = "Golden" Then strConn = (string)
        If .strSQLApp = "Teradata" Then strConn = (string)
        If .strSQLApp = "SQL Server" Then strConn = (string)
    End With
    On Error GoTo ErrorHandler
    cnn.Open strConn
 
9:59 PM
At work, we use 2 seconds, which is ridiculous.
 
Yes, hosch, the timeout can be changed but it's also a code smell if you must change it to make some query work
with ADODB, there are 2 separate properties, one for connecting itself and other for executing queries.
 
@this Good point: no, I do not. However, I think I dodge that because I'm only trying to retry on error. An error doesn't occur if the query's still running.
 
ConnectionTimeout and CommandTimeout, respectively
 
We still get timeouts occasionally on some stuff. I'm busy working it out to use more optimal queries.
 
I think ADO.NET has similar properties
 
10:02 PM
@this Yeah, I have my ConnectionTimeout as unlimited, basically. But I think it's returning that error when the server can't be reached to start the command.
 
@puzzlepiece87 again, I don't know Oracle and therefore if it'll yield a different error when the query itself has timeout.
Generally, i think unlimited is not really helpful. If anything, I'd actually decrease the ConnectionTimeout
 
@this Okay, I'll verify that I have a connect timeout rather than a command timeout. Pretty sure it's a connect timeout.
 
more so for a LAN-based connection.
no point in waiting 10 minutes to connect. :)
 
Lunch time.
 
and even across WAN, no response in ~10 seconds usually is bad news.
 
10:04 PM
@this Good advice. I instantly understood the distinction between connect and command timeout. I think that the SO answer I originally got ConnectTimeout = 0 from was actually trying to protect against commandTimeout and didn't know how to properly do so.
 
so yeah, first find out for a fact that your query shouldn't take more than 30 seconds
and that it doesn't cause blocking.
if it could get blocked, that would cause timeout especially if you only get it during busy time.
Again, I don't know Oracle but I am told that it's more optimistic than SQL Server WRT its locking model so that might be less of an issue but #VerifyFirst.
 
Um
I definitely want to allow queries that take more than 30 seconds
 
ok - and it returns something?
 
And I'm not very concerned with blocking on Oracle, we rarely have that problem on Oracle
Peoples' queries do return something after the connect timeout error is passed by trying again, yes
(because the server is unavailable momentarily, it seems)
This momentary unavailability also happens in other clients
 
well if the queries in fact take more than 30 seconds to execute, you might need to increase CommandTimeout
or make it unlimited (which does make sense)
 
10:09 PM
Got it. I think CommandTimeout is already unlimited
I have had no examples of people's queries raising errors when taking too long.
I'm fairly sure this is a connection issue.
Like we're not able to make the initial connection
 
I'm just hoping it's not something lame like Oracle reusing the error message for connect timeout with that query command timeout
Again, I don't know Oracle.
 
That's fine, I still appreciate your expertise.
 
Ok I lied. I know enough to dislike it. :p
4
 
I dislike it too :P
I hate how few options I have for temp tables
 
It's Oracle. What's not to dislike?
5
 
10:11 PM
i wasted way too much time only to find that this isn't valid in Oracle SELECT a As d.MyThing, b AS d.MyOtherThing FROM MyData AS d WHERE a = 'foo';
wonders who else will spot it
 
I'm going to do my "wait one second and retry up to 30 times" solution, thanks for teaching me and helping me narrow it down.
 
one more dumb question, probably -
when you do retry, is it "done" sooner?
Like, if it takes 40 seconds to run, you time out on 30 second, retry 2nd time, then you get result in 10 seconds?
 
NAFAIK.
 
@this Unknown, but I'll guess no. It happens so rarely that it's impossible to say. #Can'tRecreate
 
OK, didn't think it was likely but again, I don't know Oracle.
Hosch, you use Oracle?
 
10:14 PM
Thanks for asking.
@this Is the issue that you didn't put "WHERE d.MyThing = 'foo'"?
Is it that you can't reuse a?
 
nope
oh drats that's wrong
it should be WHERE d.a = 'foo'
but that's not the actual mistake that wasted too much of my time
let me rewrite the query to eliminate that stupid mistake.
that is what I get for typing too fast. :(
SELECT a AS ColumnOne, b AS ColumnTwo FROM MyTable AS d WHERE d.a = 'foo';
 
Can you do select a [ColumnOne]?
 
yes that is legal
the AS is optional
(not sure about the square brackets, though - don't remember off the cuff how Oracle uses brackets for identiifer)
 
So, that doesn't work in Oracle?
It's perfectly valid T-SQL.
 
No it does work in Oracle
 
10:26 PM
Oh. Then what's the problem we are trying to see?
 
yes, the complete statement above is valid T-SQL but it's not in Oracle
 
That's what I said. That statement isn't valid in Oracle?
 
This is valid in Oracle: SELECT a AS ColumnOne, b AS ColumnTwo FROM MyTable d WHERE d.a = 'foo';
 
Oh, the MyTable as d doesn't work?
Whatever.
 
yea yet it's OK to do a AS ColumnOne or a ColumnOne
 
10:27 PM
I never use as anyway, so...
 
that was very very WTFish. Pissed me off when I was trying to debug this with Oracle.
 
LOL.
 
I do think only evil people will design a language where they make it optional in only some place but illegal in only certain places.
 
Probably an oversight.
 
for last 2 or 3 decades? Yeah, sure.
 
10:29 PM
Yeah.
 
and let's not worry that it violates SQL standard, too.
#NoBigDeal
 
It was probably originally an oversight, and then they didn't care.
 
or they're that arrogant.
 
@this, that would piss me off too.
 
of all databases I've installed, my experience with Oracle was the worst. Kept had to restarting it because.... I don't know what. It's kind of sad that it was an "express" (which mean it should be easy to setup).
 
10:31 PM
Yeah, T-SQL is the only one that's any good.
 
Teradata is also frustrating
 
well, postgresql is pretty nice.
 
Private Function OpenConnection(ByVal QueryMetaParameters As classQueryMetaParameters) As ADODB.Connection

    Dim cnn As ADODB.Connection
    Set cnn = New ADODB.Connection
    cnn.ConnectionTimeout = 0
    Dim strConn As String
    With QueryMetaParameters
        If .strSQLApp = "Golden" Then strConn = (string)
        If .strSQLApp = "Teradata" Then strConn = (string)
        If .strSQLApp = "SQL Server" Then strConn = (string)
    End With
    On Error GoTo ErrorHandler
    Dim lngRetryCount As Long
^ For reference/anyone's interest, that's my quick solution for now.
 
@this if only I could pronounce it lol
 
FWIW, I think that function tries to do too much - it should be only concerned with opening and returning a connection but in the error handler you say the query will be skipped. that's not really in the function's scope of responsibility. Better to throw an error back to the calling function which presumably knows what query it needs to run and thus report it properly from there.
@Mat'sMug at least it's something more manageable than Bobby's full name. ;)
 
10:39 PM
@this Oh okay, that's good feedback. I like that.
@this The function doesn't do any skipping itself, just notifies the user there, but I could move the message up a level.
@this Thanks!
 
Yes that's what I was getting at - it belongs up there, not in that function.
you're welcome!
 
@this Okay, so if I asked you "But how will I know which error occurred outside of the function", you answer would be one of the following, right:
1. Pass the error back up instead of just the connection
2. Err. is global, you don't even need to pass it
Something like that?
 
#1 - just do a Err.Raise, set your custom message if you want
and let the calling function deal with it
 
Great, thanks, that's proper bubbling up right there.
Mat will be happy that you helped me bubble up better.
Do I even need to Err.Raise? It's already been raised, right, since I'm in the error handler? Or are user-raised errors dealt with differently than system-raised errors?
 
Err.Raise inside an error handler is as if there never was an error handler
in essence the error handler is now saying "oh, that error? Not my problem! Let the guy before me take care of it! I'm outta here, thxbye!"
 
10:46 PM
Got it. But if there's already an error, can I just do Exit Function? Will higher level function still be able to detect the existing error without re-raising it? Nm, I'll just Google if errors are global.
 
no, no know what, why don't you write few functions, and do step by step with different error handlers set up
try different combinations
get to really know how VBA handle errors and how it acts on various control statement
you'll then see that Err.Raise inside an active error handler (or a procedure with no error handler) will exit the procedure immediately
 
11:06 PM
@this I did some experimenting, and I see you need to Err.Raise because (system errors, at least) errors aren't global. So you have to use Err.Raise to pass it up.
 
@Computergaga1 Hmm, I'd consider logging to a text file before writing what boils down to trace logs to the registry.
So many bloggers give dangerous advice...
Logging to the registry with SaveSetting, seriously?!
 
@puzzlepiece87 there is no distinction between a system or user error
but yes, Err.Raise will cause you to leave the current procedure unless there's an error handler and it's not already active.
@Mat'sMug at least he's only overwriting only one entry. I guess he didn't want the burden of dealing with a text file.
I wonder how hard it would be to get logging to event log set up. That would more "better" in the sense that there's no text file to manage but registry isn't abused, either.
 
@this there's gotta be a COM API for that somewhere
@this TBH the file logger is an easy cause of headaches, given a hard-coded path and/or multiple users. Best log would be to the Application event log indeed.
 
11:21 PM
yeah, exactly.
the worst thing about desinging something like that is that if you do it wrong, you can cascade
try to log that a drive is out of space....
not a wise thing to do.
 
11:37 PM
Yeah but abusing appsettings for logging? that's just horrendous
 
00:00 - 21:0021:00 - 00:00

« first day (1327 days earlier)      last day (1853 days later) »