The 2nd Monitor

Jeroen Vannevel

14:03

You can visit London

Handsome people there

Mat's Mug

@CaptainObvious tempted to hammer off-topic as broken code. OTOH that code is impossible to fix without first addressing the readability issues.

How does it freeze at 7K rows when you have 1,668 rows per sheet? Does it run to completion with a smaller data set? It's extremely hard to follow what your code is doing, with that thrice-nested loop, the insufficient indentation, confusing or irrelevant comments and the single-letter variable names - I'd suggest you try to improve the readability first, and then make changes to improve performance and memory footprint. Otherwise you'll be making changes and very likely introduce bugs. — Mat's Mug ♦ 1 min ago

Dan Pantry

that indentation is pretty screwy

Mat's Mug

and not all variables are declared, missing Option Explicit makes it compilable nonetheless

Dan Pantry

#NotAllVariables

Zak

@Mat'sMug IMO, "out of memory" for beginners is probably best treated the same way we treat "time limit exceeded".

Mat's Mug

14:06

@Zak do you see anything that could cause an OOM error in there? I don't...

@Mat'sMug 1st loop goes through the sheets. The second loop gets each row and the 3rd loop appends all the supplier + price... to the line. It just freezes after a few minutes and after closing I receive out of memory error. The single word variables are just iterators. — Lóránt Csaba Mihály 34 secs ago

"just iterators" ..uh, no.

after closing - could be unrelated to that code

Zak

@Mat'sMug Excel can be weird like that. It hasn't got the best GC in the world.

Captain Obvious

1

Q: Scipy matrix: optimize conditional removal of columns in large matrix

I have a large dataset (78k instances x 490k features) that is loaded as a scipy.sparse.csr_matrix format. From this dataset I want to filter certain features (i.e. columns) for which all values fall below a certain threshold. Loading the dataset as a dense matrix is not an option, nor did I fin...

Mat's Mug

it's COM, there's no GC. it's reference-counting.

Zak

@Mat'sMug Anyway, I'm happy to answer it in its' current form. OP definitely needs the advice.

Mat's Mug

aye

Zak

14:10

Also, just validated my first chat flag ^^

user54089

Sorry to bother you guys -- is CodeReview an appropriate place to get feedback on things like overall table layout? I'm developing an app and I am trying to learn good practice, but I'm not at all confident in what I've done so far and don't know anyone IRL I can ask

Mast

@Mat'sMug Nothing wrong with a readability review in the answer.

@user54089 Have you read the on-topic help centre?

Zak

@user54089 Assuming you mean *visual* layout, that's not what CR is for.

Mat's Mug

@user54089 a db schema? the schema itself isn't reviewable, but if you have a SQL script that drops/creates the tables and their constraints, that is reviewable

user54089

Not visual layout, no -- I mean more my relationship between my objects and my database tables and whether or not it makes sense / follows proper OOP principles / normalization principles

Mast

14:12

> On a [email protected] machine this takes 42 minutes on my dataset.

(source)

Highly inefficient.

user54089

It seems very difficult to find places to get feedback on design but CodeReview is the closest place I could find

I can pay someone if necessary

Zak

@user54089 If you just want a design/schema reviewed, no, not On-Topic. If you have code built around/on top of it though, we can review it, and reviewing the architecture will probably be part of that.

Mast

Are you only interested in database table feedback?

Database Administrators might take it.

But don't pin me on that.

Zak

The key thing to understand is that CR reviews actual *implementations*

Mat's Mug

unfortunately we're not design review. we can review your script and say "you should drop your constraints first, then drop your tables, then create them and reinstate the constraints, that way the script can run multiple times in a row without breaking anything" for example

Mast

14:15

Please read their on-topic help centre as well.

It doesn't sound on-topic for Code Review I'm afraid.

Mat's Mug

have you tried dba.se?

dba.stackexchange.com

user54089

I'm asking in their chat now

Mast

Very good!

Mat's Mug

good luck!

Zak

Even if it's not On-Topic for a proper question, if you ask you'll probably find people willing to give you some informal feedback.

user54089

14:16

What is the difference between code review and design review exactly? I would have thought they were the same

Is it more akin to "bugfixing"? Because I thought that was more SO?

Zak

@user54089 Code Review is "Here's some code I wrote that does X, how could I have written it better".

Mast

Code Review requires working implementations.

user54089

Isn't that still design?

Making it "better"?

Mast

Design review could be whiteboard sketches and pseudo-code.

Zak

@user54089 Design often comes up in the course of a review, but it's always in the context of an actual implementation.

user54089

14:18

I do have actual implementations if that changes anything

Mast

Yup.

user54089

I am just not confident that I did it well

Mast

Doesn't matter.

Mat's Mug

@user54089 Code Review is: "Here's my working code. Does it make my ass look fat?"

Hosch250

Well, what about this question? codereview.stackexchange.com/questions/111430/…

Mast

14:18

Do you want a review of the given implementation or a review of the design regardless of implementation?

user54089

@Mat'sMug What does "ass look fat" mean in this context though?

5

Mast

The first can be on-topic here, the latter not.

FINALLY somebody dares to ask the question.

2

Duga

@hey: Maybe you should post a code snippet on CodeReview - if you have a problem like that, it's quite possible you could restructure things in a way that would mitigate it. — Baldrick 13 secs ago

Mat's Mug

lol

user54089

Did I ask the wrong question? lol

Mast

14:19

Nope.

Dan Pantry

You asked entirely the right question.

Mat's Mug

most database questions are borderline, gray area

Mast

Yup.

Mat's Mug

CR works better for optimizing your queries and making your DDL scripts readable and easy to maintain

Zak

@user54089 It sounds like you might have a CR question, but difficult to say without seeing it. Do you want to try writing a question in a pastebin (or similar) and we can tell you if it would be On-Topic.

user54089

14:21

I can't access Pastebin from here unfortunately (work computer)

Blocked as a filesharing site

Mat's Mug

if you're more interested in the design itself and the relationships between entities, the normalization and DB theory stuff, IMO DBA.SE is a better place to ask, but they have their own rules too (so asking in their chat is the best thing to do)

Mast

@user54089 How about gist?

user54089

I guess what I am interested in is "Here's my code. Does it make sense / is it designed well, or does it reek like something you'd expect a total noob to make?"

Mast

We do accept noob code.

We even got a tag for that.

beginner

Mat's Mug

we LOVE noob code

2

user54089

14:23

I want to un-noobify my code

7

Mat's Mug

and JAVA

user54089

It is in Java

Mast

JAVA is special.

Don't mind us.

Mat's Mug

lol

JAVA is to Java such as PHPOOP is to PHP

Mast

32

A: What's a Zombie? And what are the many other memes of Code Review?

Meme: Java vs. JAVA Originator: Jamal(?) (to be determined) Cultural Height: The 2nd Monitor (as usual), but also in various comments scattered across meta. Background: Question with the word JAVA (instead of Java) in the title or body, is often an indication that the poster is a beginner and/...

Zak

14:24

@user54089 Tell you what, put a question up with your code (maybe read a couple questions before to get a feel for how to write a good CR one.) and we can always tell you if it's On-Topic or not.

Mast

@user54089 I assume it works as intended?

user54089

Yes it all works

But I mean, it also "works" to carve through a mountain with a spoon

Doesn't necessarily make it right

Mat's Mug

52

Q: How to get the best value out of Code Review - Asking Questions

I have a project I am working on, and I would like some, or all of it reviewed, how can I ask for this review on Code Review in a way that produces the best possible value? This is not about a question being on-topic, or off topic. Rather, this is about making on-topic questions great qu...

discussion faq asking-questions

Mast

Make it work first, make it shiny later.

2

Mat's Mug

^ if you feel like reading

user54089

14:25

Reading now

Mat's Mug

@user54089 we get a bunch of those. you'd be surprised how many people carve through mountains with a spoon

3

Mast

♪ Diggy Diggy Hole

►

Vogel612

@Mat'sMug so I was carving tunnels in minecraft with stone axes ... does that count?

Mat's Mug

if you could have used TNT instead, ..yeah ;-)

Vogel612

14:40

iron axes would've already been significantly faster...

but I didn't get to find iron ore because I was busy not mining

Dan Pantry

/r/minecraft is leaking

Mast

@Mat'sMug TNT doesn't give you the materials. But it would definitely be faster. Less accurate though.

I vaguely remember destroying an underground complex by accidentally blowing away a wall which kept the lava at bay...

And that is why you shouldn't use unsafe methods.

2

Not in Minecraft, not in code.

2

Phrancis

Monking!

user54089

After asking in that db room I feel even worse than I did before lmfao

Now I really have no idea what I am doing

arda

@Phrancis monking

Duga

14:55

If this is working code, it seems like a codereview.stackexchange.com question. — jaco0646 41 secs ago

Mat's Mug

@user54089 hey, why not just put your DDL script up for review on CR and see what you get?

user54089

DDL?

Vogel612

Data Definition Language

CREATE TABLE and related

there's DDL and DML (and if you will DQL, though that's basically the same)

Phrancis

It'd be interesting maybe to see the DDL along with the Java side by side

Vogel612

definition, manipulation and querying

and together it's SQL

Phrancis

14:58

e.g., this table [DDL code] is used by this [Java code]" etc.

user54089

I'm not sure if my question will be appropriate but I'll try to put something together

Vogel612

if it's not on-topic here, we'll still be friendly, you know?

Phrancis

We will be happy to help you, in case it needs any improvements

Vogel612

it's not like we're gonna rip your head off like SO would

user54089

I appreciate it guys

I'll write it up now

N3buchadnezzar

15:00

@Mast Why not link to the original?

Mat's Mug

15:11

@Zak someone just made me discard my draft.

@LórántCsabaMihály I will appreciate that you remain civil please. I'm only telling you what this site is for, and warning you that you may not receive an answer that solves your OOM issue, because fixing bugs is not what we do here, that's Stack Overflow's territory - the only reason your question isn't closed yet is specifically because it appears to work with smaller data sets. Please read how to get the best value of Code Review and our help center to know what we're about. Stack Overflow users don't always know what's OK on CR. — Mat's Mug ♦ 7 mins ago

Vogel612

@Mat'sMug hmm.. meh

I don't see the need to call out "incivility" or something yet

Mat's Mug

> If you can code then help and stop saying that this topic is irrelevant.

aka "fix my bug or knock it off"

Tushar

Why are all talking with Mat's Mug, talk with Mat

Hosch250

Hit them with an exploding bear trap.

Vogel612

@Mat'sMug that's not incivil just yet. It's just a jerkass move when considering the on-topic guidelines of the site

Dan Pantry

15:15

op is being rude but not uncivil, yet

Vogel612

@Tushar handles are just that. handles. Don't interpret more into them than there is :)

Dan Pantry

IMO

Vogel612

^^ that :)

Phrancis

Mat has been AFK for a few years, his mug has taken over

Mat's Mug

lol

Vogel612

15:16

@Tushar I had a hard time convincing some people to not interpret things into my handle (calling me Mr Vogel had taken over)

Dan Pantry

Mrs Vogel, obviously

Vogel612

actually neither

Tushar

:D

Vogel612

because Vogel is simply not related to my "legal name" in any way whatsoever

Dan Pantry

4

Vogel612

15:19

@DanPantry there was a joke there?

Dan Pantry

just yanking your chain Mr Vogel <3

N3buchadnezzar

Wondering why you have Mr Vogel chained up.

The 29th Saltshaker

May have to pass on the question -- as I am writing this I am very quickly realizing it is more complex than I thought

If anyone wants some extra money though I'd be willing to pay to talk someone's ear off

Dan Pantry

How complex is "more complex"?

We have some very lengthy, complex questions here.

One of our biggest zombies was recently killed by @syb0rg and that had to do with neural networks

The 29th Saltshaker

Because at the level of advice I am looking for I am basically asking for app consulting really

Phrancis

15:23

@The29thSaltshaker You have the option of breaking it down into a few smaller, more focused questions, if you feel that would be better

Dan Pantry

^^^^

The 29th Saltshaker

It's hard to ask more focused questions when they all seem to rely on understanding the broad picture, which in itself takes a lot of text to explain

Mast

@N3buchadnezzar Because the newer sounds much better.

N3buchadnezzar

Eh, pffft

Dan Pantry

@The29thSaltshaker it's your call, but there's no harm in trying to break it down anyway

you'll likely want to do that if you do get a consultant in

Mast

15:29

@Tushar His mug is much more interesting to talk with.

The 29th Saltshaker

I have way too many questions, yeah it just won't work

Tushar

that I can see

Mat's Mug

@Mast I'm still lurking

;p

Mast

We got mugs, monkeys, ducks, birds, blue canaries, fusion reactors, what else?

Oh, a dog.

Hosch250

Dogs.

Captain Obvious

15:32

0

Q: Database abstraction layer for multiple providers

There's been a lot of questions lately about database provider and repository design especially without entity framework or alike. I thought I try myself to create a reusable framework for this kind of stuff. So let's begin ;-) TL;DR - There's an example at the bottom Core The main part of ...

c# database generics framework fluent-interface

Mast

Snakes, sponges, megaman, etc.

Phrancis

A fedora... @N3buchadnezzar :p

N3buchadnezzar

Huh?

Mat's Mug

aye.. makes me miss the facepalming thinker statue at times lol

Phrancis

Me too

Donald.McLean

15:35

Greetings, Programs.

Mat's Mug

greetings!

Olzhas Zhumabek

greetings

arda

greetings

Dan Pantry

greetings

Tushar

greeting cards

he he, train wreck

Duga

15:40

possible answer invalidation by MaMu on question by MaMu: codereview.stackexchange.com/posts/138821/revisions

Olzhas Zhumabek

is storing product info (name, price, etc) in extensible way in an inventory database is only storing as a file with some particular grammar? I had headaches with it 2 years ago, and I still have now :(

Phrancis

@OlzhasZhumabek Are you asking if it is better to store in a database or a file of some kind?

Olzhas Zhumabek

@Phrancis, in fact I just found that they differ

Hosch250

You really should create a normalized DB.

Phrancis

Personal opinion, if you are storing only products (or only one type of "thing") a file would be way simpler

Hosch250

15:45

A little harder to break.

Phrancis

Something like a JSON or XML is flexible enough to give you plenty of fields that each product may have

Hosch250

I can open any file in Notepad and edit it. Doing that for a DB file (if you even knew where to look for the physical file) would break the DB.

Which would force you to go through and find the bad data and fix it.

Donald.McLean

@Duga No, that's ok.

Hosch250

Listen to Phrancis, though--he's the DB expert.

Phrancis

Databases are great if you are storing a large number of multiple different types of things and need to keep them in a centralized structured entity, but they are a LOT more work to setup and to maintain

Olzhas Zhumabek

15:48

@Phrancis, I actually nailed down some very easy grammar for it. XML seems to be too big for that. Though seems like parsers should never have hard faults

Zak

@Mat'sMug Meh. I'd already sank an hour into my answer by that point, wasn't going to discard it. Hopefully the OP learns a thing or two.

Didn't get as far as refactoring though. Pity that.

Phrancis

@OlzhasZhumabek Maybe JSON would be better then? It's usually simpler to handle, and supports a similar hierarchy than XML but without most of the pains that XML comes with

Then again, depending on the type of products themselves, a flat file like CSV or TSV might be fine.

Dan Pantry

if it isn't going to change a lot just use a flat-file json file

if it is going to... database is the way forward

Phrancis

^

Captain Obvious

1

Q: move files to specific directory, according to file extension

I've written the following script import shutil,os,glob # setting up destination folders PYDEST = "/home/ubuntu/python-scripts/python-test" TEXTDEST = "/home/ubuntu/python-scripts/texte-test" def move8files(pwd): PWD = pwd + "*" for files in glob.glob(PWD): if files.endswith(...

python performance beginner

Phrancis

15:53

We should make a collab article / blog post on options for persisting data, it seems to come up very often, and there are lots of variables to consider to help someone choose the best one

Mat's Mug

@Zak the outer loop body should be extracted into their own procedures, simple as that - make a procedure that processes one sheet. then discover wtf those "counters" mean and what these inner loops actually do, and extract further procedures.

Olzhas Zhumabek

@Phrancis, I'm not sure about it yet. Elements should have tags on them. I will make some dev tool for the inventory so that it is straightforward to put things into the file. I believe that it will cast away most of the parsing headache

for now, it is only an idea. The first try to implement something more than a stand alone class or a function

Zak

@Mat'sMug It wasn't too hard to figure out once I'd got my head around what the data sheets look like.

Phrancis

@OlzhasZhumabek Something like

{ "product" { "id" : 42, "name" : "Some Product", "cost" : 12.99, "colors" : [ "red", "blue", "yellow" ], "dateAdded" : "2016-08-16" } }

maybe?

Zak

Just suppliers across the top, services down the left and price data in a grid.

Hosch250

15:57

@OlzhasZhumabek Sounds like XML would be good.

Olzhas Zhumabek

@Phrancis: seems nice! I will add that when will implement basic skeleton.

Hosch250

Or something.

Olzhas Zhumabek

@Hosch250, well, it seems like there is very little difference if it comes to extensibility.

Hosch250

Yeah, JSON is good too.

Phrancis

@OlzhasZhumabek OK, a JSON format like that is widely supported by pretty much all programming languages/platforms so it's easy to work with

Hosch250

15:59

I know XML better, so ;)

I do agree that JSON is more readable, though.

Transcript for