@Brandon hey

Hey Adam

what's your thoughts

How long does it take to create the 100k entities?

(Shouldn't take all night, right?)

lol, taking forever.

started this on wed. it did about 2,000 entries then crashed

i'm now 8000 loops in, so about 40,000 total entries

about 1 second per loop

Okay so it seems like something else is performance bottlenecking you

I have an EF implementation that bulk inserts into a database that's been running for about a week. It has 20 billion records

have to call a random word generator a bunch of times

wow.

That's why I'm confused here. EF was not really designed to do that volume but you can beat it into submission.

How long does the actual insert take

Not the word generation but the physical go-to-my-db-and-insert operation

not sure, i have no timers. there are 22-26 data.table.create(), and data.SaveChanges() in each loop.

You can change the code though obviously right?

Like move things around in the loop?

yes.

Okay that's good then

do you mind if I have a look at the code in the method we're talking about?

Would you be willing to paste it to a private notepad on teh web?

i can. nothing in there sensitive.

Okay

(except my stupidity)

notepad.cc/eframework

password is Adam

(I think the link went through, yes?)

yea, you see the code?

Yes

Hmm, so it looks like you're just populating dummy data right?

yup. for load testing while we're in development.

Yeah see, this is why I hate autonumbers

we're doing cloud stuff so i want worst case scenario times.

Yeah I totally understand

K so the reason for the SaveChanges() calls is to make EF generate the autonumber, yes?

yes. clearly i have more SaveChanges than I need with the other data.

Yeah exactly... It wont help much but move the r = new Random(i) out to the initial declaration of r

seed it with DateTime.Now.Milliseconds if you really need a seed. Creating a new random every time isn't going to generate much more randomness

Another thing you could do, and again this won't solve too much but might help until the actual problem is resolved: Create a list outside of the loop that holds a number of Random word entries equal to the total number of iterations

so instead of generating a random word in the loop, they're pre-generated and just referenced as labr.FillerInfo = randomWords[i](for example)

thought about that, but seemed a little crazy to get 100,000 words or so in an array. you think that would really be faster?

Absolutely... you're saving the processing time of generating them on the fly

Yes you'll have a larger memory footprint but unless you're memory sensitive and using up a whole bunch of memory -- who cares?

I'm going to add some code to this and re-paste it into notepad.cc

ok.

brb

Sure

back

Alright, notepad.cc/eframework

You may need to change a few things I did it from memory... that should tell you (if you inspect the dictionary), how long stuff is taking

ok, cool

Yeah, give that a run and let me know where the bottleneck(s) are. I have to photocopy some stuff, hold on

ok, thanks i'll do that now

Don't let it run though the full 100k times obviously. Breakpoint it and have a look at that dictionary after a while

Interesting, Appts is the biggest

it is a 10 iteration loop

that is pretty cool. thanks for the code and tips.

Yeah no problem... I generally use that pattern when I'm trying to find why something is slow.

If you can, really try to break that stuff out of the loop.

Actually.... you can still

I already added the patients, that is why my loop starts at 10000 or 12000, that is how far it ran so far.

so, the loop just goes through the existing ID's without calling the DB for them

Yes I saw that part

(again, why I hate auto numbers)

how would I not use a loop?

But the object returned by _data.EMRVisits.Create() should just be an object you can instantiate though right?

Oh, you will use a loop

it'll just be very tight... Get in, update/create, get out

You should have a class called EMRVisits am I right?

yes.

Okay so you can instantiate that object outside the loop. Use EF's 'Attach' method

So create the whole thing, attach it (at which point EF will apply the ID) and then use that object's ID where you need it

so every non-database-related property for all those objects, EMRVisit, EMRAppts etc.. etc.. is created outside the loop 100k times. Then, inside the loop, you attach and get the ID then apply the ID to the existing objects where you need them

so, 2 loops then, one to build the objects and another to attach and save?

Yes

the second loop has the using block

don't call _data.xyz.Create() in the first loop

you're instantiating the objects with their class in the first loop, not by creating them in EG

EF

I'll brb, company lunch

ok

So, we already had moved the using statement inside of the for loop. That definitely helped the overall speed as time went on. Before it got bogged down a lot.

I have not rewritten it into 2 loops because there are more important things I'm working on and I think doing that right will take me a couple hours. However, I did tweak the appointments and lab results inner loop which appears to have helped some.