@AaronBertrand not been there but I've been to Russia

that was a long flight

@bluefeet sochi?

wow, what an intro

@Kermit all over. I was in moscow, st. petersburg, Minsk, and a bunch of other places. I went for 3 weeks in highschool

@bluefeet that's bad ass! during cold war?

@Kermit I'm not that old!

geez

lol

jk

I did get to see Lenin when he was still in his tomb, that was weird seeing someone who had been dead for a long time

that's neat

this is why i hate this guy

@Kermit flagged the answer

@swasheck LOL you're awesome

i'll see you all in a few weeks :)

@JNK or South, you can meet @Lamak.

@swasheck where you going

@Kermit anticipating my ban :)

@swasheck lol!

> not an answer – swasheck 6 mins ago declined - a moderator reviewed your flag, but found no evidence to support it

@swasheck i wonder who denied it

flagged again

@swasheck It's easier to delete. Or just close and delete the question.

It's nonsense after all.

@ypercube yeah

man, this site feels empty

> does not address the question at hand – swasheck 6 mins ago declined - flags should not be used to indicate technical inaccuracies, or an altogether wrong answer

@swasheck lol, try again for the "altogether wrong answer"

@Max It's better to have 10 views here than 10k views in SO.

how so?

answers here will be correct

@Max we're like the SEAL team 6

XD

@Max StackOverflow is a bunch of children in a sandbox throwing their poop at each other

As billinkc said, you may get a good answer at SO, but we've seen cases where a question gets lots of nonsense replies.

i'd prefer the poop over nothing in this case, i think

@Max just from quickly reading your question, it seems as though it's an application issue, not db

read deeper

if you can spare the time

Here, it may take some time, but the answer will be worth something.

@Max i have to go stick a screw driver into a wall socket. if i survive, i'll be back to read it

do questions that receive no traffic initially get answered eventually tho?

in that case, best of luck .d

don't get me wrong, i prefer a great answer after a few days over a bunch of instant terrible answers

Could it be that something is slagging your underlying storage from time to time resulting in your lower throughput?

can you elaborate/rephrase?

Only upside to asking on SO is that there are far more MySQL practitioners over there.

@Max Do you insert one row at a time or multiple rows per insert statement?

I read your question as I have an app that is feeding data at a constant rate but the time to complete an insert for the same sized object is consistent for most of the time but then it goes nuts

currently one at a time, but i get the same result with multiple rows

i wouldnt say nuts, but yes. insert speed varies from .16ms to .3ms

During the nut producing time, is it possible that something else, perhaps on the host is contending for the disk and introducing your latency?

You mention you have many duplicates that get ignored. What percentage of the inserts is rejected?

Could it also be that during that time you've filled the current storage container and MySQL needs to allocate more disk space to a new data bucket and so those inserts are waiting for that housekeeping operation to complete

@ypercube 80% to 99% are duplicates usually

@billinkc i don't have control over the system the VM is running on, so i have no way of knowing, but that is the direction i have been leaning to

@billinkc by storage container do you mean a hardware data drive? or is it a mysql thing?

Try using ON DUPLICATE KEY UPDATE pk = pk; (not that I expect this to be the issue, just to be sure)

Other applications running, as billinkc suggested. Not well configured mysql settings. Low RAM. How much RAM does the VM have?

@Max I don't know diddly about MySQL or its storage but I assume based on other RDBMS systems that the data is initially stored in some bucket and once it gets so full, it either automatically grabs more disk space and creates buckets or an operator needs to proactively add them.

@ypercube it says 16G Mem and 8G Swap, i don't really know the difference

Maybe innodb/myisam doesn't work that way

@Max How many inserts are you sending to mysql per second?

@billinkc i think that doesn't fit the pattern of volatility, because the timeframes are too inconsistent for that.

@ypercube at current minimum insertion time i'm at 7k rows a second in myIsam or up to 23k with innodb

Do these data come from the web? Or are you trying to bulk load data?

i'm crawling a web API, but the input stream from the java app to the mysql database is consistently kept above 50k rows

That means that the rows are wide, do you store web pages?

hey, you are the guy that solved my last question, too! :)

the rows are not very wide, let me check row size quickly

i eman row length

mean** geez

Check this answer by Roando: dba.stackexchange.com/questions/58994/insert-take-too-much-time/…

It may not be 100% relevant but there are several mysql variables related to inserting and bulk inserts, different for myisam and innodb.

The bulk_insert_buffer_size for example has a default of 8MB which is too low.

So I haven't found a quick way to get the KB size per row, but the tables have 4,4,9,10 columns with mostly 10 digit integers

alright thank you, i'll check it out

Anyway, add the table definition in the question. Rolando is one of the mysql top users here, he many answer it - or ask you for clarifications.

should i also add my mysql config / part of it?

Yeah, if you can figure the relevant parts.

i can't >:D i guess i will wait then until someone asks for a specific var

alright, thanks for your input, i'll update the question later and see if the answer you linked will help :)

although i have to clarify one thing - while performance tuning would be great, i am currently not as concerned with that. rather, i would at first like to find the root of this phenomenon, before i take the next steps towards improvement.

@Kermit now you have me worried about your survival