Room for Bob and jokerdino

Feeds

11:42

jokerdino has unfrozen this room.

jokerdino

back to life

@Bob boom

Bob

@jokerdino Thanks

@jokerdino My GetData script needs optimisation.

I just tried to grab every single AU question

jokerdino

woah

Bob

Ate up like 200 MB RAM

jokerdino

once it is cached, would it go down?

Bob

11:44

Got to 459 before it crashed.

Not even sure why it crashed...

jokerdino

yeah, definitely needs to be thin

Bob

Might need to change it to a proper database, and to actually write to the DB

right now it's just storing it in memory and writing it out to a JSON flat text file

...processing is done on-request, which could be painful, too

jokerdino

hm wow

DB and stuff?

Bob

@jokerdino ?

jokerdino

getting bigger than i expected

Bob

11:46

Weird thing was it crashed with a "badstatusline"

Requesting page 459
<!--: spam
Content-Type: text/html

<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> -->
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> --> -->
</font> </font> </font> </script> </object> </blockquote> </pre>
</table> </table> </table> </table> </table> </font> </font> </font><p>A problem occurred in a Python script.
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/cgitb.py", line 271, in __call__
    self.handle((etype, evalue, etb))

Just saving the error in this room

jokerdino

ta

meh python has lost its shine

Bob

lol

ignore the first block

it's like..

error creating an error

jokerdino

good thing you didn't call it error-ception.

i would have hated you for that

Bob

well

I forgot to create the errors directory

jokerdino

just tell me you know what you are doing.

Bob

11:48

nope! :P

jokerdino

because i dont know what is happening.

@Bob lol

do we have some kind of writeup on what this thing is?

Bob

I might have to shut down apache

it's eating up a chunk of RAM

jokerdino

kill it

Bob

@jokerdino What thing?

jokerdino

@Bob this project.

do we have the notes?

Bob

11:50

notes? :P

jokerdino

i think i had some on my mind but it's lost somewhere.

Bob

oh

somewhere in this chatroom

jokerdino

hm, better than nothing. phew

Bob

I'm a bit more worried about getting it to retrieve data, first

might restrict it to 2013 for testing

not sure why it crashed

Requesting page 435
Requesting page 436
Backing off for 10 seconds

LOL

Requesting page 445
Requesting page 446
Requesting page 447
Backing off for 10 seconds

jokerdino

that's a lot of questions.

Bob

11:52

That's like the first time I've ever handled a backoff

not crashed yet

jokerdino

sooon

Bob

Good to know my backoff works

It just did another two

Requesting page 544
Backing off for 10 seconds
Requesting page 545
Requesting page 546
Requesting page 547
Requesting page 548
Backing off for 10 seconds
Requesting page 549
Backing off for 10 seconds

SE API's not happy with me

and another one

jokerdino

one page = 15 questions

Bob

@jokerdino How many questions?

I'm grabbing 100 Qs per page

jokerdino

112,433

so, 1000 pages

Bob

11:54

fuck

jokerdino

1124.33 pages lol

Bob

RAM is already going into the burst RAM

300/500 used

jokerdino

on fire yet?

Bob

I'm running it on the VPS

much faster there

jokerdino

good lord, we are not into SO questions.

6000 questions / day

Bob

11:55

most of it is just networking

@jokerdino OH GOD NO

jokerdino

haha

Bob

I'd just use the data dump

and preanalyse the shit outta it

jokerdino

the dump itself is couple of GBs.

Bob

it'd be a static page

screw trying to parse that in real-time

page 826

lots of backoff requests

jokerdino

we are almost there.

Bob

11:56

350/500mb

I'll be so annoyed if it runs outta ram

the script is using 282MB

page 925

jokerdino

just another 100 pages

Bob

1001

backing off

WAIT

Requesting page 1000
Requesting page 1001
Backing off for 30 seconds

jokerdino

you are grabbing the question itself or just the meta data?

Bob

O.O

30 seconds. wow.

jokerdino

they must hate your script

you already have the API key or how?

Bob

11:58

my backoff times are taken from the SE response

...I think I triggered the flood protection

Requesting page 1001
Backing off for 30 seconds
Requesting page 1002
Backing off for 30 seconds
Requesting page 1003
Backing off for 30 seconds

Also, chat wants me to wait 20 secs...

jokerdino

they got your IP now?

Bob

I think it's just metadata

@jokerdino I'm running the script via the VPS

jokerdino

you are being monitored. ruuun

Bob

@jokerdino 100 pages at 30 secs each

jokerdino

that's gonna take a while

Bob

12:01

almost an hour. hope it doesn't crash

jokerdino

wow, that's a lot of time

Bob

or piss off the VPS hosts (I'm currently like 70% into my burst RAM)

@jokerdino how do I put it to sleep? :P

don't want to interrupt a request, either. that would be bad. actually, the backoffs use sleep, so that might be a safe time

jokerdino

if you stop half way, you start from start?

if yes, don't interrupt.

and by sleep, i mean backing off for longer.

but since you are already caught by the filter, no point really.

Bob

7

Q: Request Throttling Limits

What are permissible requests "speeds?" The API will cut you off if you make more than 30 requests over 5 seconds to any single endpoint. We consider this a breach of the API interface contract, so you won't get a nice error back. The exact manner in which this constraint is enforced can (and ...

@jokerdino yes

@jokerdino can't change it now

it's all in RAM and I don't have the time/will to change the backing storage right now

jokerdino

@Bob we come back in one hour then

@Bob seems like we might have exhausted the limits for the day

Bob

12:06

urk

There's a really nice API for C#

I might end up rewriting the data retrieval in C#

and just leave the serving code in Python

Or maybe even redo the whole thing as C#-as-CGI

jokerdino

you mean wrapper?

there is a wrapper in python.

check stack.py

Bob

@jokerdino not sure how it handles throttling :P

4

Q: API compliant request throttle implementations

Balancing compliance with Request Throttling Limits with maximum allowable throughput in our applications and libraries with any reliability requires self-throttling. This post is meant to be a repo for Stack Exchange API compliant request throttle implementations.

support dev-tip

you know what the worst part is?

jokerdino

what?

Bob

I didn't run it in screen. If my SSH session disconnects, it's gonna quit.

Have to leave my laptop on and hope the WiFi doesn't eff up.

jokerdino

have a nice day :S

Bob

12:22

@jokerdino It went to 31 seconds at some point :P

jokerdino

lol

Bob

lim{pages=>1113} backoff=>infinity

Bob

12:40

@jokerdino 1083 pages, 32 sec backoff

jokerdino

30 more pages. whoa

Bob

@jokerdino No guarantee my display scripts will work with that kind of crazy data :P

jokerdino

hm, we need to analyze and table the data

i think it is just number crunching at this point

Bob

@jokerdino I actually don't have the correct API filters

so some of hte info you wanted won't be there

this was just supposed to be a test

jokerdino

hm okay.

Bob

12:48

1100

but I have to go D:

it went to 33 seconds

@jokerdino DOH

I thought it was 1113

1124? still a ways to go

1107 right now

jokerdino

wut?

oh

Bob

pages

jokerdino

another 10 more

Bob

fark. I''m going

back in 1hr

jokerdino

oi

cya

Bob

12:54

@jokerdino hm?

cya

mem usage is like 50MB free

no cached :S

@jokerdino decided to hang around. 1120 now.

jokerdino

lol

another 10 pages?

Bob

1123!

the moment of truth!

this could very well crash on write :P

1124!

33 secs!

jokerdino

there

Bob

FUCK

Backing off for 33 seconds
Requesting page 1125
<!--: spam
Content-Type: text/html

<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> -->
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> --> -->
</font> </font> </font> </script> </object> </blockquote> </pre>
</table> </table> </table> </table> </table> </font> </font> </font><p>A problem occurred in a Python script.
<p> /var/www/cgi-bin/errors/tmpwlidtJ.html contains the description of this error.

real    75m10.224s
user    0m4.083s

jokerdino

the hell happened?

Bob

13:05

austats.tk/cgi-bin/tmpwlidtJ.html

jokerdino

404

Bob

austats.tk/error.html

Huh.

jokerdino

no access

Bob

@jokerdino try now

forgot to set read access

jokerdino

hm that's a lot of random crap

Bob

13:09

memoryerror lol

@jokerdino see the last line

so...

fuck.

jokerdino

lol

Bob

I'll rewrite it later

don't expect it anytime soon

jokerdino

nope

Bob

it has a copy operation

effectively doubles memory usage

and it was already using 70% by itself

140% with no swap makes nasty stuff happen

jokerdino

not enough memory to load the file

Bob

15:05

@jokerdino Ah, I'll come back to it later :P

Transcript for

Room for Bob and jokerdino