The h Bar

enumaris

8:00 PM

I have not found a V, Z or O yet...

danielunderwood

Also I've overthought things that ended up getting tossed out anyway

Anonymous

But then, brute force is something I try to avoid even in the first version, lol

Anonymous

Last week I had a file with 5 million nodes (in binary) and was going to run Kruskal's on it by sorting the edges

Anonymous

Some divine intervention stopped me

danielunderwood

I think there are multiple version of crappy as well...looking at you if ... if ... for ... while ... switch

Anonymous

8:03 PM

I'm getting interested in possible ways to speed up Python code these days

Anonymous

It's much slower compared to C but has a lot more standard library features which I like

ACuriousMind

@enumaris Sadly, it is even lacking in many software engineers :P

bolbteppa

Stru not Stur :(

enumaris

lol

danielunderwood

@Blue you can write native extensions in C (or C++ with boost and such)

enumaris

8:04 PM

@Blue are you using the appropriate libraries?

Anonymous

@danielunderwood Yeah, that's one. Also PyPy has some of them (C implementations of some of the standard libraries)

enumaris

using numpy for arrays would be much faster than using lists...

Anonymous

@enumaris Depends on the context, really

Anonymous

In some cases it might be slower

Anonymous

I had proof, lemme find

enumaris

8:05 PM

oh dang, I just realized, it's 6 days to simply correct all the spelling, and then another 30 hours or so to run the NLP pipeline on top of that

@Blue fringe cases probably...

but generally there's a library to do what you need :D

I mean even massive deep learning projects can be done on python with speed...I'm sure there's something out there to speed up the vast majority of use cases someone might be interested in...

Anonymous

17

Q: Why is numpy.array so slow?

I am baffled by this def main(): for i in xrange(2560000): a = [0.0, 0.0, 0.0] main() $ time python test.py real 0m0.793s Let's now see with numpy: import numpy def main(): for i in xrange(2560000): a = numpy.array([0.0, 0.0, 0.0]) main() $ time python test.p...

Anonymous

@enumaris Indeed, there are tons of libraries. It takes years to know be acquainted with a majority of them

danielunderwood

@Blue if you want something really interesting, you can look at SWIG. It takes C++ code and generates bindings for various languages. It was an interesting experience working with it

And it came from physics!

Well physicists

enumaris

@Blue well the answer there sort of explains it...I'm not advocating you creating millions of tiny arrays in numpy lol

Anonymous

There's this book by Hellman which I'm thinking of buying

Anonymous

8:10 PM

amazon.com/gp/product/0134291050/ref=dbs_a_def_rwt_bibl_vppi_i0

enumaris

I guess I should write my code so it doesn't take so long to finish...hmmm

Anonymous

@danielunderwood Ah, never heard of it

Anonymous

I'll see sometime. Thanks

enumaris

now I have to think of how best to write it lol

danielunderwood

I only used it since I needed to have code that was accessible in multiple languages, but it was a neat thing

Anonymous

8:11 PM

@enumaris I faced such a situation last month where I had to deal with a few hundred thousand 4 element arrays

Anonymous

Tried numpy and failed

enumaris

can't you put them into 1 giant array?

Anonymous

@enumaris A giant numpy array?

enumaris

yeah

if they aren't jagged, it should be ok right :DC

welp, at least putting the numerical fudge factor of 1e-8 doesn't seem to have affected results while getting rid of divide by 0 errors

Anonymous

Iirc I didn't do that because the it takes up a lot more memory (and time is spent in copying to a large array) than the lists. Also I wasn't doing any heavy operation on them. Just addition/subtraction . So lists worked pretty well ~1 sec

Anonymous

8:17 PM

While the small numpy arrays took 12 secs

enumaris

¯_(ツ)_/¯

ACuriousMind

@Blue A giant array certainly does not take up more memory than a bunch of lists holding the same amount of data.

enumaris

if your array is sparse you can try the sparse matrices in scipy

Anonymous

@ACuriousMind A numpy array and a list takes up the same memory space?

enumaris

but you can't do all the same operations on a sparse matrix as you can do on a general numpy array

danielunderwood

8:19 PM

I'd imagine a python list is a linked list, so it also has to store pointers while a numpy array would just allocate a chunk of memory

ACuriousMind

@Blue I'm reasonably sure there's not much of a difference.

@danielunderwood It isn't. A python list is much closer to what one would ordinarily call a dynamic array than a linked list

Note that you can't call "next" or "previous" type operators on the list - if it were linked, one would expect that

Anonymous

> NumPy's arrays are more compact than Python lists -- a list of lists as you describe, in Python, would take at least 20 MB or so, while a NumPy 3D array with single-precision floats in the cells would fit in 4 MB. Access in reading and writing items is also faster with NumPy.

Anonymous

Numpy arrays apparently take lesser memory than normal lists

Anonymous

So the slow-down shouldn't have occured due to that

danielunderwood

Ahh yeah it looks more like they allocate a chunk and resize it when needed

github.com/python/cpython/blob/master/Objects/listobject.c

Anonymous

8:22 PM

I don't know why using numpy slowed it that much, hmm

Anonymous

12x more than lists

Anonymous

15

Q: why is converting a long 2D list to numpy array so slow?

I have a long list of xy coordinates, and would like to convert it into numpy array. >>> import numpy as np >>> xy = np.random.rand(1000000, 2).tolist() The obvious way would be: >>> a = np.array(xy) # Very slow... However, the above code is unreasonably slow. Interestingly, to transpose th...

python performance numpy

ACuriousMind

The main reason a Python list can't ever be as memory efficient as a numpy array is that it isn't limited to a single type of object to contain, so it has to basically store a list of pointers to the objects it contains. If you know that you are gonna have an array of 4 byte integers, you just need the pointer to the start and you're good.

enumaris

I can only talk from a end user perspective

I don't bother to get into the nitty gritty details of how python implements its data structures and functions and stuff like that

Anonymous

@ACuriousMind But on the other hand accessing an item from a numpy array creates new python objects each time

Anonymous

8:26 PM

Which makes iterating through them pretty slow

ACuriousMind

@enumaris Technically, "Python" doesn't implement them. There are various implementations and as long as their behaviour conforms to the standard, they're free to do whatever under the hood

enumaris

u mean the OS does that?

Anonymous

I suppose the slow-down I faced was due to that reason and not because it used more memory than lists

enumaris

or the machine code itself...

danielunderwood

See I find those details a bit interesting

Well the python interpreter. Normally that would be cpython

But there are also various other implementations that should run python files in the same way, but with potentially different implementations under the hood

enumaris

8:28 PM

meh, already too detailed for me to care much lol

danielunderwood

So you don't think it would be neat to write your own interpreter? That's crazy!

ACuriousMind

@enumaris No, I mean that you can get different interpreters (CPython, Jython, PyPy, ...) and there's no guarantee that they implement lists the same way from a low-level viewpoint

enumaris

mmm

do they all deal with GIL the same way

ACuriousMind

@enumaris No. E.g. Jython has no lock at all.

enumaris

hmmm...

ACuriousMind

8:30 PM

See wiki.python.org/moin/GlobalInterpreterLock

Anonymous

I think pandas offers some advantages over numpy while dealing with structured data

enumaris

so multithreaded code will perform differently given different interpreters?

danielunderwood

Is it worthwhile to use one of the other interpreters? I've looked at them, but never actually tried them

Anonymous

Although I never used it much

ACuriousMind

@enumaris Certainly, yes

@danielunderwood Only if you have a performance bottleneck that one of them can alleviate, imo

enumaris

8:31 PM

pandas is kinda like SQL for python imo...but a bit more flexible

@danielunderwood write some multithreaded code, run them through different interpreters and get back to us on performance

ACuriousMind

I've never written performance-critical code in Python, so I have no idea about how that actually works in practice

danielunderwood

@enumaris well I do multi-threading with multiple processes so I don't have to worry about GIL

enumaris

a couple months ago I wrote code to segment the analysis...now I can't remember if it's theoretically better to not segment the analysis and I had to segment it only because of some requirements for periodic updates to the analysis...

I don't want to go back to look at my months old code now...

@danielunderwood you and everybody else who writes in python apparently

ACuriousMind

@enumaris That's why you should write elegant code.

Anonymous

I have no idea how multi-threading works in python

enumaris

8:35 PM

@ACuriousMind nah, future me is a douche, I hate him so I write ugly code for him to worry about

Anonymous

Where from did you people learn it?

enumaris

@Blue multi-threading in python is...fun...

danielunderwood

@enumaris Just learn to think in queues and you're good to go. That does take some thinking though. This guide changed how I thought about parallel processing zguide.zeromq.org/page:all

enumaris

I implemented a parallel process to parallelize loading of training batches and training on those batches

was pretty successful, but it's super annoying

danielunderwood

The wonderful thing is once you can think in queues, you're just limited by the queue implementation. If you have a queue that can work over a network, you can put together a cluster of workers

enumaris

8:37 PM

I've had to deal with multiprocessing only superficially

ACuriousMind

::whispers:: purely functional programs are trivially parallelizable.

danielunderwood

I'm still trying to twist my brain to work with functional programming

But parallel map/foreach are the best

@enumaris if you think in terms of maps, Pool.map() can be quite powerful

Danu

@EmilioPisanty Thanks for reminding me about torus knots; I now invested some time in improving my previous picture. This is the result:

A lot nicer.

(and only took me like half an hour to an hour! :D)

2 hours ago, by Danu

Old one

I based it on your picture but drew the knot "by hand" using TikZ' Bezier curves

Anonymous

In its C incarnation, a Python list is basically an array of PyObject pointers. To iterate over it, you increase a pointer, dereference it and you are done, you already have a fully qualified Python object. NumPy arrays are arrays of basic types, in your example probably C int. Getting the value in its int form is about as fast as getting the Python object from the list. But you then have to convert it into a Python object, probably with a call to PyLong_FromLong or something similar, hence the extra overhead. — Jaime Mar 26 '15 at 15:55

Anonymous

This is interesting

danielunderwood

8:49 PM

Wow I struggle to get anything barely passable out of TikZ

Danu

@danielunderwood practice makes perfect ^^

Anonymous

I can only make quantum circuits with TikZ

Anonymous

Couldn't get about making any mathematical diagrams like that tho

danielunderwood

I think my main problem is that I don't really know the variable system, so I end up specifying coordinates manually and bumping them when something doesn't look right

Anonymous

There are ways to measure the exact coordinates beforehand

ACuriousMind

8:52 PM

@danielunderwood Ah, you're using TikZ as a YDSWYG (You Don't See What You Get) editor.

Danu

lol

enumaris

YSWYG

YGWYS

YGWYW

ACuriousMind

Stop using your spelling correction network, it doesn't work on acronyms :P

Danu

Man, I love making TikZ pictures. So much fun

and good memories of taking topology

danielunderwood

I don't have to use it very often, so it's a real struggle when I do

Anonymous

8:57 PM

171

Q: What are the various units (ex, em, in, pt, bp, dd, pc) expressed in mm?

How can one determine the length of the different units (measured 1ex, 1em, 1in, 1pt, 1bp, 1dd, 1pc) in mm?

conversion unit-of-measure

enumaris

@ACuriousMind lol, yeah it does get caught up by acronyms...

Anonymous

This is a very helpful thread ^

ACuriousMind

@Blue Sure, now I'll never forget again that $1\,\mathrm{cm} = 0.99985\,\mathrm{cm}$!

More Anonymous

Hey Curious get a chance to read my latest question?

I'm quite happy it;s got a net positive score so far

Anonymous

@ACuriousMind lol

enumaris

9:00 PM

cancel cm on both sides

$1=0.99985$

Anonymous

More Anonymous

Also who keeps deleting my comment " Reason for downvote?"

ACuriousMind

@MoreAnonymous I've had a long and tiring day and not really up for technical arguments right now

More Anonymous

ah ohk ... thats fine

maybe another day

Anonymous

@danielunderwood To avoid the coordinate hassle, I usually first draw a background coordinate system in TikZ using mm measurements and overlap everything over that

enumaris

9:06 PM

just get a UI team to draw your graphs...

Anonymous

@enumaris My annual salary is negative.

enumaris

cus you're still a student?

o.o

Anonymous

I can totally imagine my parents coming up one day and saying: return all the money we wasted on feeding you :P

enumaris

lol

you can ask them if you can return the food

throws up

3

hmmm...

Anonymous

@enumaris Some people have the ability to throw up at will ;)

Anonymous

9:18 PM

I had an uncle who could do that

More Anonymous

ewwww

Anonymous

And I can't throw up even if I'm poisoned (most probably)

Anonymous

I don't remember throwing up in the last 14 years at least

enumaris

o.o

Anonymous

@enumaris That reminds me: Is it a common norm in the US to kick out kids to earn for themselves after 18? I don't know whether it is a running joke or actually something real

danielunderwood

9:30 PM

I think it's less of a thing now, but most of the people I know just went to college

Evidently my parents were out once they turned 18

enumaris

@Blue it does happen, but I think there's a lot of parents in the US who support their children through college or later

danielunderwood

It does seem like a fair number of my college friends moved back home after graduating

enumaris

Certainly there's also a lot of kids going to college on their own work and effort

and then a lot of kids who don't go to college and go straight to a job

Anonymous

I see. It seems there's much less taboo associated with working at a burger joint or pizza shop or something like there than here

Anonymous

That actually sounds like some good experience which everyone should gain once in their lifetime (like working as a waiter, etc)

enumaris

9:32 PM

The US is quite diverse so...

it's hard to pinpoint

I mostly have a view as a S. Cal person

danielunderwood

@Blue yeah I think it's a good experience. My parents made me get a job in high school in order to have my license and I think I learned a fair bit from it

Anonymous

@enumaris I presume they're not from the well-to-do families?

enumaris

Also as an Asian, living with parents as an adult is not nearly as..."loserish"?...as it might be for someone of a different cultural background

A lot of Chinese people don't move out until they get married

Anonymous

@danielunderwood Nice! It's pretty much a taboo here if you are from a middle-class family and do work like that. Funny how cultures differ around the world

Anonymous

Parents here mostly support you till you get a job after graduating college

Anonymous

9:35 PM

That's good in one way and harmful in another

danielunderwood

@Blue I think it varies a lot here. There are families that are well off and spoil their kids, but it also seems like a number of families are well off and make their kids work

enumaris

alright, I think I've segmented out the computing...now I can process the comments in batches

save the result to a table

then draw from the table again to do further processing

mmm...

danielunderwood

Do a lot of jobs get posted and never hired for? I see some companies have job postings that are like 6 months old, which seems odd to me

enumaris

dunno

danielunderwood

ooh I just saw one from 2017

enumaris

9:38 PM

I don't work in HR

I have 177 million words to spell check lol

one character at a time

Anonymous

@danielunderwood Or maybe they never update their webpage after getting an employee for that position :P

Anonymous

Until they require new ones

enumaris

I should vectorize the process...but that's a lot more work...

danielunderwood

@Blue I kind of wonder if some of them were meant to hire a specific person but that didn't work out

lılostafa

@danielunderwood Parents win suit to kick 30-year-old deadbeat son out of their house in NYC

it certainly seems it varies a lot there

Anonymous

9:41 PM

@lılostafa The title made me chuckle XD

enumaris

There's a lot of different cultures in America as well

I'm sure the different cultures handle things differently...

danielunderwood

Well I feel there's a difference in kicking someone out when they turn 18 and when they're 30

enumaris

But there's a lot of people who went out, did something, failed, and then went back to their parents

some might find that acceptable...like the parents being a "back up plan"...some might be like...gtfo

Lozansky

Hmm

Anonymous

There's also a difference between not being able to find employment and not willing to find employment (because you're already too comfortable) :P

enumaris

9:45 PM

yeah..

Anonymous

That guy in the news article seems to be in the latter category

danielunderwood

Yeah I think there's a difference in working on something that may fail and sitting around all day watching netflix. I've seen people in both categories

Anonymous

>

They later offered him $1,100 to “find a place to stay” and gave him parental advice including, “organize the things you need for work and to manage an apartment” and “sell the other things you have that have any significant value.”

They even implored him to start making money. “There are jobs available even for those with a poor work history like you. Get one — you have to work!” the Rotondos wrote.

enumaris

I'd like to sit around all day and just...chill...

danielunderwood

I just saw a job posting that it wanted FORTRAN experience...nope

enumaris

9:46 PM

play some video games, maybe read some books...

@danielunderwood I have FORTRAN experience

(but I don't like FORTRAN)

danielunderwood

But do you want a job that wants you to have FORTRAN experience?

enumaris

probably not lol

danielunderwood

It also said C++, but the FORTRAN made me run away

enumaris

and yet I put it in my skills

ACuriousMind

@danielunderwood Don't let KyleKanos hear that :P

enumaris

9:48 PM

@ACuriousMind you forgot the @

ACuriousMind

(that's easy because he isn't around all that often anymore... :/)

danielunderwood

Does he like FORTRAN?

ACuriousMind

@enumaris Doesn't work anyway because he's not been in the room in the last few days.

danielunderwood

shudder

enumaris

ah...

danielunderwood

9:49 PM

Though admittedly I haven't actually worked with it since my initial attempt to learn programming in high school that did not go well

enumaris

it's not really all that horrible...unless you're in FORTRAN 77

death to FORTRAN 77

danielunderwood

Luckily my research advisor preferred working in essentially any other langauge

enumaris

But it's pretty basic...you gotta declare all your variables up front, including loop variables...

danielunderwood

You have to do that in c95 too I think

Actually nevermind

enumaris

my block of code just declared all the single letters up front lol

and then you can use them to loop

danielunderwood

9:51 PM

I think you're saying you have to declare all variables at the beginning of the file. In c95 you just have to declare before the loop

enumaris

Yeah, beginning of the file

before the executables

pretty sure

danielunderwood

hah fortran 66

enumaris

fortran77 counted the number of spaces

cus continuation characters go into the 6th slot

danielunderwood

Yeah I just saw that...it cares about column position

enumaris

indeed

your lines can only be X long

danielunderwood

9:52 PM

I'd call that a con

enumaris

6th slot is reserved for continuation character

all actual code must start on 7th

danielunderwood

Though it probably results in better things than some c code I've seen

enumaris

if you have code that got formatted weird you'll have to go back and count spaces

like when my prof published his fortran77 code in PDF format

lolwtf?

and I copy pasted it from PDF to a text doc

and just got overrun with syntax errors

Anonymous

@enumaris ....in a paper?

enumaris

yep

as part of the appendix

in a paper he published in the 80's I think

Anonymous

9:54 PM

good old days (wasn't alive then, lol)

danielunderwood

Well I'm surprised they had PDFs in the 80s

Emilio Pisanty

@Danu that's not bad

enumaris

I dunno if he published it originally in some other format

Anonymous

@danielunderwood I'm guessing it was converted to PDF later

enumaris

and it got converted to PDf

Anonymous

9:56 PM

PDF was developed after 1990

Emilio Pisanty

I'm not too sold on the three-dimensionality of the thing, but I'm not sure that's possible on tikz

Anonymous

I wonder how it might have felt to write papers by hand

Anonymous

In the earlier days

Emilio Pisanty

@Blue laborious as hell

enumaris

doth professor knoweth your keming is poor?

Anonymous

10:02 PM

Emilio Pisanty

@Blue ugh

danielunderwood

@Blue you know exams with essay questions that make your hand cramp? I'd imagine like that, but for days/weeks/months

I kind of wish we had essay questions/homework in physics. We only had stuff like that in lab courses

Anonymous

So the world-changing paper had so many strikethroughs? :D

Anonymous

@danielunderwood I can imagine

Anonymous

I admire the drawings

enumaris

10:06 PM

fun beans

Anonymous

Some of them could really draw diagrams very neatly, by hand

danielunderwood

I took a "graphical communication" class that was mainly about engineering drawings. Drawing them by hand was interesting

Anonymous

Oh the nightmares....engineering drawing

Anonymous

We had a whole semester of that....and it does make you cry

danielunderwood

It wasn't too bad. I actually took it after I did a bunch of engineering drawings for my lab

Inventor > SolidWorks though

Working with Blender is an entirely different skill for me though. I can't seem to do anything useful there

Anonymous

10:12 PM

Anonymous

We had to draw stuff similar to this ^. Used to take me whole 2 days

Anonymous

If you have the patience that's fair enough, but I don't :P

danielunderwood

Yeah the ones we had to do in the course were nowhere near that bad. I don't remember if we even did assemblies in that course

Assembly constraints were a lot more of a pain to me than the parts themselves. Though I never did any crazy parts

enumaris

I drew in autocad in highschool

for my architecture class

man, I wish I could just go home when I finished my work for the day

instead I'm just sitting here with not much to do lol

Anonymous

We have CAD classes next semester I think. I didn't understand why they insisted on EE students having to draw those by hand. Made no sense to me

danielunderwood

10:18 PM

Do you have classes for PCB drawing?

Anonymous

@danielunderwood Not yet. But afaik they're gonna teach us some software for that

enumaris

fun beans

Anonymous

The 1st year professors were a bit crazy, so

Anonymous

One problem with these software meant for EEs (at least the free ones) is that they're usually not up to date. It feels like you're using Windows XP

danielunderwood

Yeah I haven't had a ton of fun the times I've tried to use any software for circuits

Though I think there are some that are pretty expensive that are supposedly nicer

enumaris

10:27 PM

hmmm

Physics Meta

10:47 PM

0

Q: Tag for *Personal Theory*?

We get a fair number of questions that are simply people "asking" about their personal theory. These get tagged in all sorts of random ways, typically. I've just seen one suggesting a black hole powers every Sun and it's tagged "experimental physics". I'm suggesting we have a dedicated tag for...

discussion

Transcript for