hmmm interesting, apparently b'this is a string' is a bytes-like object and not a string...

so that's what the b stood for in the print out lol

if I convert to string it brings in the b' wat

Time for the strings talk! :D

funky data

my string sub also failed

lol

This may be of some help to you stackoverflow.com/questions/11339955/…

I'd start with s.decode('utf-8') and go from there

You get to play Guess that Encoding!

I mean...I can read the strings that I print to screen

it just adds a b' to them

That means they're byte strings. You need to decode them I believe

Though string encoding still trips me up

hmmm...

I tried just doing str()

but that seems to make the b part of the string

for some reason

is it just...stringing the...uh...representation...

looks like I do need to decode...

wtf, how am I supposed to know the string encoding ._.

That's where our favorite game show Guess that Encoding comes in!

Really it's probably either ascii or utf8

I think text with emojis would be utf16...maybe

if I use the wrong encoding will it give me gibberish or mostly the right stuff with a few unknown errors

like...would it be easy for me to know I've used the wrong encoding or not...

Depends on the second argument to decode

I believe by default it will throw an error if there's an issue

hmmm...

There's also an ignore option that I think drops the character

@enumaris That depends entirely on how the encoding you used and the original one differ :P

It gets even more fun that str is different between python 2 and 3 and sometimes different libraries seem to do things differently

looks like utf-8 worked...

If you need to convert a python 2 to a python 3 program might as well throw it TO THE TRASHQ

lol

It's not that bad! I think the string handling is the worst

Oh right iterables and such are a bit annoying too

And map/filter/dict keys changed what they return

oh fun

the twitter dataset is not utf-8 encoded

Try utf-16. If it's the same one I had, it has emojis

One time I had some software engineering project

And part of the project was testing other projects

@danielunderwood Nothing about emojis requires UTF-8 or UTF-16. Both encodings can represent all unicode code points.

According to the documentation given

the twitter set should have scrubbed the emojis

And the documentation of the group we had to judge DIDN'T SPECIFY ENCODING

So I gave them a UTF 64 file to test

it specifically says "noemoticon" in the filename lol

Very devious

they should be using the emojis to generate the data labels

but I will try utf-16

@ACuriousMind ahh I thought they required utf-16 for some reason. Encoding always trips me up though

@Slereah 64?! You're a mad man!

damn

@danielunderwood If you have many symbols exceeding ASCII, it is usually better to use UTF-16 so that you don't get so many multiple-character code points, but it is not required

@danielunderwood i flagged that one as rude/condescending

@KyleKanos Rightly so.

Also: Hi, Kyle!

even though the question it's on is a "do this test for me!" that's pretty inappropriate

hi @ACuriousMind

I am surprised people think writing a title with all capitals is going to get them answers...

well it does make the post stand out

just not necessarily in a good way

nope, utf-16 failed

fun times

should be this dataset: kaggle.com/kazanova/sentiment140

now I just have to find out the encoding...

what are you testing?

not testing anything...just trying to open this file in pandas lol

what kind of file?

e.g., csv? xls?

csv

right now it's throwing an encoding error

even without specifying encoding?

invalid continuation byte for utf-8 and UTF-16 stream does not start with BOM for utf-16

yeah

I suppose I can just try latin and see if it just gets rid of non-latin characters lol

have you opened the csv in Excel/LibreOffice?

it's 1.6 million rows

Or with a text editor to check the encoding?

so I have not

i've opened a 7M row (and 20 col) data set in Excel...1.6M isn't that bad

yeah if I use 'latin' encoding it actually loads the dataset at least

I can try lol

he file contains more than 1,048,576 rows or 16,384 columns. To fix this problem, open the source file in a text editor such as Microsoft Word. Save the source file as several smaller files that conform to this row and column limit, and then open the smaller files in Microsoft Excel. If the source data cannot be opened in a text editor, try importing the data into Microsoft Access, and then exporting subsets of the data from Access to Excel.

yep...

oh you know what...i might've opened it in access, now that i read that

well, I think the latin encoding should suffice

you can always use powershell to get encoding type too, you know

@danielunderwood i had to fuck their day up!

its not quite a one-liner command, but just stick it in your ps1 profile & it becomes a one-liner

oh fun, this csv file has no headers

It was my job

pfft headers are over-rated

I just have to do it manually myself lol

frickin door dash won't let me log in...

ubereats it is then

uh oh...python frozen...

can't ctrl+c out of it...

JBP news rise/ fall patreon thedailybeast.com/… theguardian.com/technology/2018/may/14/…

@enumaris the fact that ctrl-c is "abort" in python is something I'm having to get used to

(copy as ctrl-insert is definitely not the norm)

i've been making a c++ parser in python

though to say I'm doing python is not entirely honest; sagemath is built on python, but I'm not having to worry about setting it up

i just load the interactive command line and start doing stuff

i'm trying to move defined functions from header to the source files

in an effort to reduce compile time

@enumaris That's when you pursue more aggressive methods

pkill python

(really more ps aux | grep python then kill the process)

@Semiclassical Ctrl+C is the standard interrupt sequence in unix terminals

makes sense

that seems to be the norm for python: "it's standard in linux"

Isn't ctrl+c the standard interrupt in powershell too?

Perhaps? I've never used powershell :P

git is about all I use powershell for, so I'm not terribly familiar with it

my main problem at this point is probably more to do with the math than the programming

the only way I know how to do the computation is to start from something relatively simple, go to something absurdly complicated, and extract something simple from that

it'd be really nice if I could circumvent that middle step

(with my hardest case so far, it takes about an hour to create the complicated intermediate object and then about another 15 minutes to reduce it to the final simple object)

Does anyone know the mathematical relationship between anode potential and the emission current in photoelectric emission?

As you increase anode potential in the positive direction you get more current which levels off until saturation

Is there a formula for this current with respect to anode potential

(for anyone who is somehow curious about the problem, it's essentially the same one posed here: math.stackexchange.com/questions/184955/…. the first approach is the one I've been doing)

@danielunderwood I just closed the window

Does that kill it?

seems so

I haven't checked task manager

yeah I don't see python in task manager

I tend to kill python in task manager

though only when ctrl-c isn't working

Has anyone looked at a Gaussian charge distribution? I mainly just get Gaussian surfaces when googling

I have not

this recruiter...so funky...asked me for a "couple minute conversation" to talk about a role...then straight up sends me a list of data science questions "he might ask" during the conversation...

feels like a study guide lol

alrighty...did some preprocessing...now I gotta figure out how to I want to use these...

Anyone know the difference between static and dynamic Coulomb blockade?

nope

Is it worth reading Knuth's books or are they going to be like reading MTW?

is the implication that reading MTW isn't worth it?

I read MTW to learn GR...

And your head didn't explode?

Like it has all the knowledge

Oh boy. Supposedly he created TeX to typeset his books?

I didn't read it front to back unfortunately

I read a lot of the chapters

I started reading it but then ended up just going to it for a reference

Yeah actually that's the article that I'm reading at the moment

living legend™ ... he used his own programming language in his books which think decreases their value substantially, although maybe almost nobody else is saying that... out loud... this is your fathers oldsmobile™ :P

@danielunderwood when were you in china? what for? exotic :P

I went for a few weeks as a high school trip. Evidently it was something partially sponsored by China's government for some reason or another

looks like this Data Science job is more about linear regressions and sampling...probably not for me then...

Isn't that the way a lot of non-research data science jobs are?

> Following Dr. Knuth’s doctrine helps to ward off moronry.

@danielunderwood so did you visit places etc

Well the first week or so was in Beijing and we were in dorms. That was mainly kind of a culture thing where we took Mandarin classes and did some sort of cultural activity during the evening. That also happened to be the time that there was an eclipse there, which was neat. There were also students from the US and UK staying in the dorms and in the courses, which was cool. Then we traveled around Shandong for another week or so

At some point we also went to the Great Wall of course

yeah was gonna ask about great wall... cool

> “He’s a maximum in the community. If you had an optimization function that was in some way a combination of warmth and depth, Don would be it.”

I think a good number of data science jobs are forward facing

they want to get into the more sophisticated ML models and the like

less about pure Linear Regression power

man...replacing values in a pandas dataframe is such a pain

ok, looks like the replace method is what I'm looking for lol

spent time trying to get apply to do it

There's also fill_na and such if you're just wanting to fill null values

And there's a bunch of stuff under DataFrame.str for strings

replace worked :D

now I'm trying to append all these dataframes together to make one giant training set

interesting, the append failed lol

ooh exploding dataframes are my favorite!

And a company sent out a survey of how happy I was with the recruitment process. Neat!

nice