English Language & Usage: Multi-Layered Discourse Room

« first day (876 days earlier) ← previous day next day → last day (4342 days later) »

tchrist

23:14

If this is asking about bigram frequency distributions in English, it should be a lot more clearly worded. It sounds like one needs a programmer and a substantial corpus to check. However, I’m pretty sure this is a solved problem.

WendiKidd

@aăâ If you're working on learning English, you might also want to take a look at ell.stackexchange.com. It's a site similar to this one where you can find lots of answers to common problems non-native speakers have!

/shameless plug

tchrist

Also, questions relating to computational linguistics tend to be frowned on in ELU as being “too hard for regular people”, and get summarily shuffled off to Linguistics.SE.

@WendiKidd I actually think this one is on the high not the low end, at least, if it is to be professionally answered.

It’s a very very very easy programming problem, but you need a corpus to run off.

aăâ

So, Is that question OK to ask here? On this site. If not, I'll delete it.

tchrist

You need to clarify what you want.

I am just guessing.

WendiKidd

@tchrist Oh I agree! I just meant, learners in general might like to know ELL exists :) Sorry if I didn't make that clear.

tchrist

23:17

I think you are saying, for each letter, what are the most frequent next letter following, and in what order/percentage.

So say A might be followed by N 10% of the time, by S 8% of the time, by T 6 % of the time, etc.

Repeat for all letters.

aăâ

Yes, @tchrist

tchrist

It’s a programming problem. Not a hard one, again, but you need the data to start with.

WendiKidd

Yeah, that sounds like a linguistics/programming problem to me. Assuming correct interpretation of the question

tchrist

@aăâ Are you a programmer?

aăâ

Yes, I'm iOS developer on Stackoverflow.

I join here by stack exchange.

tchrist

23:19

Ah, then you could figure this out for yourself then! Good.

If you get a good enough model, you can use Markov chains to generate English-looking nonsense. This is useful for memorable passwords, amongst other things.

We have a migrated question about this.

english.stackexchange.com/questions/108161/…

Q: Common English bigrams / trigrams - recognising that a jumble of letters contain only valid English words

I need to figure out the best method of ranking strings against one another so that I can tell which contain meaningful English words / sentences. This string contains English words (with no spaces or punctutation): THISISASENTENCETHATCONTAINSENGLISHWORDS This contains a jumble of random l...

Which landed here:

Q: Common English bigrams / trigrams - recognising that a jumble of letters contain only valid English words

I have a database of one million strings which I want to rank against one another so that I can tell which contain meaningful English words / sentences. These strings contain no spaces or punctuation. Some contain real English words: THISISASENTENCETHATCONTAINSENGLISHWORDS Some are a jumbl...

english computational-linguistics

aăâ

Thanks! I'll find and learn about Markov chains.

tchrist

Well, Markov chains are used for generating things.

Imagine doing it with words.

So given a pair of words "the big", what is the most likely next word?

Suppose the most common word is "red", so you output "red". If you use trigrams, 3-word moving windows, so would then ask what is the most likely word to follow "big red".

Is that what you want to do, but with letters?

aăâ

Something like that.

tchrist

You must read about this, then: :)

Mark V Shaney

Mark V Shaney is a synthetic Usenet user whose postings in the net.singles newsgroups were generated by Markov chain techniques, based on text from other postings. The username is a play on the words "Markov chain". Many readers were fooled into thinking that the quirky, sometimes uncannily topical posts were written by a real person. Bruce Ellis did the code and Rob Pike did the design. Don P. Mitchell wrote the Markov chain code, demonstrating it to Rob and Bruce on the Tao Te Ching at first. They chose to apply it to the net.singles netnews group. Examples A classic example, from 19...

It was really funny.

Rob is such a prankster.

aăâ

I just want to get the percentage rate of 7 letters After Consonant and Vowels.

tchrist

23:25

You can’t do that.

It won’t be meaningful.

Each letter has its own frequency distribution for the letter to follow it.

You can’t very well collect it and average it like that.

Imagine the letter Q.

In English, it has a 100% chance of being followed by a U.

Including that with other letters produces nonsense.

See why?

aăâ

You mean no rule for it.

tchrist

No.

I mean you are creating nonsense.

Let us assume that the letter T is 50% followed by R, 30% by S, and 20% by E.

Now average T and Q together: you just added a bogus 50% chance of U!

That is hopelessly wrong.

It is not how English, or any language, works.

Just go write a little program to process /usr/share/dict/words and calculate all the numbers yourself. It is trivial.

That is not the same as frequency in a corpus, but it is still very useful and meaningful, and easy to write.

aăâ

Thanks!

tchrist

I could have written it in less time than the time I have spent writing about the question here.

aăâ

I'll try by myself. Thanks for your time!

tchrist

23:30

Good luck.

You should edit your question to make it clear what you are really looking for.

In what the Seattle Police Department described as its “first time ever” event, authorities returned small amounts of marijuana confiscated from street dealers as part of a police investigation.

Whoa.

I guess that makes sense. For the longest time here, cops would just confiscate their stashes and send the college kids along with a warning. I guess now they will only get the warning, but keep the pot.

aedia λ

Does anyone else caption their photos in the third person? Like, "aedia uses the Pentax Spotmatic in the park. Photo taken by [husband] with the K1000."

I don't write "Me and the blah do stuff," but maybe that's because I'm also almost never in the pictures.

Johan Larsson

caption as in text under picture on a blog?

or metadata?

aedia λ

Physical photo album, mostly.

I don't think I caption that many digital photos.

Johan Larsson

ok I was going to recommend adobe lightroom for managing digital photos but it does not apply then

aedia λ

Have it.

I just also have some film cameras.

tchrist

23:43

a: n=16% l=15% t=14% r=10% c=7% s=5% b=5% m=4% p=4% d=4% g=3% e=3% u=2% i=2% v=1% k=1%
b: l=18% e=15% a=15% i=14% r=12% o=10% u=8% b=2% s=1%
c: o=21% a=17% h=14% e=10% i=7% r=7% t=6% u=5% l=4% k=4% y=3% c=1%
d: e=28% i=25% o=11% a=11% r=7% u=4% l=3% y=2% d=2% n=2%
e: r=21% n=14% s=11% d=8% l=7% t=6% a=5% c=4% m=4% p=3% x=3% e=2% o=2% u=2% i=1% v=1% f=1% g=1%
f: o=17% i=17% e=13% l=12% u=11% a=10% r=9% f=5% t=2% y=2%
g: e=18% a=14% r=13% i=13% l=10% o=9% u=6% n=5% y=4% h=4% g=3% m=1%
h: e=23% o=17% i=16% a=15% y=13% u=3% r=3% t=3% l=2% n=1% m=1%

That discards info below 1%.

@aăâ That’s running off /usr/share/dict/words.

And rounding, of course.