There is no simple way to do this in any but the most basic way because language is very complicated and there is no simply way to specify what makes sense.
I am searching for information on algorithms to process text sentences or to follow a structure when creating sentences that are valid in a normal human language such as English. I would like to know if there are projects working in this field that I can go learn from or start using.
For example...
def sentence_generator(filename, length=10):
random.seed(1) # Set the seed for the random generator - do not remove
my_tuple1 = learn(filename)
sorted_word_list = sorted(my_tuple1[1])
i = 0
while True:
word = random.choice(sorted_word_list)
word1 = ""
...
Suppose we take some random bit of text. Give me a moment and I'll find something.
Dorothy lived in the midst of the great Kansas prairies, with Uncle Henry, who was a farmer, and Aunt Em, who was the farmer's wife. Their house was small, for the lumber to build it had to be carried by wagon many miles. There were four walls, a floor and a roof, which made one room; and this room contained a rusty looking cookstove, a cupboard for the dishes, a table, three or four chairs, and the beds. Uncle Henry and Aunt Em had a big bed in one corner, and Dorothy a little bed in another corner. There was no garret at all, and no cellar – except a small hole dug in the ground, called a…
For example that's from the start of the book The Wizard of Oz.
Now consider pairs of words from this text e.g. consider the two words "who was".
We can go through the text and see what word follows the two words "who was". We can build up a list of all such words.
So the two words "who was" can be followed by the words "a" and "the". If you're making up sentences at random and you have the words "who was" in your sentence you could randomly choose one of the two words known to follow "who was" to make your random sentence and it would probably make sense.
So what you do is take your sample text and for every two work sequence that occurs in the text you make a list of all the words that follow those two words.
You put all this data in a dictionary. I'm not sure if you've learned about dictionaries in Python yet?
OK, so in the dictionary the key is the two word sequence and the value is a list of all the words known to follow the two words in the key.
Once you've built up your dictionary you can generate a random sequence by:
1. chose a key at random and start with those two words e.g. "who was"
2. get the list of words that follow your two words and choose a word from this list at random. So in this case you might choose "the" and now your sentence is "who was the".
3. Now take the last two words of the sentence "was the" and you use these two words as the key and find the words that follow "was the".
Keep repeating the process until you reach some predetermined sentence length.
First decide what text you are going to use for the initial analysis. Any text off the Internet will do. You can keep this text in a file and read the file from your program. That way you can change the text whenever you want by editing the file and changing the text.
So you need to know how to read text froma file in Python.
The zip() function returns an iterator of tuples based on the iterable object. If a single iterable is passed, zip() returns an iterator of 1-tuples. Meaning, the number of elements in each tuple is 1. ... It's because iterator stops when shortest iterable is exhaused.
The loop for i in range(0, len(word), 3 takes values of i starting at zero and increasing in steps of 3 up to len(word)-1. So the values of i will be 0, 3, 6, 9 etc up to the length of the list word minus one.
And word[i:i+3] takes the three characters from the string starting at i. So if i=0 it will take word[0] + word[1]+word[2] or if i=3 it will take word[3] + word[4]+word[5].
So if you look at your string word = 'CatBatSatFatOr' it's going to be split up into three letter groups.
There is an optional second argument that specifies how many times the string is split. The default is to split the string at every occurrence of the split character.
If instead we do word.split(',', 1) this only splits once. So our original string 'geeks, for, geeks, pawan' is split into two string 'geeks and ' for, geeks, pawan'.
@JohnRennie a='Dorothy lived in the midst of the great Kansas prairies, with Uncle Henry, who
was a farmer, and Aunt Em, who was the farmer's wife. Their house was small, for
the lumber to build it had to be carried by wagon many miles. There were four
walls, a floor and a roof, which made one room; and this room contained a rusty
looking cookstove, a cupboard for the dishes, a table, three or four chairs, and
the beds. Uncle Henry and Aunt Em had a big bed in one corner, and Dorothy a little
That is, when we look up the word pair "the cat" in the dictionary it tells us that the word "sat" can follow the word pair "the cat".
Now we need to look at the next word pair and this is "cat sat" i.e. we move along one word. Now the word that follows is "on". So our dictionary now looks like:
Now we take the last two words of our sentence i.e. "cat sat" and we look up this pair in the dictionary and we find "on". Append this and now our sentence is "the cat sat on".
@JohnRennie >> a = "the cat sat on the mat" >>> b = a.split(" ") >>> b ['the', 'cat', 'sat', 'on', 'the', 'mat'] The way I would do this is to start by splitting the string at spaces to give a list of the individual words in the string. OK so far? Now we go through the string a word at a time to see what word follows the pairs: >>> for i in range(0,len(b)-2): ... print(b[i] + " " + b[i+1] + ": " + b[i+2]) ... the cat: sat cat sat: on sat on: the on the: mat
OK, so in the dictionary the key is the two word sequence and the value is a list of all the words known to follow the two words in the key. Once you've built up your dictionary you can generate a random sequence by:
1. chose a key at random and start with those two words e.g. "who was" 2. get the list of words that follow your two words and choose a word from this list at random. So in this case you might choose "the" and now your sentence is "who was the". 3. Now take the last two words of the sentence "was the" and you use these two words as the key and find the words that follow "was t…
Hang on, what exactly are you asking? The text above assumes that all the processing of pairs has been done. Aren't we still wondering how to process the word pairs?
I just chose some random text. In the real program I'd have a large amount of text that I'd read from a file. You want a lot of text so you have lots of different pairs of words to work with.
I chose that text because it was short enough that I could work through it to show how the dictionary is constructed.
c="my name is percy jackson.I am not the lightning theif."
def text_reader(x):
a=x.split()
worddict={}
for i in range(0,len(a)-2):
key=a[i]+" "+a[i+1]
worddict[key]=a[i+2]
print(worddict)
text_reader(c)
is this correct
can you see the output and tell me
it looks kinda messed up...like the key with full stop
c="my name is percy jackson. I am not the lightning thief."
def text_reader(x):
a=x.split()
worddict={}
for i in range(0,len(a)-2):
key=a[i].replace('.', '') + " " + a[i+1].replace('.', '')
worddict[key]=a[i+2]
print(worddict)
text_reader(c)
So you remove the full stops from the keys but leave them in the values.
c="my name is percy jackson. I am not the lightning thief."
def text_reader(x):
a=x.split()
worddict={}
for i in range(0,len(a)-2):
if a[i].find('.') == -1 and a[i+1].find('.') == -1:
key=a[i] + " " + a[i+1]
worddict[key]=a[i+2]
print(worddict)
text_reader(c)
Suppose your two words are "the dog". There will be lots of words in your learning text that follow this e.g. "the dog sat", "the dog barked", "the dog rat", and so on.
That's OK. You can have a list with one word in it. Then the random choice will always return that one word.
Anyhow, this is what I did. You'll see it's very similar to your code but with a few tweaks to make the value a list:
for i in range(len(wordlist)-2):
if wordlist[i].find('.') == -1 and wordlist[i+1].find('.') == -1:
key = wordlist[i] + ' ' + wordlist[i+1]
if WordPairs.get(key) is None:
WordPairs[key] = [wordlist[i+2]]
else:
WordPairs[key].append(wordlist[i+2])
I've been playing with this, and if you're interested I have the text of The Wizard of Oz in a form that you can use with your program. If you would like a copy of it I can put it on my website for you to download.
# Define the name we use for the file containing the learning text
TEXTFILE = "sentences.txt"
# Function to read the text
def readdata():
with open(TEXTFILE) as f:
contents = f.read()
return contents
# Main program
text = readdata()
# Build the dictionary of word pairs
builddictionary(text)
I don't know if you've done reading stuff from files yet.
I must admit I just Googled for how to read text from a file :-)
This is an interesting project. It needs a lots of stuff!
So you've basically done reading in the source text and constructing the dictionary. You just need to write a function that builds a random sentence using the contents of the dictionary.
The way I did it was to ask the user to type the first two words of the sentence.
e.g. the user might type "the dog"
Then I use those words as the key and build the sentence from there.
For example my code gives sentences like this:
D:\rhs\Python>python sentences.py
Enter two words to start the sentence.
First word: the
Second word: dog
The dog and the tin woodman came to another tree, as he spoke there came a day longer, said the princess.
D:\rhs\Python>python sentences.py
Enter two words to start the sentence.
First word: the
Second word: dog
the dog in her had dried up many years older.
def sentence_generator(filename, length=10):
random.seed(1) # Set the seed for the random generator - do not remove
my_tuple1 = learn(filename)
sorted_word_list = sorted(my_tuple1[1])
i = 0
while True:
word = random.choice(sorted_word_list)
word1 = ""
...
