hi!

Hi.

thank you soooo much for all of that!!!!

you are incredible!!!

No problem, I'm glad to help.

i will surely award you the bounty tomorow

i just want to go over the code once more to see if I have any further questions, but unlikely

Okay.

May I please discuss another perl-related inquiry?

Sure. What is it?

I'm not sure where to get started on code or even if it is possible which is why i"d prefer to chat before posting a formal q

so you know that familysearch website I'm using?

Yes.

there is that and there is ancestry.com

I have a list of about 8 million names.

For each person, I know their name, birth year, birth state (or birth country), gender, and race.

Name: First name and surname

Birth year: may be off by exactly 1 year.

+ /

My goal ?

Within +/- 1 birth year, some of these 8 million entries should be unique.

For those that are unique in the list, I want to see if I can "find" them in the 1940 census (the website I took you to earlier) and pick up some of the information under their name if they are also unique in the census

Well, the program should be pretty easy to adapt to support names.

Here is an example record.

familysearch.org/pal:/MM9.1.1/K9H8-44J

Unfortunately family search does not give that much additional info. Residence in 1935 would be good to pick up.

I'll get an example from the ancestry website.

Ancestry is not as good of a website though. Its API is harder to follow and uch

That example includes a "residence in 1935: Same Place"

yes exactly

i mean ancestry.com gives some more detail

ancestry.com also gives educational attainment, for example (like high school grduate, college graduate)

i guess i could have the information available in a .txt file ?

Do you have the information in a .txt file, or are you trying to produce a .txt file?

well i have it in a different format. i mean i could put it in one i guess if that would be productive

do you know R?

Sorry, I don't know R.

that's ok

have you heard of R?

I have head of it; it's a statistics programming language, right?

it is a statistical software package used for regressions and the sort

There seem to be plenty of people who on Stack Overflow answering questions about it.

well my problem is not about R

the point is that at the moment, i have the data in an R file

(or something similar to R)

and I can convert that to .txt

i'm not sure if that answer you q about the .txt

I see. Then you should be able to use it in your Perl program, yes.

search.ancestry.com/cgi-bin/…

that is the record for the same guy in ancestry.com

it has more infomation but ancestry.com is just not so pleasant to program with

It won't let me see that without registering -- which is certainly going to make scraping harder.

for example, it says that he was in high school

it is free to register

not sure if that should be a problem

but yes, i've had to register to scrape from them

the code was a real pain in the rear

Well, you'll have to have the program keep track of cookies and so forth. I guess you've already discovered that. :)

It gives some more informatoin like:

Inferred Residence in 1935: San Diego, San Diego, California

Marital Status: Single

Resident on farm in 1935: No

Some information I'd like to collect

@nandhp yes

that was a real pain, but i have some code for that

my question is whether it is possible to take the parameters that I have and somehow add that additional information . . .

for example, if I had David Abbott in my list (which chnaces are that I do), and suppose is the only white guy named David Abbott born in California between 1922 and 1924, I'd like to collect the extra information

So you need help scraping the data from Ancestry.com? Or is that not the part you're wondering about?

i have never scraped at this level

i've only scraped like i scraped in the thing we dealt with at the county-level

never scraped individual records and all of this extra stuff (like only if it is a unique match)

Well, it shouldn't be too hard to scrape the person detail pages. They're both plain-HTML (not JSON).

which one? the familysearch one? or the ancestry one too?

Both of them.

are you located in the united states?

Yes...

i am so excted

i just wanted to check, partly because it is clearer the type of data and terms i'm uing.

I'm not sure what advice to give you about scraping the pages. There's basically two ways to do it.

this code is really not difficult to write?!?

you saw my original code on that post, right?

until today, that was the only way i knew how to scrape.

and for ancestry.com, getting those county-level totals, it sufficied

You were on the right track, and when the data's not loaded via AJAX it's sufficient.

It's probably fine for parsing both of these tables.

@nandhp it didn't work for familysearch.

But that was for the search results page, which loads the results with AJAX.

Pages like familysearch.org/pal:/MM9.1.1/K9H8-44J have the data in the page.

If you do "View Source" you can see it.

That's different from the case of familysearch.org/search/collection/…

@nandhp that is great. the url is not as easy to work with though. the county-level one had a transparaent url

But familysearch gives you the URL in the search results.

i'm not following, but I would not know how to loop over a url like: familysearch.org/pal:/MM9.1.1/K9H8-44J

When you download the search results, in addition to the total number of matches, it includes some information about each of the first twenty people.

The Perl program I wrote downloads that data.

@nandhp oh ok

And that includes the familysearch.org/pal:/MM9.1.1/K9H8-44J for each of those people. It'll be something like $obj->{searchHits}[0]{person}{url}.

the 2nd thing I'd want to do is similar and as follows:

Take the same 8 million people

Here is David Abbott in the 1930 census: familysearch.org/pal:/MM9.1.1/XC6W-KJ9

In the 1930 census he is still young

n the 1940 census he is older

We learn that he is siblings wth:

suppose I have Paul Abbott in my data

and suppose Paul Abbott matches uniquely

In sum, I'd want to know that this David Abbott and this Paul Abbott in my list are brothrers. And, they lived in: event place: Ontario, San Bernardino, California

I did not mean to cut you off

That one might be a little more complicated.

We can make a list of potential brothers who share things like last name, race, (birth state or country ?)

You could, and also keep a list of people who are brothers. That's a little more tricky.

how complicated is this all?

Maybe. I'm not sure what to suggest for this, since I don't have a very good idea what you're planning to do with the data. If all you need is a list of sibling relationships, that's relatively easy, you can just append them to a file when you encounter one.

@bloodworks hello?

But at some point you're going to have people, and where they went to school. And then you have who their brothers are, and I assume you want to associate that with the other data you have about them. You might need something more complicated than a list of siblings -- like a database.

they are 2 seprate but reltaed things

first task is matching unque people to 1940 records

second task is seeing whether anybody are brothers and where the brothers lived in 1930

Well, once you have saved all the 1940 records, then you can walk through them matching up brothers. Since you have the data from the websites, you may not have to guess (although if you have anything like 8 million names at that point, you may want to use guesses to limit your search space).

If keeping a list of brothers is sufficient, it's probably not too difficult. But if you need something more complicated, the programming will probably get more challenging.

I hope I've been of some help with this.

I'm afraid I have to go.

@nandhp My computer failed on me. Can we keep in touch?

@bloodworks Hey! :)

perl + Latex : )

Hey @nandhp!

I would greatly appreciate continuing along if you have a few extra moments? Thank you so much. I appreciate everything.