Hello

Heya! Sorry, didn't see that invitation until just now.

returns from breakfast, sticks nose in

Thank you foir your time )

re: OSM Planet, it very much appears to still exist -- planet.openstreetmap.org/pbf has lots of content.

That's what I've just noticed, it appears the file also big, 60GiB

that's not really convenient

granted, but neither is doing a huge number of lookups via a freely available webservice :)

that said, OpenStreetMap is certainly much more data than what you actually need -- they have not just country borders, but also, well, street maps.

that said, if you successfully apply a rough filter, maybe you can cut down the number of external lookups to something reasonable

the place I would start would be to just draw some rectangles over a map of Germany -- one around the outside, so anything outside that rectangle is definitely not in Germany; and then some in the inside, where anything in those rectangles is definitely in Germany.

if something is not in the outside rectangle, you can safely discard it without looking further

if it's in one of the inside rectangles, you can safely include it without looking further

only things that are not outside the outside rectangle, and not inside one of the inside rectangles, need have further work done

that should cut down the dataset requiring more intensive investigation quite a lot.

I just thought about something. I have a csv file that contains the gps coordinates of 86 cities in germany, what if I just loop through the entire file and search for the lightning strokes that occurred, let's say in a 40km radius and then save them ?

Would this approach reduce the computation time, even for a 14GiB file ?

i'd use the haversine formula

Personally, I'd still go with polygons over city proximity. If you care about the details, the city-proximity approach requires filtering by how close those cities are to the borders; and cities may not be evenly distributed

but that's an accuracy concern

if what you're worried about is performance more than accuracy, and you think it's good enough, then sure; it'll be vastly faster than what you have now.

BTW, statsilk.com/maps/download-free-shapefile-maps has a substantial listing of sources for GIS data that are going to be a lot smaller than the full OSM dataset

statsilk.com/files/country/StatPlanet_Germany.zip, f/e, is 7.9MB.

adding shapefiles to PostGIS database becomes relevant from there.

I don't really get what is the data that is contained in the second link that you've sent

I think the idea is to do the filtering inside/outside yourself locally.

You don't put load on any public service with that and it's probably going to be much faster than when you have network communication in between.

Also, if the service does mapping from coordinates to the country, that's a more complex algorithm than just determining whether the coordinates are in one country or not, which improves performance again.

Still, the filter by bounding box is just soo simple that it's hard not to implement it. I'd expect it to throw out >90% of all samples.

That's what I said earlier, it it very likely that this approach would missclassify a lot of lightning strokes

@YassineElBaaj, not if you do it right.

The only way it could misclassify anything is if you set your bounding boxes wrong.

If you do it right, it should be able to have three outputs: True (in Germany), False (not in Germany), None (not known if it's in Germany or not).

Nothing should ever be wrong; things should only ever be None

which means you need to use a different, more accurate lookup method.

Okay, I understood the rough idea surrounding this approach, it is however still unclear how can I draw such polygons ?

Code-wise I mean

That's the point of using squares. "Is something in a square?" is trivial.

Is it above the lower bound, below the upper bound, to the right of the left bound, and to the left of the right bound? Then it's in the square. If any of those conditions is false, it's not.

so first, you make one big rectangle that includes everything that is in Germany (and also some areas that are not). If something is outside that square, you discard it. If something is inside that square, you need to investigate further.

Then, you make a few squares inside Germany, drawn to cover as much space as they can and to contain nothing outside Germany. If something is outside those squares, you don't know what they are. If they're inside the squares, you know with certainty that the datapoint is inside Germany.

so the only thing where you need a more precise GIS lookup are strikes that are inside the outer rectangle, but not inside any of the inner rectangles.

that should be a small number of points

a much more tractable number.

Yes, I already got this idea, it is really clear in my head. Thing is, how can I draw such rectangles or polygons or whatever shape it is ?

this is precisely the part that I have not yet grasped

Well, I'd start with a map. Once you've drawn some actual rectangles on an actual map, you can look up the latitude and longitude of them and plug those into your program.

all you need to do is select the map's projection so its X and Y axis are aligned with latitude and longitude properly.

So convert lat lon to X,Y right ?

There's no conversion needed -- lat and lon are X and Y.

if you choose your projection right

also, once you've filtered your dataset you can look at the remaining points and decide if there's an easy way to exclude more of them -- f/e, adding some boxes that specify regions that are definitely in neighboring countries.

"overlay a map with points from this dataset" is a pretty common feature for widget sets / graphing toolkits / etc to provide out-of-the-box, so that shouldn't be something you even need to write your own code for.

pypi.org/project/PyGeodesy

Just found this library, you can define polygons and then call the isEnclosedBy() method to check if a particular gps coordinate is enclosed within a given polygon

To be clear -- I suggested PostGIS earlier when you were talking about actual borders in large part because there's been a ton of work over the years in efficient indexing

a real database engine is built to work with datasets larger than available memory, and has a lot of optimizations (when correctly configured; as with pretty much any other datastore one can need to explicitly create indices) to be able to do indexed lookups quickly.

While I haven't done GIS work specifically, there have been plenty of occasions earlier in my career when I wrote what I thought was reasonably-efficient in-memory code for working with a large dataset, and had its performance improved by moving to an indexed database backend.

that said

part of being a student is having a chance to learn your own lessons :)

so by all means do go your own route

(also, "earlier in my career" memory sizes were much smaller than they are today; things do change over time)

@CharlesDuffy absolutely:)

Well i'm just trying to learn as many thingsd as possible here, feel like the topic is really easy to understand, on the other hand there many aspects that I still haven't explored yet

only the size of the dataset makes it harder

nod. If you'll excuse me, I'm going to go about my day; been good chatting, and I hope the project goes well for you.

thank you