Barry Carter

you bet!

You bet, and good luck!

That should work also, yes. It's slightly better because it uses a good library

I think you have to convert UTM to pixel yourself based on the edges, but I think you've already done that part

OK, stackoverflow.com/questions/176137/java-convert-lat-lon-to-utm has a formula

let me see if I can find the reverse transofmration

Oh wait, my bad, you're going the other way

I copied their code to pastebin.com/e6Sv7bwJ

You could cheat and use the CSV on engineeringtoolbox.com/utm-latitude-longitude-d_1370.html which allows multiple entries at once

OK, let's see if we can find a formula

I get it-- the coordinates on the map are UTM

You're right, but it seems like you've already translated the data to latitude and longitude, now we just need to know more about concatenate.jpg -- that's an image you from Sentinel?

What projection is concatenate.jpg ?

OK, but that doesn't tell you what projection your specific map is in?

(actually, you may need to divide, but the key is that longitude degree is smaller than a latitude degree except at the equator)

OK, when computing longitude distance, multiply by the cosine of the central latidude

That's also not a projection. A projection is how a map converts latitude and longitude to pixels (in computer graphics)

Wait, I don't think WGS84 is a projection? I'm guessing it's Mercator.

What projection is the original map? In non-cylindrical projections such as Mercator, one degree of latitude uses a different number of pixels that one degree of longitude.

sure thing... when you get back: what is the project? Something like commoncrawl.org? My email is [email protected]

Right.. and you're using single quotes to quote the double quotes?

And the special characters didn't create any issues?

Personally, when I see a directory whose ls -ld shows it's over 1M, I clean it up and move everything to a new directory

Also, if you create directories with millions of files at the top level, operations like ls get very slow

You only have a finite number of inodes

do df -i

That might be overkill in the other direction-- having a billion... no, Unix does have a limit

If you could structure the data in such a way that it's "byte aligned" (or whatever), you could use a text file. I actually do that with large chunks of data

I mean databases would be the obvious next step

That would be my guess too. To be fair, actually trying to move around billions of characters inside a file would be insanely painful

I see only a few special characters in those strings-- add more and see if it still does what you want.

I have at least one Perl program somewhere with 8 backslashes in a row, so I feel your pain, but I don't see an easy workaround-- the hex thing may work, treat characters as bytes-- you might be able to write your own short C code to do this, but even C treats things like \n as newlines

And --posix of course

superuser.com/questions/532420/sed-without-regex suggests using hex codes-- you can also quote with single apostrophes to stop a lot of the regex matching

tr might also work

You can use a small text file to test-- I think there must be ... let me poke around. Have you looked at ed?

I mean, you can always backslash special characters (but double check that you get the right thing as the shell might interpret the backslash first)

Super large files can be useful if they have some byte-based structure. For example, if you know where the data you need is.. let me check re sed

OK, then sed -i should work

Oh, then what was the problem using sed or whatever?

Do you have 74GB free?

I think your lack of diskspace is going to be fatal though

Right, make a 1GB file and edit that, repeat 74 times, glue results

Oh, so you could process it one block at a time?

Sometimes given as ^@

ASC character 0

Also, is the whole thing a single JSON object, or multiple objects you could break apart programatically?

Do you know if your file has a large chunk of nulls in it? That sometimes happens with bad file writes

Oh, then gawk is just writing to a temporary file and will do a 'mv' later, which is technically not in-place editing

Can you share any details of how you got this file-- can you recreate it, etc?