Hello! I wonder what does it mean to "SO Ready To Help" ? when somebody has that on their profile. Is it on "offical" SO team or just a loose expression for a programmer willing to help someone?

votest down, close and delete please: programmers.stackexchange.com/questions/330471/…

^^^ entered HNQ, how nice. Let's tell thousands SO lemmings that this site welcomes questions like "Is it possible to be a great programmer with poor language skills?"

eh, thousands of stack overflow lemmings already think that.

OK, I must be really dumb these past couple of days. I can't figure out why I'm only ever getting the first record in this JSON file. I would totally implement some kind of looping if I had any idea how I was supposed to do that, but everything I've read doesn't seem to think that's necessary, so maybe my file isn't formatted properly?

What are you using to read the JSON?

I've tried a number of things. I think it's looking at the curly braces and deciding there's only one record.

I am using JSON.Net atm.

Mostly just trying to figure it out. I have only worked with JSON as the output of an API call before.

Reading from a file is a little kooky feeling.

Oh. Oh! OK, maybe I just figured something out finally.

I was trying to alter it to force it into an array by adding []. What I didn't do was put curly braces around the whole thing.

Which might work if I could then get this ginormous file onto the server to process.

how are you reading the file itself and passing to the parsing lib?

you could just feed the json parser the dummy tokens before and after the data

Oh. Yeah. I could do that.

Hmm.

I mean, I can do it for the code. Ultimately I have some kind of component I'm meant to configure.

Well, I could write the tokens in with file operations easily enough.

I think I'll try that.

Are you generating the JSON?

Yes, but I don't have much control over it.

Even if it has errors that make it invalid?

It's all this crappy Mongo data.

So Mongo has a builtin export tool that creates a bson file which can then be converted to a json file, but it's all boxed up CLI stuff.

I would be surprised if that file is invalid.

It's not invalid.

I don't understand the problem, then. I thought your reader wasn't finding an array where there should be one.

The file has about 10000 correctly formatted json objects. My tool only returns the first object.

So I made it into an array and that seems to work.

I think.

What do you mean "made it into an array"?

{"products":[ + set of objects + ]}

Did you edit the JSON?

Yes. Well, a sample file.

I edited the huge file, but it's still transferring so I can't test it yet.

I wouldn't expect that you should have to do that with the output.

Neither did I, but I don't know how to iterate the records otherwise.

How often do you need to do this? Because hand editing files is error prone and time consuming, if you need to do it more than once.

Well, that's why I figured I'd write a little script for it. I need to do it as often as I need to do it. It will probably be more than once, less than one hundred times.

I'd be more suspicious of the parser. But I'd look at bug reports against the tools that create it, too. I'd have to read about what JSON needs for an array. You shouldn't have to even have a script, if you are using two off-the-shelf tools to produce and then read the JSON.

Well...I'm repurposing the json. That probably has something to do with it.

The purpose of the file is to backup the Mongo db, so there are tools to reconvert it to Mongo. It's just that I want to strip it into SQL instead.

Have you put the JSON (or a sample of it) though a JSON validator?

'not sure what you mean by 'made it into an array' but remember that "property": [{}, {}] is valid JSON but "property": {}, {} is not.

jsonlint.com

@PeterTòmasScott No, I didn't, but I'll do that.

Yeah, it checks out.

Oh. Hmm. Yes, the test file checks out, but the actual file doesn't.

Because I also don't have commas in the original. I have LFs.

each line is a separate json?

It looks like this:

{"_id":"100","ProductNumber":"000100","ProductName":"Product1"}
{"_id":"101","ProductNumber":"000001","ProductName":"Product2"}
{"_id":"102","ProductNumber":"000200","ProductName":"Product3"}

Well, way more complicated than that, but that's what I was messing with.

Well if that is the case then it explains why the JSOn parser was stopping after one object

If this is only happening at the highest level in the JSON structure then you could just pass each line of the file to the parser separately and build it up that way. If that format is repeated at multiple levels in the JSON structure then I guess you will have to correct the syntax.

OK, that makes sense. I'll have to get each line and pass it through the parser.

I can grasp that conceptually.

Seems like every change I make ends up with an out-of-memory exception. I guess I'm really good at breaking this.

too large of a file?

It's really huge.

For a flat file.

400MB.

You may have to do it at the text level, one line at a time.

Yes, I think so. At least I feel like I can understand that process.

I don't know why I didn't see that solution before.

Seems obvious now that you've pointed it out.

Well, first class objects are better, but not if you're always running out of memory. And the text parsing problem here shouldn't be all that difficult.

had a bit of a XY problem there didn't you :)

Although I suppose there are probably line breaks in bad places too, just because Mongo is annoying and wants to send me to an early grave.

By 'line breaks in bad places' do you mean newline characters as part of a text blob?

If so then when you are reading the file line by line you'll need to check that the line ending has a curly brace }. If not then read the next line as well, etc.

{"products":[
{"_id":"100","ProductNumber":"000100","ProductName":"Product1"}
{"_id":"101","ProductNumber":"000001","ProductName":"Product2"}
{"_id":"102","ProductNumber":"000200","ProductName":"Product3"}
]}

@KitZ.Fox Like that?

once you go deep enough with correcting the bad format you may as well have written your own parser...

{"_id":"100","ProductNumber":"000100","ProductName":"Product1","MiscData":[
{"stuff":"1"}
{"stuff":"2"}
]}
{"_id":"101","ProductNumber":"000001","ProductName":"Product2"}
{"_id":"102","ProductNumber":"000200","ProductName":"Product3"}

more like that^

also missing the comma after {"stuff":"1"}?

That's my point. It's a line break, not a comma.

I could replace all the LF with commas and then break on },{ or something.

Or subsequently replace the },{ with } NewLine {

or count the { minus the } and break when a new } makes it 0

},{"_id" would mark the new lines.

I'd have to double-check that, but I'm pretty sure.

It would help if I had an application that would open the file. Notepad++ can barely manage it.

Well, you don't need the whole file to build the utility. You just need enough of it to cover all possible JSON configurations.

Use a small portion of the file for testing.

Yeah, except I don't know what those are without inspecting the file.

Build it using the first megabyte or so of source JSON, then turn it loose on the entire file and see what it does.

Yeah, that's my usual method. Build and break and fix. Repeat until it stops breaking.

I'm an excellent programmer. That's why I'm a BA now.

laughs at self

heh.

Anyway, I just did a quick check and I guess there aren't any weird line breaks.

So hooray! I think I have a new approach. And days of development ahead.

Thanks.

wait json mangling is days of work now?

can be if you want it to

A little text processor ought to take a couple hours to build.

Well, that just gets me the objects. I still have to deserialize them and insert them into SQL tables.

I thought you just needed the transformed JSON text.

And the objects are more complex than I demonstrated.

@RobertHarvey Yes, maybe. Maybe not.

today seems to be the day of blatant software recommendations. Two questions waiting for votes down and close: 1, 2 and two - for votes down and delete: 3, 4