@ScottS Heroic salvage job (or "epic edit", I believe these are sometimes called).

@Caleb - Got any advice on the best SE network to post a question about internet XML search and server loads (= brief to the point of meaningless summary!)?

@Davïd As in you want to search the internet for XML content or you ar hosting an XML search and have a server load issue? Or something else?

@Caleb I have a form that searches this file on github, but I have no idea where the "work" happens.

@Caleb I was going to post my question on Superuser, then thought it might not be the best place, and further thought you would be able to give some sage advice. :)

@Davïd And the question is how does the "work" happen? Where does servel load come into it?

"How" or "where" - does any "server load" come into it? I don't imagine it happens "in the browser"...

@Caleb You can see the code I'm using if you like.

@Davïd No the magic is happening in a server side script (in this case in an object xml-query.html he's added to the base Wolf CMS install). I can't find anywhere he's posted the source for that, but I imagine it's just using some stock XML parser and formatting the output. Load should be minimal, but it's happening on his servel not in the browser.

@Davïd Oh. My mythical "somebody" is you? ... so the question is "will this kill my server?"

@Caleb :))

@Caleb Or "make my hosting provider very annoyed with me and want to charge me more"... something like that. :)

So far this is all falling in some grey area between StackOverflow, Pro Webmasters and ServerFault, but I'm still not sure what the question is.

@Caleb OK - that's a help, and maybe you have already told me what I was hoping to find out.

@Caleb I'm thinking of moving this to a more "public" setting, and was concerned that the server load would be a problem.

@Davïd Probably SO or Webmasters then. And the answer is "yes", not because the XML search takes much resource but because you fetch the source from Github over the wire every time and don't appear to cache it (or results) anywher.

@Caleb Right - so having the source "local" would at least make a difference. Don't see much point caching results, though?

@Davïd You should have the script operate on a local copy of the file (that can get cached in RAM by the system if the read frequency get's high) and then just have the script check periodically if the github version is returning a newer last-modified header than the date on your file and fetch it across it it's out of date.

@Caleb That's most helpful! Will try it out and see how it goes. Thanks for taking the time to explore my woolly question!

@Davïd @Caleb The search can also be performed on the client side (using jQuery) ;)

That would be either an SO or Webmasters question depending on how you put it. After that I wouldn't worry about it until evidence of a problem actually crops up. It's a bit inefficient but fixing it for the drop in the bucket difference it will make is probably not worth it (and that would be an SO question).

@Caleb Will have a look at the logs (so far it's only me :) and bear that in mind. @PaulVargas - thanks for that - beyond my skill level completely! I've never dabbled with jQuery, I'm afraid.

@Davïd Seriously, if the server has limited resources, you can use the resources of the client. :)

@Davïd If dealing with HTTP headers to check for updates is too scary, you can also cheat and fetch the file one a day and just have the script download a copy on run IF your local version is older that 24 hours (or whatever). That would save you from triggering a bandwidth issue and crippling the server if the site got a lot of traffic some day (suddenly popular link, DDOS attack, etc) and the programming is really simple.

@PaulVargas Not sure now if that was a serious suggestion (I suppose it was!), or meme-fodder. ;)

@Caleb Since I am managing that version of the XML file, I'll know when/if it's updated (fortunately for our purposes).

@PaulVargas XML parsing in browser clients is a PITA.

@Davïd The issue here is that anybody sending you a 1kb HTTP request makes your server go out and do a 2.7 MB download on every run even if they don't stick around to listen to the answer. That could easily become an issue with just a handful of normal visitors and it would be better than Santa Claus for a DDOS attacker!

@Caleb I have no wish to play Santa Claus to anybody, let alone nefarious DDOS types! Point well made.

@Caleb I agree with you. It is best to load the file into RAM. And check each time if the file has changed to upgrade the memory. - I do not know how to do it in PHP. :/ With Java, no problem. :)

@PaulVargas Loading into RAM is usually overkill for an application like this. You would do that if you had a long-running process that was interacting with it, but it doesn't make much sense in a stateless web page environment. It can be done in PHP, but if it was a problem the real answer would be to cache the output in memcached and avoid parsing the XML more than once per query completely.

In the mean time (until he hits some breakthrough in popularity) I think this script can actually rely on file system caching. If this is a Linux server the file system driver will throw the most frequently accessed files into whatever RAM is not being used by other apps and that will kick in to cover most usage from one-off scripts like this one.

@Caleb @PaulVargas ... and this little tool is NOT in a bid for any sort of popularity! Something like the opposite, I reckon. But thanks for all the thoughts and help - most grateful!

@Caleb @PaulVargas - I've now got it searching local copy. Odd ... it seems faster. ;) Actually, the "gloss" search is very useful for getting English-to-Hebrew results.

@Caleb I was thinking of a Singleton. And save the file contents already parsed (in RAM) in a fast data structure. Less processor, less network traffic ... although a few MB of RAM in constant use. What do you think? ;)

And, FWIW, just checking - I get 250 GB monthly "bandwidth" on that account, and that subdomain has generated 14.5 Mb so far for August. I can't interpret my "Resource Usage" stats ... except that they seem to hover around 0. :)

@PaulVargas You would have to switch to a FastCGC implementation of PHP and keep persistent threads around just serving this app. Even so you would have one copy of the memory structures around for every thread and you would need a thread per active visitor + overhead. That just doesn't make sense on the web the way it would in a specialty app.

This is the problem that memcached and the like exist to solve: do your heavy processing once and push the result out to RAM as close to the visitor as possible and fulfill incoming requests directly from the results cache without even bothering with knowing how the result was arrived at.

@Caleb Humm ... I think that PHP has many problems.

@PaulVargas It does. Many problems. But this is really more symptomatic of the way the Internet works as an application platform in general than a PHP specific issue. Almost every server side language you would build a project like this in has the same issues: a stateless environment where processes don't correlate with user sessions.

It's a fundamentally different paradigm that a piece of local software like you might write in Java and it's unfair to blame a language for not giving you an easy way to bootstrap a solution designed for a different design paradigm into a different environment.

@Caleb what you mean the fact that the internet is a hack on a hack on a patch on another hack of a system originally designed to cart text and some basic images around a network hundreds of thousands of times smaller with no security concerns causes issues?

@AJHenderson You mean the internet is a hack of some system? I thought it was just raw hackery.

well, it started as a designed system for making text markup with links

and pretty much everything since then has been patching on hacks to make it do more than it was designed to handle

HTML is so supremely inefficient at what it does

as should be evidenced by the fact that in 20+ years, nobody has ever managed to make a decent WYSIWYG editor

@AJHenderson yeah, markdown for president :)

@JackDouglas What is meant by that expression?

@PaulVargas ha, just that it's harder to get malformed html from markdown (not that markdown is actually perfect)

@JackDouglas Ah, perhaps my favourite -- and simultaneously the most frustrating -- thread on Meta.SE! =/

^^^^ My own tiny pebble thrown into the pond that is META, so small it created scarcely a ripple. ;)

@Davïd at least you didn't get downvoted - see my answer just below yours ;)

but yes, the crickets are deafening

@JackDouglas and @Davïd: Adopting Markdown Extra or some other commonly used extension to Markdown is on my list of "things I want to accomplish". But I have too many other irons in the fire at the moment.

I've gotten as far as forking code.google.com/p/pagedown/wiki/PageDown, but not much further.

@JonEricson oh, I didn't know it was done client-side (presumably just when editing/asking questions?)

so you need to have a client-side and server-side engine that match exactly? That can't be easy.

@JackDouglas I don't know off hand where the conversion actually happens, but I figured this was the way to do a proof of concept.

@JonEricson you seen this: github.com/jmcmanus/pagedown-extra

and this toptensoftware.com/markdowndeep !!

@JackDouglas No. Wow. That does just about everything (including anchors).

oh, MarkdownDeep has a "Client Side - Javascript port, 100% compatible, tiny at about 30k"

no to mention "Extremely fast C# implementation (typically 15x faster than MarkdownSharp)."

Ok. I think I better bring this up as an option.

:)

(But what will I do to teach myself JavaScript. ;-)

you haven't had that pleasure?

it's a beautiful language

(with a few quirks of course)

@JackDouglas I understand its actually very lispish. (lispy?)

@JonEricson never thought of it that way, but apparently others do make the link. Syntactically it's nothing like it of course

what languages are you at home with?

Perl, C, shell scripting, SQL, and smatterings of Lua.

that's an eclectic mix :)