Discussion on question by TrashBinNp2019: Threats that JavaScript poses to a web crawler

Feeds

11:02

0

Q: Threats that JavaScript poses to a web crawler

I'm writing a simple crawler with node.js, which searches for web pages and conditionally executes any JavaScript present. The problem is that in doing so, I execute code form untrusted sources in my node.js environment. Can running untrusted code on node.js in such a way be dangerous (i.e. when ...

schroeder

it will depend on how you code your crawler. You are basically asking, "what could a malicious site do to my custom client that I have not finished yet?" And that's a little like asking "how long is a piece of string?"

TrashBinNp2019

@schroeder any tips on that?

schroeder

The question is too broad. You are asking how to code your project securely. There are entire libraries of tips. Please look up OWASP as a start .

TrashBinNp2019

@schroeder could you please clarify

schroeder

What would you like me to clarify?

TrashBinNp2019

11:02

@schroeder if there are libs that solve problem of a crawler executing malicious code, how do find and integrate them?

schroeder

When I said "libraries" I did not mean code libraries. I mean many, many books dedicated to explaining secure coding. I cannot possibly summarise them. You need to refine what problems you want to solve.

Sir Muffington

Welcome to the community. The answer for your question is already in the comments. Please edit the question to have your current problem at hand.

Steffen Ullrich

"Can running untrusted code on node.js in such a way be dangerous ... Code is executed by jsdom" - To cite from jsdom README: "his is also highly dangerous when dealing with untrusted content. The jsdom sandbox is not foolproof, and code running inside the DOM's <script>s can, if it tries hard enough, get access to the Node.js environment, and thus to your machine.". In order to mitigate this run your nodejs inside a contained and restrictive environment like VM, container, sandbox, ... and don't trust anything from this environment.

TrashBinNp2019

@SteffenUllrich I considered that, but still. Perharps, there are other measures to be taken.

Steffen Ullrich

@НикитаПучинский: I'm not sure what other measures you expect. If the library you use explicitly says that the attacker can get access to the nodejs environment, then you must make sure that you don't trust this environment by containing and restricting it.

pcalkins

11:02

You need to protect against calls made to nodejs. (There really is no way to do that, but some ideas to at least make it more secure are checking each call to localhost against a specific whitelist. IP/Port... you may also want to choose a randomized port for your listener.)

TrashBinNp2019

@pcalkins not sure what listener are you referring to. Do crawlers normally listen for something?

pcalkins

Maybe you wouldn't need that... I'm assuming you are running some code outside of node server? This advice is really just top-of-the-head type stuff, so take it with a grain of salt. As I said before nothing will be secure, but you might be able to "harden" a bit. I've never used jsdom but it's a good sign that there is a sandbox there. I'd look carefully at the source code to understand how the sandbox is constructed.

Even with a "sandbox" in place, you will have code somewhere that is bridging that... (otherwise you couldn't inject into or read the DOM) That is the real danger here... You should have code running outside of the sandbox in order to run your bot/scrape content. That's the bridge, and that's where untrusted content can run at the same privileges as the code itself.

TrashBinNp2019

@pcalkins only code outside the sanbox is an http client (axios, to be precise), which supposedly isn't too vulnerable. And, thinking of your earlier comments, i won't be able to implement any whitelist system, since my http client searches for random hosts and passes any discovered pages to an inspecting system. My perceived problem is that if any of those pages contain phishing, hooks or any other malicious javascript targetting browsers, my system will fall victim to them, since this javascript will be executed.

guess i should edit my question to include all that as a clarification

pcalkins

The sandbox in place will take care of javascript execution... (though calls to localhost may still get through and be able to execute something at node.js server) It's what you pull out of the sandbox that is the bigger danger. (or what you send in, if anything confidential is sent) That can easily break out of any code used to process it and/or control your node server.

TrashBinNp2019

@pcalkins as another clarification, i don't own any of the crawled pages, so confidential information is not a problem for me

pcalkins

11:02

Bottom line is it's probably impossible to fully secure a web crawler. Even google had problems such as this: infoworld.com/article/2609489/…

TrashBinNp2019

@pcalkins that's pretty insightful, thanks. Guess I should consider other ways of bot exploitation as well

pcalkins

To be fair to Google, that article is from 2013... you'll be most concerned with taking in untrusted content, protecting your locahost, and protecting your code from injections. Using a fresh VM for each run is also a good idea.

TrashBinNp2019

@pcalkins thanks a lot, that actually covers all i wanted to know. Maybe you would want to elaborate this into an answer?

pcalkins

One of the nastier things a malicious site could possibly do is cause your node server to crash and replace it with their own. Again this is just spit-balling... I have been using Selenium for many years and have never been attacked, but I don't crawl to just any web site, but quite a few I have no control over. I've never been attacked (that I know of...) I don't feel like I have enough expertise with node or jsdom to offer a good answer here, though.

TrashBinNp2019

@pcalkins yeah, that would be quite a surprise. Thanks again for the info, I will be sure to dig futher myself.

schroeder

11:02

@pcalkins please do not provide answers in comments. Please convert the comments into an answer.

Transcript for

Discussion on question by TrashBinNp2…