Conversation started Aug 22, 2012 at 13:23.
Aug 22, 2012 13:23
I had a guy ask my opinion on tools for, specifically, ColdFusion, PL/SQL, and Java. He's already got a call scheduled with HP to talk about Fortify SCA and said he's also looking at AppScan Source.
@ScottPack I'll ignore that one. Mom doesnt like when we fight.
I've heard of both of those, but that's about where my experience ends.
@ScottPack in general - they all suck.
Appscan source (aka Ounce) sucks a lot less than fortify.
Fortify are considered the leaders in the field, but only cuz they suck so much more than the others.
Is the suckage a function of the fact that noone cares enough to make decent tools, or because of the inherent difficulties of what they're trying to do?
partly, I'm pretty sure they have "support" for all those languages. not sure ounce does.
@ScottPack bit of both, really.
in fact, there is one that I'm familiar with - Checkmarx - which doesnt suck (though their tool still shows that they are a startup), because they went in a whole other direction.
Aug 22, 2012 13:26
@M'vy true, just trying to see if any other light can be shed on that topic. the bronze badge that comes with awarding a bounty on another persons question is nice as well. :P
here's the thing, though - if you have medium- to low- quality apps that need scanning, fortify can (sorta, kinda) do the job.
can pick up most (well, a lot) of the low-hanging fruit.
@AviD you say suck a lot.
and it doesnt take a lot of up-front investment of time, unlike some others
So when you say low->medium quality you mean as in how security conscious the coder was, not in regards to complicated?
'course, then you still tend to spend a lot of time sifting through the pile of duplicates, false positives, and other crap.
Aug 22, 2012 13:28
sure
@ScottPack mostly, yeah.
if the code is overly complex, fortify (and others) will likely drop the ball more often.
My only experience in this area has been splint, which seemed to work kind of ok on small projects. What annoyed me about it was the fact that to really scale to anything it required a lot of tagging of the code so that you were, in essence, telling splint what this section of code did and whether it was ok to ignore it.
but a mostly-straightforward swiss cheese app (as most of them are) will do well with having the low fruits picked up.
Does that complain still hold true?
@ScottPack sort of.
Aug 22, 2012 13:29
Are these bigger commercial tools assuming that you'll add in those meta-tags to your code?
While fortify does support some customization, its mostly a waste of time, and sca will just guess. Often wrong.
Well played.
same goes for most other tools, not just fortify.
as an adjunct to AviD's comments, if cost is a factor from what I've seen Checkmarx is way cheaper than either HP Fortify or IBM Ounce...
though splint is pretty much mostly text oriented, without real code-parsing, and the bigger tools do a lot more work. and, they look for a heck of a lot more than splint.
Aug 22, 2012 13:32
@RoryMcCune The follow up question to that is, of course, whether or not it's good enough for the cheaper price tag to matter.
@RoryMcCune it depends. there is the list price, then there is the package price that HP/IBM can undercut to get a big client onto some of the other products.
@ScottPack I find checkmarx so much better than the others, its silly to compare.
@AviD Sure sure. While I have used splint, I pretty much ignore it as experience in the area. :)
@ScottPack indeed, the only real way to determine from what I've seen is to PoC them on a real app.
@AviD Is it better in that it finds more, has lower false positives, easier to use...?
usually I've seen companies go with fortify simply because it was order of magnitude cheaper.
@ScottPack yes.
Aug 22, 2012 13:33
@AviD true, I've only 1 or 2 datapoints but from what I saw it was a pretty large gap, and the big boys weren't too keen to try and close it.
well, except for the last - its pretty easy out of the box, but to get real value - you need to invest serious time customizing.
the beauty of checkmarx is that its not a closed engine.
Whereas, if I'm understanding right, Fortify requires less customization?
unlike the others, where you have closed engines with very small customization points (read: meta-tagging), checkmarx is more of a static analysis platform - you can literally write queries to find almost anything you want in code.
While that's cool, you have to be willing and capable of doing that to get the most out of the system.
one of the more beautiful queries I saw was to calculate code complexity.
@ScottPack absolutely.
so as I said - if you're looking for low hanging fruit, go with HP/IBM.
Aug 22, 2012 13:36
Are the queries, generally, portable enough to be meaningful if shared, and is there any kind of query marketplace?
if you want really good security, and long term (there is also better support for rescanning etc) - Checkmarx, but with no little investment.
btw, between hp and ibm - language support can tilt strongly in hp's favor. general level of non-crappiness goes to ibm.
@ScottPack ha! I wish.
so here's the thing.
it does come with, out of the box, a couple hundred vulnerability queries, which are very generic, and very useful.
e.g. XSS, SQLi, passwords, etc etc. the usual stuff.
you can also customize these existing queries - using a form of polymorphism - to fit your code/environment/infrastructure.
e.g. SQLi doesnt find anything, because you're using a special DB - with Cx you can customize just the part of the query that defines what a database is.
good luck with getting any sense out of any tool. Except maybe splint.
on top of that, you can tailor queries specific to your app, or special things that you want.
e.g. a custom authorization mechanism - find misuse, lack of use, etc.
so these customizations dont really make sense to share generically.
the company does tend to roll in any new vuln queries offered them, though.
I was actually assuming something like that but hoping.
another big drawback they have - very limited language support.
.net, java, and limited C++ and I think now php.
and certain frameworks are not yet added.
Checkmarx has quite a bit of new language support although I'm not sure how mature it is yet, but definitely covers things like ruby + Rails ..
@RoryMcCune ah! thats news to me. good to hear.
Aug 22, 2012 13:44
good point on framework support, that was one thing that tripped me up when I started looking at static analysis. Java alone isn't enough if you're looking at a struts 2 app...
@AviD yeah initial release was tail end of this year I think, Android specific has been added recently too which is nice for mobile reviews...
@RoryMcCune do you get to play with Cx a lot?
I really enjoy when I get to use it for a client. way too pricey for me to get myself, but I've help custom develop queries for many clients after they buy it.
as opposed to when they insist I use their other tool, e.g. fortify. I tend to curse a lot when that happens.
@AviD one of the companies I work with is a reseller, so I get a lot of the sales docs. When I get some time really want to spend some more time looking at it. I've can get access to their on-line version for eval/training ...
@RoryMcCune cool. it's really a development-side security tool, and not a security-guy's app tool.
i.e. you really need to be comfortable with code, both reading and writing.
 
Conversation ended Aug 22, 2012 at 13:50.