If their paper's rejected, but you then publish a paper criticizing the algorithm that wasn't published, then how's that work? For example, would you publish their algorithm yourself for the sake of also publishing criticism of it?

I mean that I would publish some improved version of my brute-force method, since the problem itself is kind of interesting, even if the current paper I'm editing has a weak way of solving it. (They submitted to a very high-profile journal and I think it is not of that caliber)

So you're advising that a paper be rejected because it uses a weak algorithm, and you're asking if it'd be ethical to essentially republish the paper yourself, except with a fix to the problematic condition that'd caused you to recommend its rejection?

No. My algorithm is entirely unrelated to what they're doing. I'm concerned about the fact that it never would have occurred to me to consider this problem if I hadn't read the paper.

If it's unrelated, then there shouldn't be any problems. As a reviewer, many times I've been inspired by what I read. As long one is not stealing ideas or deliberately rejecting a paper to one's advantage, there shouldn't be any problems.

@Nat, the analysis of the brute-force method is not entirely trivial, and I think it is marginally publishable in a suitable journal, albeit a weak one. The actual work is all my own, but the inspiration to study this problem is 100% due to the other authors. As an extreme case, if one of these authors happens to be a referee for my hypothetical paper, I want to know if they have any grounds for accusing me of misconduct, even if I'm not using any of their ideas other than the impetus for the problem.

Yeah, it's hard to tell. I mean the big thing's that it really depends on how much of the value of your future publication would be attributable to the earlier authors in the eyes of a third party. If, as you say, the credit really wouldn't belong to them, then presumably you should be good. But from what's been said here, I'm not entirely confident that that's necessarily the case. Which isn't to accuse you of anything, just, it's not clicking for me.

Just to explain where I'm coming from, I think that you said a lot of it in this: "I mean that I would publish some improved version of my brute-force method, since the problem itself is kind of interesting, even if the current paper I'm editing has a weak way of solving it.". It sounds like they basically came up with an interesting problem, framed it, etc., such that they did most of the work and deserve most of the credit, even if you'd go back and fix a weakness in their solution approach.

Overall, I'm unsure that rejecting the paper's even the right way to go. If it's really an interesting problem, then it seems like it should be published. I'd think that you should talk to the editor about your concerns and ask for clarification on how to proceed; there may be multiple ways forward that'd be better for everyone than crass rejection. For example, maybe you'd revise part of the paper yourself and become a co-author; or maybe they'd omit the weak part of the paper. Or maybe they'd publish as-is, and you'd publish a follow-up later.

@Nat, thanks for your helpful feedback thus far. I should mention that the other authors are submitting to the top journal in the field, and the paper is really not suitable for publication there. I wouldn't mind sharing my result with these authors and working together on submitting to a more appropriate venue, but I worry that they'll be mad at me since I was the mean editor that rejected it. :).

Perhaps don't give them your solution, but point them to what level of journal you think the paper would be appropriate for? There seem to be two problems with the paper: the level of the journal, and that you can do better. Do you need to give both reasons to the authors, if you use both to justify the rejection to other editors?

A safe option is to wait for them to publish the paper and then publish your improvement.

Without knowing more details, my inclination is to agree with @Prof. Santa Claus. However, for me a very important missing detail is whether a preprint exists somewhere, perhaps at arXiv or at the author(s) web pages. If such a preprint exists, and I'd be surprised if it didn't, then I think you can lower your concern (but maybe not eliminate it entirely).

@Nat "It sounds like they basically came up with an interesting problem, framed it, etc., such that they did most of the work and deserve most of the credit, even if you'd go back and fix a weakness in their solution approach." It doesn't sound like that to me at all. To me, it sounds like they came up with the problem and produced a very bad solution to it that needs to be completely thrown away, because it's worse than the simplest possible algorithm (try every possibility in turn until you find one that works). It wasn't a "weakness": their solution was throw-it-away-and-start-again bad.

@DavidRicherby The OP's stated that they believe that the paper's publishable if one algorithm's replaced with brute force. That's a simple fix.

Is it that you are loathe to offer the authors something for free? Yet, they would probably cite "the reviewer" if they come up with an improved version of the paper based on your algorithm. I think it is worth it.

@Nat My impression, as somebody who works in algorithms, is that the algorithm is likely to be the whole point of the paper, so replacing it wholesale and providing analysis of the new algorithm is a total rewrite of the paper (probably upwards of 75%).

@DavidRicherby Definitely, your perspective one make more sense in that case. Just, whatever the paper's really about or whatever it contains, apparently taking out that algorithm and putting in brute-force would make it publishable. I think that prohibits the possibility that the paper was primarily on that algorithm.

In addition to the other commentary, I'd like to add that the context of use can make a world of difference. For example, in the electronics industry that I come from, a good design for a circuit board can be terrible for a silicon chip and vice-versa because of vastly different component costs. So what may seem like a rube-goldberg solution in one (sub)field might be amazingly elegant in another.

Just because their method may not be fastest, it's not a reason to reject them if their method works. The only reasons for rejection should be technical problems or the topic being out of scope of the journal. Let them publish and then publish your own, better version, directly referencing their paper.

Why would you ask rejection just because you have a better idea than what their paper puts forward? Let them publish it. I would think it very unfair in academia if an editor may block a paper, because the editor has a better opinion. Unless there are rules that the editors may not publish papers related or blatantly build up on papers that they've reviewed, then keep your idea to yourself. Then feel free to publish it later. It'd be like reading it independently upon publishing and then writing an independent paper.

Of course I have a VERY DIFFERENT opinion if an "editor" were to just steal my work, while I am choosing to withhold it. Then again, my circumstances are very unique. I would actually even object for them to "build up" on my unpublished work, as they cannot possibly "build up" on it, without establishing claims in my paper... Which, like I just said, I chose to withhold. How about this? After I publish my paper, the "reviewers" may publish their build up. That is, if I can't publish my own build up before they do.

@Dehbop, I don't think they're asking for rejection simply because they have a better idea. But because the method used is inefficient and thus not up to the caliber of the publisher. The original authors can always submit elsewhere.