I don't know why you got downvoted. Your answer is amazing due to how simply and true it is. Have my +1.

Wonderful and how does this help set a lower limit on an unexplorably large sea? You have pointed out that a different pocket universe may be constructed to house a truly infinite ocean, which is fun, for someone else, but has no bearing on the question asked.

This question makes no sense to me, where did this even come from? What do you mean by "topology"? Topology to me just means a description of the shape and formation of a surface, object or structure (such as a node graph). You clearly mean it to describe something totally different and with out context is not very useful to most readers.

@Ash you stated that your own universe is a pocket one. As such, it can have whatever topology you wish. With a fractal topology, it could be possible that no matter how much one advanced, they would get no closer to their destination. The most well-known fractal is the Mandelbrot set, given by the formula $z^n = C$, where any $z^n$ is a point in a complex plane. Within the set, the further you move, the clearer it becomes that you are going nowhere.

@snb Topology is exactly what you mean. It does not make Konchog's answer wrong or senseless. I'll concede that it does take a little math to understand the concept, but it is not rocket-science. Jonathan Coulton has a nice song about this.

@Renan in such a surface, in what way would it even look like a sea?

@snb it could look just like a regular sea to us. You would notice that you could sail for days and the shore would never seem to get farther - and when you tried to get back to the shore, you would find that it would take as many days to get back to it as you spent trying to get away from it.

@Renan Yes I could have that topology if it was in anyway useful or relevant but suggesting a truly infinite space doesn't help answer the question of how small an ocean can be and still defeat transit exploration.

@Ash in a fractal geometry your ocean could be one hundred meters wide. It would not be traversable.

@Renan Because it wouldn't be 100 metres across in terms of physical traverse it would be infinite in physical surface area, I am fully conversant with fractal topology, it just has no relevance here.

@Ash I disagree. It would be 100 meters across. However, trying to move within those 100 meters would be pointless. You would need to move outside the fractal to go anywhere. If only the ocean is fractal, you would be able to fly over it. You might launch yourself as a human cannonball and it would work. But were you to sail, you would be trapped. An analogy would be someone trying to get to the other side of a black hole by going through it (sailing on the ocean) vs going around it (flying, going under the seabed, etc.).

@Renan Like I said the traverse distance is not 100 metres, it's infinite and as such completely unhelpful.

@Ash I think we could be talking in circles the whole day, and the comments will soon be moved to chat... Still, this is IMO by far the best answer - and from a mathematical point of view, the traverse distance would not be infinite. Not being able to move in the direction you want does not imply in infinite space.

@Renan The distance you would have to travel across an infinite fractal topology to get from one side of it to the other is by definition mathematically infinite, a subset infinity to be sure but still an infinite. The answer specifies an infinite surface area.

It'd seem simpler to just say "the shoreline is a portal to another plane of existence that's infinitely large"; dressing that up in the language of fractals doesn't seem to add much beyond a touch of silliness.

I admit to not having a super-deep background in math, but I don't get it. Sure, the Mandelbrot set is infinitely "deep", but if you move across it in, say, steps of "0.01", you can do that just fine. Aren't shorelines in the real world already subject to fractal-related measuring problems (with the Spain-Portugal border being the canonical story)?

@mattdm only in terms of shore length - a 1d line curved through a 2d plane in such a way as to be infinite. This answer implies a 2d surface curved through a 3d space in such a way to be infinite (just a different 3rd dimension than we're used to)

Ah, I see. But from the point of view of someone in that universe, the fractal nature is entirely irrelevant. It's indistinguishable from "simple" infinity, taking us to @Nat's comment.

A fractal topology makes sense since that pocket reality was created to imprision the Great Old Ones.

This answer appears to be well-received, so I guess the community likes it! I mean, it's weird to me in part because there's no need to talk about fractals to get the described effect and in part because actually having a fractal topology in this context would literally destroy the world (it'd wreck the symmetry behind the conservation of energy and basically boil the oceans over).

@Renan Topology is "Not rocket-science"? No, it's harder!

I'm with @Ash here. OP said: "what I'm wondering is how narrow an ocean might be and still be effectively infinite", but in this answer the ocean is literally infinite, albeit in a finitely large container. Interesting, for sure, but not really an answer to the same question. I read the question as equivalent to "what's the farthest distance man can travel by sea in the Age of Sail?", to which renan's answer still holds, but this one falls apart.