I don't publish in your area, but it seems like you'll have to redefine everything if it's not standard. It's not good for someone to have to have another paper in hand to understand the notation in yours.

@BryanKrause If I choose to do that, how can I write it the way that it doesn't look like plagiarism? Maybe construct a new set of notation? or explain the exact same model in different words? Re-explaining a large model for one more time is not very pleasant...kind of redundant to me. I was hoping there can be a way to re-use the same model/notations...

If your field makes use of appendices or supplementary figures/tables you may be able to put some of the structure there; it really depends on how closely the paper you're working with needs that model to be understood or whether you can merely reference concepts from it and leave the previous paper to explain the details.

How would it be plagiarism if you state what's new in your paper and that you find it convenient to use the terminology in a referenced paper?

@BryanKrause I understand. Since my model and notation set is relatively complex, would it be fine if I just say "In this paper, we follow the notation and system model explained in our previous paper [cite to that paper]"? However, if I have to put the system model and notations in appendices, then do I still need to mention this is from previous paper?

@MichelGokanKhan "do I still need to mention this is from previous paper" Yes, certainly. Like Bruce suggests, I don't think plagiarism per se is the key issue here, or at least it is easily resolved with a clear citation. However, if you get into figures in particular...well, journals can be more persnickety about copying those due to copyright rather than plagiarism. Best to get permission before reproducing figures in another paper, if the original paper is published in a journal (or will be in the future).

@BruceET I meant that if I have to use the exact same explanation for the notations and the system model in my current paper verbatim, then it may be considered as a form of self-plagirism unless I re-word it and do proper citations. That's the question actually: how to do it properly?

@BryanKrause Okay, as in this case I'm citing my own paper, there will be no copyright issue. So if I understood correctly, your recommendation is that I still mention something like "In this paper, we follow the notation blah blah" and put a summary of that notation and system model in appendix with proper citation?

@MichelGokanKhan Even if it's your own paper, sometimes when you publish with a journal you sign some copyright over to them. If it's just your own preprint, then yeah, you should be in the clear. But yes, if you make it explicit like that how you are following from a previous paper, I see no plagiarism concern. Key words like "reproduced from" or "verbatim from" can add to the clarity. Normally you'd want to block quote, but I don't see a need for that for something that fits in a table or list.

@BryanKrause You are absolutely right. In this case, it's an open-access paper and I'm holding the copyright.

Open-access is irrelevant for this issue, but if you mean that it's your own preprint on arxiv then yes, you don't sign away any copyright to arxiv so you're good on that front.

I don't know about your field, but it's common in mathematics to simply refer the reader to a previous paper for undefined terms. For example, even the specific phrase "For terms not defined in this paper" gets over 3 dozen hits in google scholar.

@DaveLRenfro I see. That was really helpful. Thanks for that. I'll certainly look into papers related to CS within the results of your provided search query.

@BryanKrause Okay, I meant "a pure Open Access journal allow the author to retain the copyright in their articles." I'll check that once more with my journal though.

@DaveLRenfro That's partly why I was cautious to post an answer rather than comment, in case there are some field-specific norms here. However, I must say that if I encountered that in a paper I'd find it quite frustrating unless an understanding of those terms was not particularly necessarily in understanding the conclusions of the present paper.

Similarly to the comment by @DaveLRenfro I'll mention that I've encountered a fair number of physics papers leaving out definitions of (some or all) symbols but stating something to the effect of "the notation is that of X et al. [14]". I will say that reading such papers can sometimes be annoying, and that there are good reasons to typically aim for self-contained papers.

@Bryan Krause: In math it's often the case that one paper depends crucially on several other papers, and thus necessary to have copies of those previous papers lying about for reference when trying to go through the later paper. Once a certain "critical mass" of previous papers is reached, the terminology might be considered sufficiently standard in that subfield to no longer merit explicit explanation, although one often winds up citing a well known survey paper or monograph for other reasons anyway (e.g. for historical and mathematical context), which the reader would be expected to consult.

@Anyon I understand your point here. But do you think it would be better to re-explain/re-define/copy-paste those notations and explanations in the new paper though? I don't why, but I feel like the opposite...

@MichelGokanKhan I would err on the side of re-explaining, but I also think it partially depends on how much repeated material we're talking about (page limits might be a problem), and on how likely it is the readers of your new paper will have read the previous one. For example, if it's part of a series of papers likely to be read together, that's different from a paper trying to reach a different audience.

Your second update threatens to turn the question into a "shopping question" subject to closure: academia.stackexchange.com/help

@Buffy Good point! I updated that update with more clarifications!