You can say about a function that it is (or isn't) a bijection.
I do not know what you mean when you say that a subset is a bijection.
If $G$ is a group and $H\subseteq G$ is a non-empty subset, then it is a subgroup
if and only if it is closed both under the operation and under the inverses.
$$(\forall a,b\in H) ab\in H, a^{-1}\in H$$
Or you can put these conditions into one: $$(\forall a,b\in H)ab^{-1}\in H.$$
However, if $H$ is finite, it is enough to verify that it is closed under the group operation. (It follows automatically that it is closed under inverses, too.)
What you wrote is one possibility how one can define the quotient group. (If we define it in this way, we'll need to show that the operation is well-defined.)
One could define a quotient group using congruence relations. Or one could use different meaning for $Ha\cdot Hb$ - namely as multiplication of subsets.
But I would recommend to stick with the definition you've been using. I am afraid it will only get more confusing if several various definitions are thrown at you at the same time.
@JackRod And what text are you reading?
Moreover, I am afraid that explaining congruence relations would take a long time.
Sorry, I'll have to leave. I might have a look at this room later. (But maybe somebody might notice that there was a new activity in chat and respond here.)
Any case, this should not really be that difficult - in fact, it is a bit similar to the example with Z/4Z which is one of the examples in Gallian's book. (Although this is closer to 3Z.)
First thing first.
H={0,3,6,9} is a subgroup of Z_{12}. (BTW you have a included redundant brackets there by mistake.)
The group Z_{12} is commutative - so every subgroup is normal.
We get three cosets \begin{align*} 0+H&=\{0,3,6,9\}\\ 1+H&=\{1,4,7,10\}\\ 2+H&=\{2,5,8,11\}\end{align*}
And then we can form the group table which looks rather similar to $\mathbb Z_3$.
So in this case we have $G/H\cong (\mathbb Z_3,+_3)$.
As a side note, once we know that $|G/H|=3$, this already means that it is isomorphic to $\mathbb Z_3$.
In the part about Lagrange's theorem, Gallian mentions that any group of prime order is cyclic.
And a cyclic group of order $n$ is isomorphic to $\mathbb Z_n$.
Of course, the fact that 3 is a prime number and thus we know this is Z3 is only tangential here - you have asked about this mainly as an example of a quotient group.