There are two criteria Pro Tools uses: group size and creation order. If a track is in two active groups and one has more track, that group wins. If the groups have equal numbers of tracks the older group wins.
Capital and Lower Case Group Letters
In the video, you'll see how the group taking priority is displayed in uppercase when it is taking priority over an active group, a helpful visual reminder.