Chroma Swap

You have two arrays $A = A_1, A_2, \ldots, A_N$ and $B = B_1, B_2, \ldots, B_N$. Each of these elements also has a color associated with them, which is an integer. This is denoted by the arrays $ColorA_1, ColorA_2, \ldots, ColorA_N$, and $ColorB_1, ColorB_2, \ldots, ColorB_N$. In a single operation, you can swap any element of array $A$ with any element of array $B$, if they have the same color.