Solve A^2+b^2=(Ab)^2

$$A ^ { 2 } + b ^ { 2 } = ( A b ) ^ { 2 }$$

Solve for A (complex solution)

$$A=-i\left(1-b^{2}\right)^{-\frac{1}{2}}b$$

$$A=i\left(1-b^{2}\right)^{-\frac{1}{2}}b\text{, }b\neq 1\text{ and }b\neq -1$$

Solve for b (complex solution)

$$b=-i\left(1-A^{2}\right)^{-\frac{1}{2}}A$$

$$b=i\left(1-A^{2}\right)^{-\frac{1}{2}}A\text{, }A\neq 1\text{ and }A\neq -1$$

Solve for A

$$\left\{\begin{matrix}A=\sqrt{-\frac{1}{1-b^{2}}}b\text{; }A=-\sqrt{-\frac{1}{1-b^{2}}}b\text{, }&|b|>1\\A=0\text{, }&b=0\end{matrix}\right.$$

Solve for b

$$\left\{\begin{matrix}b=\sqrt{-\frac{1}{1-A^{2}}}A\text{; }b=-\sqrt{-\frac{1}{1-A^{2}}}A\text{, }&|A|>1\\b=0\text{, }&A=0\end{matrix}\right.$$