Solve system of equation in x and y

and obviously (0,0) also

The trivial solution is (0,0).
Next we assume x≠0, y≠0.
The two equations can be rewritten as
x(x^3-y^3) + y^2 -\frac{9}{8}x=0
y(y^3-x^3) + x^2 -\frac{9}{8}y=0
That is
x^3-y^3 + \frac{y^2}{x} -\frac{9}{8}=0
y^3-x^3 + \frac{x^2}{y} -\frac{9}{8}=0
Subtracting gives
2(x^3-y^3)+\frac{y^2}{x}-\frac{x^2}{y}=0
that is
(x^3-y^3)\left(2-\frac{1}{xy}\right)=0
Therefore either x³ = y³ or xy = 1/2
If y³ =x³, we get
y = x,\ \omega x,\ \omega^2 x
where \omega is a complex cube root of unity. With this condition, the first of the original equations gives
y^2=\frac{9}{8}x
The non-trivial solutions are:
• for y = x, (9/8, 9/8)
• for y = ωx, (98ω, 98ω²)
• for y = ω²x, (98ω², 98ω)
On the other hand, if xy = 1/2, the first of the original equations gives after simplification
8x^6-9x^3+1=0
which gives x³ = 1, and x³ = 1/8 whence it follows that
x= 1, ω, ω² or x=12, 12ω, 12ω²
Thus in total we get the following 10 solutions
• (0,0)
• (9/8, 9/8)
• (98ω, 98ω²)
• (98ω², 98ω)
• (1,12)
• (ω, ω²2)
• (ω², ω2)
• (12,1)
• (ω2, ω²)
• (ω²2, ω)

@nishant sir,
can you please mail me the pdf which you were talking about here , http://targetiit.com/iit-jee-forum/posts/rmo-2009-12793.html

my id : rishabh.bohara20@gmail.com

Adding the two given equations gives (x+y)^4+4x^2y^2-5xy(x+y)^2+(x+y)^2-2xy=\frac{9}{8}(x+y)

Similalrly subtracting gives -(x+y)^3+xy(x+y)+(x+y)+\frac{9}{8}=0.........(\tex{*}) provided x≠y.

Now putting the value of 98 from (*) in the 1st eqn, we have \boxed{x^2+y^2=-xy}

From the boxed relation we get that if (x,y)\ne (0,0), then either of x or y is negative - which means we can have (x,y)=(x,-y') WLOG

Putting this in the second given relation gives y'=0.

The conclusion follows.

[1]