Assuming you mean [tex]f(x,y) = xy^2[/tex] over the domain
[tex]D = \left\{(x,y) ~:~ x\ge0 \text{ and } y\ge0 \text{ and } x^2 + y^2 \le 3\right\}[/tex]
we first observe that [tex]f(x,y) = 0[/tex] for all [tex](x,y)[/tex] on the coordinate axes.
There are no critical points elsewhere in the interior of [tex]D[/tex], since
[tex]\dfrac{\partial f}{\partial x} = y^2 = 0 \implies y=0[/tex]
[tex]\dfrac{\partial f}{\partial y} = 2xy = 0 \implies x = 0 \text{ or } y = 0[/tex]
Parameterize the circular arc boundary by [tex]x=\sqrt3\cos(t)[/tex] and [tex]y=\sqrt3\sin(t)[/tex], where [tex]0\le t\le\frac\pi2[/tex]. Then
[tex]f(x(t), y(t)) = g(t) = 3\sqrt3 \cos(t) \sin^2(t) = 3\sqrt 3 (\cos(t) - \cos^3(t))[/tex]
Find the critical points of [tex]g[/tex].
[tex]g'(t) = -3\sqrt3 \sin(t) + 9\sqrt3 \cos^2(t) \sin(t) = 0[/tex]
[tex]-3 \sin(t) (1 - 3 \cos^2(t)) = 0[/tex]
[tex]\sin(t) = 0 \text{ or } 1 - 3 \cos^2(t) = 0[/tex]
[tex]\sin(t) = 0 \text{ or } \cos^2(t) = \dfrac13[/tex]
[tex]\sin(t) = 0 \text{ or } \cos(t) = \pm\dfrac1{\sqrt3}[/tex]
In the first case, we get
[tex]t = \sin^{-1}(0) + 2n\pi \text{ or } t = \pi - \sin^{-1}(0) + 2n\pi[/tex]
where [tex]n[/tex] is an integer; the only solution on the boundary of [tex]D[/tex] is [tex]t=0[/tex] corresponding to the point [tex](\sqrt3,0)[/tex].
In the second case, we get
[tex]t = \cos^{-1}\left(\dfrac1{\sqrt3}\right) + 2n\pi \text{ or } t = -\cos^{-1}\left(\dfrac1{\sqrt3}\right) + 2n\pi[/tex]
with only one relevant solution at [tex]t=\cos^{-1}\left(\frac1{\sqrt3}\right)[/tex] corresponding to [tex](1,\sqrt2)[/tex].
In the third case, we get
[tex]t = \cos^{-1}\left(-\dfrac1{\sqrt3}\right) + 2n\pi \text{ or } t = -\cos^{-1}\left(\dfrac1{\sqrt3}\right) + 2n\pi[/tex]
but there is no [tex]t[/tex] in this family of solutions such that [tex]0\le t\le\frac\pi2[/tex].
So, we find
[tex]\min\left\{xy^2 \mid (x,y) \in D\right\} = 0 \text{ at } (0,0)[/tex]
(but really any point on either axis works)
[tex]\max \left\{xy^2 \mid (x,y) \in D\right\} = 2 \text{ at } (1,\sqrt2)[/tex]
