Execute the affine transformation.
M ((3,3)-array) – affine transformation matrix
x, y (float) – 2d coordinates to transform
res – transformed coordinates
float, float