Why do you want this capability? There are several reasons you might want to create an odd-shaped window. The classic example is a circular clock. What if you were writing a flowcharting program? You could make each flowchart element a separate child window of your main window. By using SetWindowRgn(), you can make the windows into any shape you like.
Here's some example code that sets up window to be triangular (sort of; see below).
void SetTriangleWin(HWND w)
{
HRGN rgn;
POINT points[5];
RECT wrect;
SetWindowRgn(w,NULL,FALSE);
GetWindowRect(w,&wrect);
points[0].x=points[0].y=points[1].x=0;
points[1].y=GetSystemMetrics(SM_CYMENU)+GetSystemMetrics(SM_CYCAPTION);
points[3].x=points[4].x=wrect.right-wrect.left;
points[4].y=0;
points[3].y=GetSystemMetrics(SM_CYMENU)+GetSystemMetrics(SM_CYCAPTION);
points[2].x=(wrect.right-wrect.left)/2;
points[2].y=wrect.bottom-wrect.top;
rgn=CreatePolygonRgn(points,5,WINDING);
SetWindowRgn(w,rgn,TRUE);
}
The region used is like a rectangle (to contain the caption bar and
title) on top of a triangle (that encompasses the client area). The example
program calls the function when the window is created and when the window
resizes. Notice that the region prevents you from operating the resize
border except at the apex of the triangle.
When you move or resize the window, you will see a rectangle outline
instead of the shape you've set.