You want to create a ClutterActor subclass, but don't want it to be rectangular; for example, you want a star-shaped actor.
Use Cogl primitives to draw the actor.
Below is an example of the pick and paint implementations for a star-shaped StarActor class (an extension of ClutterActor).
Like ClutterRectangle, it has a private struct internally, which contains a ClutterColor denoting the color it should be painted. This is used to set the Cogl source color.
static void
star_actor_paint (ClutterActor *actor)
{
  ClutterActorBox allocation = { 0, };
  gfloat width, height;
  guint tmp_alpha;
  /* priv is a private internal struct */
  ClutterColor color = STAR_ACTOR (actor)->priv->color;
  clutter_actor_get_allocation_box (actor, &allocation);
  clutter_actor_box_get_size (&allocation, &width, &height);
  tmp_alpha = clutter_actor_get_paint_opacity (actor)
                * color.alpha
                / 255;
  cogl_path_new ();
  cogl_set_source_color4ub (color.red,
                            color.green,
                            color.blue,
                            tmp_alpha);
  /* create and store a path describing a star */
  cogl_path_move_to (width * 0.5, 0);
  cogl_path_line_to (width, height * 0.75);
  cogl_path_line_to (0, height * 0.75);
  cogl_path_move_to (width * 0.5, height);
  cogl_path_line_to (0, height * 0.25);
  cogl_path_line_to (width, height * 0.25);
  cogl_path_line_to (width * 0.5, height);
  cogl_path_fill ();
}
static void
star_actor_pick (ClutterActor *actor,
                 const ClutterColor *pick_color)
{
  if (!clutter_actor_should_pick_paint (actor))
    return;
  ClutterActorBox allocation = { 0, };
  gfloat width, height;
  clutter_actor_get_allocation_box (actor, &allocation);
  clutter_actor_box_get_size (&allocation, &width, &height);
  cogl_path_new ();
  cogl_set_source_color4ub (pick_color->red,
                            pick_color->green,
                            pick_color->blue,
                            pick_color->alpha);
  /* create and store a path describing a star */
  cogl_path_move_to (width * 0.5, 0);
  cogl_path_line_to (width, height * 0.75);
  cogl_path_line_to (0, height * 0.75);
  cogl_path_move_to (width * 0.5, height);
  cogl_path_line_to (0, height * 0.25);
  cogl_path_line_to (width, height * 0.25);
  cogl_path_line_to (width * 0.5, height);
  cogl_path_fill ();
}If you need more information about how to implement your own ClutterActor, see the Clutter reference manual.
Note that the code in these two functions is virtually identical: the Discussion section suggests how to remove this redundancy.
The above is one approach to creating a non-rectangular actor. But it's also possible to get a similar effect by subclassing an existing actor (like ClutterRectangle) and giving it a non-rectangular appearance. You could do this by making the underlying rectangle transparent and then drawing on top of it (e.g. using Cairo or Cogl).
However, if you then made such an actor reactive, events like mouse button presses would be triggered from anywhere on the underlying rectangle. This is true even if the visible part of the actor only partially fills the rectangle (underneath, it's still a rectangle).
The advantage of using Cogl paths is that the reactive area of the actor is defined by the Cogl path. So if you have a star-shaped actor, only clicks (or other events) directly on the star will have any effect on it.
In the example shown, cogl_path_move_to()
        and cogl_path_line_to() are used. These
        take absolute x and y coordinates as
        arguments, relative to the GL 'modelview' transform matrix; in
        the case of an actor's paint implementation,
        relative to the bounding box for the actor. So if an actor has
        width and height of 50 pixels, and you used
        cogl_move_to (25, 25) in its
        paint implementation, the "pen"
        moves to the centre of the actor, regardless of where the actor
        is positioned on the stage. Similarly, using
        cogl_path_line_to() creates a line segment
        from the current pen position to the absolute coordinates
        (x, y) specified.
The Cogl API also provides various "rel" variants of the path
        functions (e.g. cogl_path_rel_line_to()), which
        create path segments relative to the current pen position (i.e.
        pen_x + x, pen_y + y).
It's important to note that the path isn't drawn until you
        call cogl_path_stroke() (to draw the path segments)
        or cogl_path_fill() (to fill the area enclosed by
        the path). The path is cleared once it's been drawn.
        Using the *_preserve variants of these functions draws
        the path and retains it (so it could be drawn again).
Note that the Cogl primitives API provides other types of path segment beyond straight lines that we didn't use here, including:
Bezier curves (cogl_path_curve_to())
Arcs (cogl_path_arc())
Polygons (cogl_path_polygon())
Rectangles (cogl_path_rectangle())
Rectangles with rounded corners
            (cogl_path_round_rectangle())
Ellipses (cogl_path_ellipse())
One important drawback of using Cogl path primitives is that they will not produce high quality results; more specifically, Cogl does not draw anti-aliased primitives. It is recommended to use the Cairo API to draw during the paint sequence, and the Cogl API to draw during the pick sequence.
If you need more flexibility than is available in the Cogl path API, you can make direct use of the CoglVertexBuffer API instead. This is a lower-level API, but could potentially be used to draw more complex shapes.
The disadvantage of the code above is that the paths are stored in two
        places: once for pick, and once for
        paint. It would make sense to store the
        path in one place and reference it from both of these functions to
        prevent duplication.
Clutter provides a ClutterPath API for storing generic path descriptions. It can be used to describe paths which translate to Cogl or Cairo paths, and can also be used to describe animation paths.
We can use a ClutterPath instance stored
        inside the actor to define the path for pick and
        paint; then, inside those functions, we
        translate the ClutterPath into Cogl path function calls
        (NB ClutterPath is effectively a declarative method
        for defining a path, while the Cogl path API is imperative).
First we add a path member to the private
        struct for the StarActor class (using standard
        GObject mechanisms). The init implementation for
        StarActor creates an empty path:
static void
star_actor_init (StarActor *self)
{
  self->priv = STAR_ACTOR_GET_PRIVATE (self);
  self->priv->path = clutter_path_new ();
  clutter_actor_set_reactive (CLUTTER_ACTOR (self), TRUE);
}One consideration is that the path coordinates need to
        fit inside the actor's bounding box. So as the actor's allocation
        changes, path also needs to change. We can do this
        by implementing allocate for the
        StarActor class:
static void
star_actor_allocate (ClutterActor           *actor,
                     const ClutterActorBox  *box,
                     ClutterAllocationFlags  flags)
{
  ClutterPath *path = STAR_ACTOR (actor)->priv->path;
  gfloat width, height;
  clutter_actor_box_get_size (box, &width, &height);
  /* create and store a path describing a star */
  clutter_path_clear (path);
  clutter_path_add_move_to (path, width * 0.5, 0);
  clutter_path_add_line_to (path, width, height * 0.75);
  clutter_path_add_line_to (path, 0, height * 0.75);
  clutter_path_add_move_to (path, width * 0.5, height);
  clutter_path_add_line_to (path, 0, height * 0.25);
  clutter_path_add_line_to (path, width, height * 0.25);
  clutter_path_add_line_to (path, width * 0.5, height);
  CLUTTER_ACTOR_CLASS (star_actor_parent_class)->allocate (actor, box, flags);
}This clears then adds segments to the ClutterPath stored with the StarActor instance. The positioning and lengths of the segments are relative to the size of the actor when its allocation changes.
The pick and paint
      functions now reference the ClutterPath (only the
      pick is shown below); and
      to turn the path into drawing operations, we implement a
      star_actor_convert_clutter_path_node() function
      which takes a ClutterPathNode and converts it
      into its Cogl equivalent:
static void
star_actor_convert_clutter_path_node (const ClutterPathNode *node,
                                      gpointer               data)
{
  g_return_if_fail (node != NULL);
  ClutterKnot knot;
  switch (node->type)
    {
    case CLUTTER_PATH_MOVE_TO:
      knot = node->points[0];
      cogl_path_move_to (knot.x, knot.y);
      break;
    case CLUTTER_PATH_LINE_TO:
      knot = node->points[0];
      cogl_path_line_to (knot.x, knot.y);
      break;
    default:
      break;
    }
}
static void
star_actor_pick (ClutterActor       *actor,
                 const ClutterColor *pick_color)
{
  if (!clutter_actor_should_pick_paint (actor))
    return;
  ClutterActorBox allocation = { 0, };
  gfloat width, height;
  ClutterPath *path = STAR_ACTOR (actor)->priv->path;
  clutter_actor_get_allocation_box (actor, &allocation);
  clutter_actor_box_get_size (&allocation, &width, &height);
  cogl_path_new ();
  cogl_set_source_color4ub (pick_color->red,
                            pick_color->green,
                            pick_color->blue,
                            pick_color->alpha);
  clutter_path_foreach (path, star_actor_convert_clutter_path_node, NULL);
  cogl_path_fill ();
}The conversion function only covers ClutterPathNode types encountered in this actor.
Instead of converting to Cogl path operations, another alternative
      would be to use the clutter_path_to_cairo_path()
      function to write directly from the ClutterPath
      onto a Cairo context.