(Flash CS3) Masking using ‘donut’ shapes within movie clips or with AS3 – it’s possible, and not too bad either

Last Updated on February 26, 2019

You may have tried creating a shape that has a hole in it for a mask in Flash, and it may not have worked. Or maybe it did work and sometimes it didn’t, and you grew some extra gray hairs and threw your laptop at a wall. Or maybe you’re thinking “what the hell is she talking about now?”. I’ll start from (almost) the beginning [if you’re not sure about making masks in Flash, read this super old tech support article.. it hasn’t changed].

More after the jump.

So a mask can have a shape cut out in the middle, and if you’re working on the Timeline (not with code) and your mask is a shape (not a movie clip), everything is fine. By “shape cut out in the middle” I mean a ‘merge cut’ in Flash: took a shape (say, a copy of what you’re masking), put it over another bigger and different colored shape, and then selected and deleted the smaller one to cut that shape out of the middle of the bigger one.

(click above to enlarge)

This is a common thing to do when creating masks like this, because you can create a “reverse mask” of whatever you’re masking, which is pretty useful. Check out this file and SWF to see what I’m talking about:

(click above to enlarge) File: masking-donuts.fla.

Now say you convert that shape to a movie clip… it looks fine on the Stage, but when you go and publish your SWF you’re in trouble.

(click above to enlarge) File: masking-donuts2.fla

Doh! Yeah, you may have run into this one. Similarly, give your mask the instance name of masker and what you’re masking an instance name of maskee and add this line of AS3 to frame 1 of the timeline:

maskee.mask = masker;

Doesn’t work either. (but hey, AS3 that short and sweet is almost doable eh)

So the player doesn’t support that way of creating a mask for now, but all hope is not lost. There are other ways to create your mask where it does work in this situation. In other words, Flash seems to care about the way you make that donut hole in the middle of your mask. And if you do it a certain way, it actually will work regardless of the movie clip.

Let’s check that out.

1. I go into the FLA (masking-donuts2.fla) and double click my mask that doesn’t work (unlock the layer).
2. Take the Lasso tool, make sure Polygon mode isn’t turned on, and drag around the edge of my cut out center. You have to be careful to cut around all the edges entirely.

3. Then Publish.

And now it works. Similarly, remove the masks and add the ActionScript – now it works there too.

Sweet! It’s a clip with a hole. File: masking-donuts3.fla

Whaaat, you may ask? Weird! you may exclaim. Essentially what’s up is that these kinds of masks (with holes in them) created using merge cut or the Eraser tool do not work as a movie clip mask (and hence with code). Made other ways, they tend to work.

So what are the other ways, you ask? Good question. Here are at least some of the other ways you can create shapes that do work as masks:

• (so what is above…) Deleting the middle of the shape using the Lasso tool. Do note that Polygon mode does not work.
• Hand drawing your mask with a brush.
• Drawing shapes around a center area (pretty much same as above – drawing tools). For example, you can draw for squares around an area, leaving the center empty. This will work.
• Broken apart text, such as the letter “O”.
• Create an oval with a stroke, remove the center of that oval (just select and delete). Select the stroke and then choose Modify > Shape > Convert Lines to Fills.
• Paste any shape like a square from Illustrator (or likely another program) as a bitmap, and then make your merge cut in that shape IN FLASH. If the outer edge of your shape doesn’t matter (like you just need a big square with your Flash shape cut out of the middle), this is probably the best workaround.
• And finally, bringing in a shape from somewhere else (Illustrator). If you have a shape that includes interior “merge cut” shapes in your FLA that you need to be exactly the same, this will give you EXACTLY what you want. I’ll describe in detail below.

So there are three ways you can work between Illustrator (or some other graphicy software) and Flash with a merge cut shape you want to use as a mask. After all, you probably have your artwork in Flash and you need that specific shape you already have as the mask (or you need it to start with and then modify). Which one you use depends on your shape and what you need.

A simple shape: You can copy the shape that doesn’t work for the mask in Flash, paste it in Illustrator, copy it in Illustrator, and paste it back into Flash. However, In some cases with Flash CS3, and particularly with curves, the shape may not look the same after pasting into Illustrator. If that’s a problem, then you need to use one of the other workarounds below.

Simple outer shape (such as a square), complex inner cutouts: Paste your square from Flash to AI, then paste it as a bitmap in Flash. Break apart and make your shape merge cutouts using whatever Flash assets you want.

You have a complex shape in Flash:

1. Export your shape from Flash as a PNG making sure to choose Match Screen so the image isn’t huge.
2. Place it in Illustrator
3. Copy and paste it back to Flash
4. Break it apart
5. Use the Magic Wand to remove transparent areas. NOTE: It helps after doing this to select the remaining shape and give it a new fill color. You may have some remaining bits to delete. This also makes it much easier to make further modifications to your shape. The freaky thing here (as noted above) is you can actually merge cut the shape at this point… and it will work.

Now you can have a mask with a merge cut hole in the middle that looks exactly the same, and use it as a movie clip and/or with ActionScript.

Refresh on what doesn’t work:

• Cutting out the middle using shape merge.
• Cutting out the middle using Eraser tool.
• Cutting out the middle using Polygon mode of the Lasso.

So is this really a workaround for “real life”? Totally, especially using something like Illustrator (thank you Illustrator and ‘integration’ – although this is just a bitmap so anything like this should work). Even without Illustrator it’s not too bad — just using the Brush tool, I managed to make it work pretty well for a recent animation. Used the brush tool to carefully draw around the shape I wanted cut out in the center (accomplished fairly easily by edit-in-place mode), and smoothed and dragged the inside edges of the shape until it was pretty much what I wanted.

So yeah, it is a little bit more work and since I use Flash, I like to complain about the quirks. But you gotta admit, it is better than nothing at all and not too bad right?

And if you do notice other stuff related to this (or not, doesn’t matter), like some other way better workflow or trick or something else that doesn’t work, please let me/us know in the comments.

Over and out.