Second Voting Experiment: Not Perfect! Argh!

Our second  round of experimental voting has ended, and we have some pretty interesting data to share about game icons and titles. Unfortunately, a lot of the data isn’t really statistically resilient. I mean, you guys provided a ton of votes — over 35,000 — and almost all of those votes look legitimate. So what’s the problem?

The problem is that the votes were unevenly spread out over 4000 different icons and titles, and most icon/title combos didn’t get the 40+ votes they needed to be particularly reliable… let alone get the 100+ votes we were shooting for. To make matters worse, to answer our core question, I needed to control for other variables, such as Editor’s Rating and game genre. When I did that, it pared the data down so that there was even less left to look at.

This dilution was caused by our desire to make the “experiment” also be immediately useful to developers by showing them an “Icon Hotness Rating” for their games’ icons. That meant that if developers changed icons or added games, new icons were tossed into the mix and old ones stopped getting votes. This diluted the data quite a bit… much more than I expected, because more and more developers used the data to iteratively improve their icons. If we had just picked 1000 existing games and not changed the data during the test, we’d have rock-solid data now. So the whole thing was probably trying too hard to be helpful — it derailed the experiment itself. On the other hand, many people let me know that the “Icon Hotness Rating” was very helpful to them, so it was certainly worthwhile in that respect.

So I’m disappointed at how hard it is to play scientist. (The guys in lab coats make it look so easy.) But we still have some fairly solid data to share.

Icons and Titles — Does Any Of This Matter?

The hypothesis was that if your game has a great icon and title, it will be viewed by more sponsors than a game with a mediocre or poor icon/title. That’s because your icon and title are the most prominent advertising elements FGL uses in emails and webpages.

So did we get enough data to tell if that was true or not? Yes. All other things being equal, a game with a great icon and title will get between 20% and 50% more unique sponsor views than a game with a very poorly-rated icon and title. No matter how I slice up the data, that seems pretty clear.

I can hear your next comment: “But most of the bottom-tier icons are joke icons, or placeholders, or so amateurish that they obviously suck. These results don’t mean anything!” That’s why the number above was calculated only using games that actually got sponsorships during our test period. So all the joke icons, beginner games, and so on aren’t included in these numbers… unless they somehow got a sponsorship of $100 or more. Furthermore, I tried to control for different Editor’s Ratings by only comparing games with the same editor’s score. No matter whether your game got a 7, an 8, or whatever, you will still get more sponsor views if your icon and title are in the top tier of voting rather than the bottom.

And once you get these extra views, do they pay off? Yes! Considering relatively-equivalent games, it looks like:

• Games with top-rated icons/titles sell about 8 days faster than games with very poor icons/titles
• Games with top-rated icons/titles sell for about $1100 more than games with very poor icons/titles
• Games with top-rated icons/titles sell within a month 38% of the time, whereas games with very poor icons/titles sell within a month 18% of the time

These shouldn’t be too surprising, because if you get more views, then it naturally follows that you sell faster and have better offers.

So that clinches our argument. Woohoo! Except.. you’ll notice that I only compared top-tier entries to bottom-tier ones. That’s because the data is too muddy to get a clear picture of the middle ground. How much more effective is a “Pretty Good” icon over a “Great” one? I dunno. It might even be that a mediocre icon is slightly better! (Maybe sponsors get scared away by really great icons, and are more comfortable clicking on slightly imperfect ones…) Seems far-fetched, but the point is we are still missing some pieces to the puzzle.

But for our sanity, we’re going to proceed with the assumption that there’s a curve of diminishing returns: a great icon/title gets more views than a mediocre icon/title, which in turn gets more views than a poor icon/title.

[A note on the rest of this article: the example icons I use below may not have been for a game that sold yet. I was only interested in the icons' ratings, and I didn't think to check anything else. That's because I make a bad scientist. However, this detail should matter very little for the purpose of critiquing the icons and titles.]

What Is A Great Icon and Title?

A great icon is a work of art. It tends to have an attractive central focus element, and a clearly distinguishable “foreground” and “background”. In most great icons, the central element is not fully contained in the scene — parts of it are out of the frame. (This is a common artistic technique to make the central element more exciting.)

As often as not, the icon also includes the game’s title in it. (This may come as a surprise to followers of the first experiment — it turns out that even though the game title is displayed above the icon, having the title in the image is still good, if done well.)

Note that the titles in each of these are attractive and interesting to look at. We will soon see that a poorly-drawn title doesn’t do you any favors.

The subject matter doesn’t seem to matter a lot; the graphical style can vary dramatically as long as it’s attractive; and whether you have the title in your icon or not seems to be ultimately a matter of taste. Compare these two extremely-highly-scoring entries:

versus 

They are quite different, but both are great icons.

What can we say about the titles? Not a lot. I couldn’t easily spot a lot of patterns here, except that:

• MOST great entries have short names of one or two words that fit elegantly overtop the icon
• Great titles never use trademarked elements
• These almost never use “2″ or “II” to indicate they are a sequel
• All “great”-class titles use proper English capitalization (it’s “Medieval Wars”, not “medieval wars”)

What is a Medium Icon and Title?

(As I mentioned earlier, the “medium” area is the hardest to categorize because the data isn’t super solid. Some of these might be in the “great” or “low-quality” areas, and some icons used as high-quality or low-quality examples might be medium icons in disguise. But here’s my best-guess analysis anyway.)

Most “medium” icon/title combinations use the same basic premise as the “great” entries above, but for artistic reasons, they don’t manage to reach the same quality level.

These icons:

• Tend to use a central image that is completely contained in the icon (instead of being partially outside the frame)
• Often have too-complex backgrounds that are hard to parse, or else completely static backgrounds of one color
• Sometimes try to use multiple focus elements, resulting in inadequate detail to each of element
• Often include titles in the icons, and even though they are skillfully done, they don’t quite reach the quality level of the “great” entries
• Are often images right from the game’s play mode. (Gameplay screenshots are never in the “great” category… they’re always medium or poor, depending on how good the game itself looks.)
• Most “pixel art” icons end up solidly in the middle range.

One surprising thing to note is that most gray-themed icons end up being in the medium-quality range, too:

A murky gray background usually can’t rise to the highest quality level. (Though there are a few exceptions.)

And what about the titles of these medium-level entries? Again, I didn’t see a lot of strong correlation, just a few details. The titles:

• Tend to be a few letters shorter on average
• Have a fair number of sequels with “2″ stuck on the end
• Mostly use correct capitalization

What is a Low-Quality Icon and Title?

[I really hope none of the developers whose icons I use here are too upset by their inclusion. It's for educational purposes! But if you want me to remove your icon, let me know and I'll be happy to do so.]

A lot of the low-quality icons are clearly just practice entries, placeholders, and the like. We’ll focus on those that seem to have an actual game associated with them, but it should go without saying that an icon that just says, e.g. “Placeholder” on a white background rates very poorly.

The data gives us several great “DON’T”‘s:

You would probably be surprised at the number of perfectly good games that have bad icons. These games can still sell, but they definitely have a harder time getting sponsorships — and tend to have to settle for lower amounts. Worse, even if you change out a bad icon for a good one later, it is often too late: your original art will have already been used in automated email blasts. So if you’re serious about selling your game, get a good icon before you put it up for bidding on our website!

What Next?

Hopefully there aren’t too many mis-categorized examples in all that. I’m certainly disappointed the data wasn’t more rock-solid. But I think the data was sufficient for its purpose, for now. I don’t plan to do another icon experiment in the near future.

However, we’ve heard from many of you that seeing your game’s individual “hotness” rating was very useful. So I’m exploring ways to make this a permanent part of the website, with a bit more customization and options than the “experiment” versions had. The permanent version will also be collecting data, of course, so eventually we’ll have enough data for all kinds of statistics. It may take a year or longer, but that’s okay!

I hope this has been helpful, and I hope your games reach your sales goals.

We’ve been doing some experiments recently to help nail down exactly what determines the number of sponsor views your game gets. There are a million variables — everything from genre, to market level, to day of the week, to — well, who knows what else? We here at FGL have always had our strong suspicions, but we decided to get more scientific about it. (Why did we do this all of a sudden? Actually it was because of the recent Developer Survey. One of the biggest requests was more information and statistics on how to sell your game.)

First up, we examined the importance icons by themselves. People chose which icons they found “hottest”, and then we correlated that “hotness factor” with how many views the game got. This correlated surprisingly well: it’s very important to have a good icon for your game! But what makes a good icon? Well, you can see the results, along with the actual best and worst icons, here.

We’re now onto the second experiment, and here’s where we need your help. This one is broader: it covers both game icons AND game titles. To get good results for this one, we need a lot of votes! We’ve also incorporated feedback from the first experiment to make this one much more robust and (hopefully) accurate. Have you got a few minutes?

Please go vote on some games!

It’s kind of addictive, and it’s helping gather really useful data!

You don’t need to be an FGL user to vote on icons, so please feel free to spread that link far and wide! After the experiment we’ll crunch the numbers and post the results here on the blog.

Thanks for your time!

Hi guys, FGL_Eric here! I used to put my trends-and-statistics posts in the forums, where they always got lost. Now that we have a blog, putting them here makes more sense. So without further ado, here’s a look back at 2009, especially contrasting it to 2008. All these stats are pulled from FGL’s database of games and sales. It’s important to keep in mind that this is a limited picture of the Flash market, because we’re only watching one of many revenue streams here. But it’s still pretty interesting, so here we go…

How Many Games Get Accepted Bidders?

First up is the most common question newbies ask: what are the odds my game will find a sponsor? Well, first we ask, how many games are there?

• Total games put up for bid during 2008: 3225. During 2009: 4087.

Then we ask how many games got bids they liked?

• Total accepted bids during 2008: 588. During 2009: 1196.

That gives us…

• Number of games that had accepted bids in 2008: 18%. During 2009: 29%.

(Those last numbers aren’t quite accurate since some games have multiple accepted bids, but it’s pretty close.)

This is a great trend! From 2008 to 2009, your odds of a sale on FGL went up by over 50%!

How Much Are Games Going For?

The range of sale prices is HUGE. It can be hard for a few statistics to convey this, but let’s give it a shot:

• Average (mean) accepted bid during 2008: $1295.80. During 2009: $1484.10

But like I said, that really doesn’t tell you much. Let’s break it down by Admin Rating. We give most games a rating between 1 and 10, which some sponsors use as a quick-gauge tool. But it’s also one (semi-arbitrary) way to break up our games by quality level.

• Average sale of games rated 1-4 in 2008: $482. During 2009: $350.
• Average sale of games rated 5-6 in 2008: $444. During 2009: $639.
• Average sale of games rated 7 in 2008: $889. During 2009: $999.
• Average sale of games rated 8 in 2008: $2017. During 2009: $2126.
• Average sale of games rated 9-10 in 2008: $3897. During 2009: $5567. *but see below

So what does all this tell us? The biggest gains are clearly in the highest-quality Flash products, but even medium-quality games are seeing a good growth rate.

It’s crucial to keep in mind that other revenue streams such as ads, microtransactions, and exclusive site-locks are not shown in this data. These other revenue sources can be very big money makers. In general they have a dramatic curve — middling-quality games (those rated 6 or less) tend not to make much money from other sources. Games 7 and above often make a modest amount from ads and a nice chunk of money from non-exclusives. The very best games can make a HUGE amount from each of these revenue streams.

* In fact, the top-end games are the least well represented here, because the best quality games have yet another revenue stream: they get performance bonuses from their sponsors, which can yield a very large amount of money, but aren’t recorded in our database here. It is not uncommon for a game that gets $6k in direct cash to also earn $15k to $20k when they reach performance goals. The best-quality games make a lot of money. But of course, very few games can be “the best quality”. Which brings us to the next topic…

How Good Are These Games?

The last numbers showed us that the top-end games are worth a whole lot more than the other games. So how many games are “top end” (in FGL’s subjective opinion)?

• Number of games rated 1-4 in 2008: 18.3%. During 2009: 10.4%.
• Number of games rated 5-6 in 2008: 49.3%. During 2009: 36.7%.
• Number of games rated 7 in 2008: 24.6%. During 2009: 35.7%.
• Number of games rated 8 in 2008: 6.5%. During 2009: 16%.
• Number of games rated 9-10 in 2008: 1.1%. During 2009: 0.9%.

Some of this data reflects how we rate games — we tend not to use the lowest part of the curve; if a game is so bad that it would get a 1-4 rating, we encourage the developer to go back and work on it some more before we review it. This is a policy we’ve been much more strict with this year, so many fewer games fall in that category.

But more generally, the bulk of games are starting to trend upwards in rating, which is slightly due to an increased pressure on our reviewers to rate games higher, but also points out that sales-quality Flash games are, in general, getting better. The competition is getting tougher, and more and more games are looking really good when they launch.

It’s also important to keep in mind that our ratings are not infallible. We’ve had a game we rated “6″ sell for $8,000. We’ve had a game we rated “9″ sell for only $250. These things happen. I am not trying to say our number is all that important — it’s just the only numerical rating we happen to have that applies to all our games, so it works well for these statistics.

How Big Are Games, in File Size?

One question we see a lot is, “How big are these game files?”

• Average Sold Game Size in 2008: 1.6 mb. In 2009: 2.13 mb.

So games are getting larger on average. But this single data point doesn’t tell a complete story. Here’s another illuminating data-point:

• Average Sold Game Size for Games Rated 8+ in 2008: 2.68 mb. In 2009: 2.76 mb.

This second statistic tells us that the highest-quality games have always been larger than the lower-quality games. But these top-end games aren’t getting a lot larger than they already were, while the lower-quality games are growing larger pretty quickly. (Remember this is a mean average, so the actual size of successful games varies quite broadly — anywhere from a few hundred kilobytes up to ten megabytes, which is the maximum size FGL allows.)

How Big Are Games, in Screen Size?

Another question asked often in the forums is, “What screen size should my game be?” Our data on successfully-sold games shows that the typical successful Flash game has a screen resolution of 640 by 480. There are some much larger and some much smaller, but for both 2008 and 2009, most successful games stuck pretty closely to this screen size.

ActionScript 2 versus ActionScript 3

Yet another constant forum topic is whether ActionScript 2 is dying out. Well, here’s some data for you:

• Percentage of Sold Games That Used AS2 in 2008: 63%. In 2009: 38%.

As you can see by these pie charts, the ratios flip-flopped — in 2008, the significant majority used AS2; in 2009, AS3 took the lead. At 38%, AS2 is not at all dead yet. But it does seem to be dying off. For 2010, that probably means that AS2 games are still fine. By 2011, we may start to see sponsors who only support AS3 (e.g. with their logos and banners and APIs). So our recommendation is to switch to AS3 for new projects, but if you’re working on an AS2 game now, don’t rewrite it — your AS2 games will still sell just fine, for now.

Can I Make This A Full-Time Job?

Sometimes seeing these sorts of statistics can be pretty depressing. The average top-quality Flash game makes only $5500 in sponsorship money? If you have a partner that splits those profits with you, then that may not seem like a lot of money, especially if you’re trying to do this full-time. Before jumping to that conclusion, there’s two key things to keep in mind. First, these are all mean averages, so a couple games in 2008 and 2009 made more than $50k in a single sale. But there are other top-quality games that just didn’t find a big buyer and ended up being sold for only a few thousand. There are many variables in determining a sale price. But if you were to make a few dozen top-quality games, then you would probably find they average around $5500 each, when you factor in the bigger and smaller sales.

The second thing was mentioned earlier, but bears repeating: this is only one of several revenue streams. The higher quality your game is, the more revenue options become available to you. Your top-end game will make some nice money from ads (CPMstar ads can earn you several thousand on a top-end game), and often you’ll have performance deals or bonuses for reaching certain goals your sponsor makes. A game that does well and uses the Primary Sponsorship model can also make several thousand from exclusive-sitelock deals for other portals. Some games fit into the microtransaction model well and can make extra cash with GamerSafe, too. All of these options require effort, but you can often double your money (or quadruple it, for the best-quality games) by following up on all the revenue streams available to you. Our numbers above follow only one source of revenue.

In the end, is it enough money to live on? That depends on lots of things — where you live, if you have a family, and most importantly, how many high-quality games you can make a year. We’re now seeing several small teams who make enough top-quality games to live nicely on the profits. On the other hand, it’s also highly variable — sometimes you may not make much money for months at a time, if a game doesn’t do well — so this a risky full-time job if you don’t have some money to fall back on.

We are also seeing a lot of top-quality games coming from hobbyists — people who spend a whole year on just one game, but they make that game count. This can be a very rewarding and lucrative part-time job or hobby. If you’re a hobbyist, keep in mind that quality is more important than quantity, because if you make a top-quality game, all sorts of money opens up for you, whereas middle-quality games will be stuck with fewer revenue sources. It’s definitely true that making one high-quality game is a lot harder than making multiple medium games, though. And just think, making this into a full-time job requires making multiple high-quality games each year. This requires extreme dedication and a lot of effort. In some ways it’s like painting or novel writing — you can make it a full-time job if you try hard enough, but it has to be your dream job, or you just won’t have the motivation to reach your goal.

The best news these numbers show is that the average sponsor sale price is going up pretty quickly. If this keeps up, we expect to see more and more people able to make this a full-time job in the coming year, and especially in 2011. Of course, game quality is going up just as quickly as sale prices are. It’s now very hard for one person to make a top-quality game, because most people aren’t good at the programming AND the design AND the artwork necessary to make them. Teaming up with another developer is now the best way to get top-end games quickly and efficiently.

The Wonder of Numbers

I could go on with tons more numbers and statistics, but I think these are the biggies — statistics that might help you plan, organize your game, or even consider whether making Flash games is right for you. But if there are other statistics you want to see, please let me know in the comments!