Tag Archives: Beta Program

Beta Program Incentives

Posted on by

I’ve been part of the beta program for over a decade.  Many cycles I bemoan the fact that I’ve found very little time to real testing before its released.  That’s bad and I suspect many of my fellow members of the beta program feel much the same way.

The incentive to be in the beta program is simple.  See the new features first and help find bugs and along the and way offer some advice and guidance on how new features work.  And in general, make the product better.  That is an incentive and it’s a laudable one but many cycles that’s not enough.  Perhaps there’s a way to give us more incentives.

I fight with finding time to beta test.  I’m often working under deadlines with real projects that I can’t afford to release a version with the beta.  There’s no way around that.  Perhaps there is something that Xojo can do to make it worth my while to do more testing.

What if Xojo gave a percentage off, say 5%, off a license purchase (either new or upgrade) if I hit some threshold of credible bug reporting?  I spend a lot of money on licenses so that might be a good incentive for me (or at least one of my developers) to work with the beta on a regular basis.

Obviously there would need to be ground rules to prevent abuse.  Bug reports must be well defined with demonstrable repeatability, example project, video, etc. to make sure there’s not a flood of crap.  After it’s reviewed by a Xojo engineer that bug report gets applied to our account.  Then when it comes time for renewal we can apply the discount we’ve earned.  Perhaps that means a percentage point for every five credible bug reports (not feature requests) during a release cycle with a maximum percentage.

Make a game of it.  Reward outstanding bug reporters with recognition.  Make the participant list public so everyone is aware that so-and-so is participating and finding a lot of bugs.  Maybe reward the yearly ‘winner’ with free or discounted admission to XDC.  As a person always on the lookout for new Xojo talent this might be a good way find someone as well as having a little fun with it.

I see this as a way to get new users involved.  For me a license is no big deal but for some it’s huge and if that user has extra time (or at least more than me) that will give them extra incentive to submit bug reports.

I get it it.  The beta program should be a win-win for Xojo and us users.  The reality is that I feel like we’re free labor to a certain extent.  I don’t feel much love from Xojo even though I *know* they do appreciate whatever beta testing we do.  Offer some discounts, make it a game.  Do something to entice us!

What are your thoughts?

Change (Not Really) To the RS Beta Program

Posted on by

Today Real Software announced to the Beta List that they were going to make some changes.  The notice, in part, says:

In the next few weeks, we will be updating the process that manages our mailing lists.  We will begin using a system that ensures everyone on this list has a current Real Studio license and that the email address subscribed to this list is one of the email addresses in your realsoftware.com account.

Of course, some on the existing beta list thought this was a horrible idea.  Doesn’t Real Software want beta testers?  Blah, blah.  It was just shy of outrage.

Well, here’s the deal:  Roughly half of the people on the beta list don’t have a current license for Real Studio.  This means that they can’t REPORT a bug even if they found something because the Feedback system requires a current license!

The Real Software beta list policy has been pretty wide open since I’ve been using REALbasic.  All you had to do was sign up for the beta program.  After that you could access the download URL’s for the beta versions.  It wasn’t until recently that you had to log in to the Real Software website to download beta versions.

Really all this change does is enforce a logical limitation.  Frankly, the beta list has needed to be pared for a long time.  There’s often a lot of inconsequential chatter that doesn’t help me as a beta tester nor helps Real Software fix anything.

Issuing beta builds is good for Real Software as we tend to use the product wildly different then they do.  Don’t get me wrong, they build the Real Studio IDE using Real Studio so they do eat their own dog food but they only have a handful of products in a limited number of use-cases.  We tend to use the product in abnormal ways either by design or by ignorance.

I wonder if even enforcing the current license restriction goes far enough.  The point of being part of the beta program is to report bugs and suggest changes.  I could probably name two dozen people that are vocal on the beta list that I’d want on MY beta list if I had choice.  Those are the people you want to keep.

Likewise, I know of a handful of people on the beta list that I’d find a way to politely leave off the new beta list.  Not that they’re wrong they just tend to be unreasonable in their reactions to beta software bugs.  When you’re unreasonable then engineers tend to ‘rooster up’ and resist the change.  Then we all lose and that’s no good.

Perhaps a good way to make a quality beta program is to start over with the list of current people that a) have a current license; and b) have submitted a Feedback report in the past 12 months.  I am almost afraid to see how low that number is.

I think this policy enforcement is probably a good thing before the 2012 Release 2 hits.  This is the release where the licensing gets simplified (for most of us).  So this means, potentially, a lot of new people using Real Studio for the first time.  So unless they get a paid license, they won’t be able to participate in the program.  Of course Real Software hasn’t announced anything on what happens to the beta program for R2 yet but I believe this to be a reasonable interpretation given the enforcement of the current policy, don’t you?

For me, this changes nothing.  Perhaps with less chatter RS can help focus our energies a bit better and if we do have something to report our voices don’t get drowned out.  I think this policy enforcement is a very good thing.

Anyway, what do all of you think about this change?  Is it fair or unfair?  Do you think it’s a good, bad, or do you even care?

Beta Program Ruminations

Posted on by

Earlier in the week I mentioned that I thought the REAL Software beta process is broken.  I’m a passionate user and I use RB all day, every day, so I have some strong opinions.  I happen to know a thing or two about testing on a commercial software product.

Earlier in my software development career I was the lead tester for a printer utility (for a very large printer company).  I was in charge of the test scripts (imagine the same test on 30 printer models for each beta release – yeah it was mind-numbingly dull) that each tester used to report bugs.  We’d find a bug in Test 3.13b (how to reproduce), explain what the error is and sent it to the developer in charge (via the bug tracking system).

The developer would fix it and then the build developer would put together the list of changes for that build and the system would then send the bug back to the original tester for verification.  If it passed our test (with the next build) we told the system that the fix was verified.  If not, it got sent back to the developer.

Then, when the developer made a public release, the fixed and verified bugs were put into the change list.  Depending upon how late in the process we were, verified bugs were listed as known bugs so the public testers didn’t report the same bug a billion times.

It’s a tedious process but it’s the only way to really do a beta program.  If you assume that you really want a quality (good enough?) product you need to slow it down and be tedious about it.

So why do I think that the RS beta program is broken?  First, in this release, several bugs were listed as fixed and were clearly not.  This, to me, says that there is no verification process on fixed bugs, or if there is, it’s not a very stringent one.  I understand how this happens because on small teams everyone is maxed out.

It could also be that the bug, as described, was fixed but it didn’t fix the overall problem.  I could easily see this happening.  The developer has a long list of things to do, looks at the bug report, fixes it, verifies it to his or her satisfaction and marks it as fixed.  This, all without a deeper look at why the bug is occurring.  I understand because it’s happened to me.

I would also posit that that the beta program, as it exists, doesn’t work the way that benefits REALbasic (and us end users) the most.  Bugs are getting introduced into the product.  Bugs aren’t getting fixed.  New features don’t get tested properly and take several releases to get working properly.

The beta program asks members to test each build against their projects.  Here’s the ugly truth:  When you ask me to test ‘everything’, it’s like asking me to test ‘nothing’.  There are a couple of dozen of controls that can be used in millions of different ways.  There are hundreds of REALbasic classes that can be used in an infinite number of ways.  Telling me to test my project against the new version only catches in-your-face, or easily noticeable, bugs.

Yes, there is a change list for each beta version but they never tell the beta list what the changes are per new beta build.  Several developers take the time to parse through this list and then publish what’s changed, but why are the testers doing this and not RS?  They should know what has changed in each release and publish the list based on beta build not just an overall list.

ARBP did a survey late last year asking about the beta program.  Most developers said they did it for early access to the next release.  This is akin to saying, “We are part of the program to make sure it doesn’t muck with my project.”  Sure, they test, but is it what RS really needs?

My recommendations.  Not in any particular order and some are mutually exclusive:

1)  Since the beta program isn’t producing the feedback RS needs/wants early enough, scale back on bug fixes and new features and do more internal testing for each release.

2)  Provide guidance on each beta build as to what to focus testing on.  If the listbox received a lot of work, then say that.  As a beta tester I should focus on the listbox.  With Cocoa receiving a ton of changes for each build, it would be helpful to know what the developer wants tested.

3)  Scrap the program entirely and rebuild it by invitation only.  This ensures quality testers and a good mix of hardware, operating systems, projects types, time commitment, etc.  Perhaps even do groupings of people to focus on different aspects of the product in each build.  Group A does controls in this release while Group B focuses on a framework or whatever and in the next release you reverse it.  It ensures that there are always different people looking at different things.  The key here is having as many eyes looking at as many things as possible.  This gets rid of the tire kickers too that provide no valuable feedback.

4)  Have a single person in charge of the beta and build process.  Give that person the authority to delay a public release if beta feedback is too negative or bugs are found at the last minute.  Don’t push the product out the door “just because”.  If there is a legal commitment (for whatever reason) to release on a certain date, then there must be proper time for the testers to vet out problems.

5)  Enforce a proper feedback loop.  Proper discussion needs to take place, both internally and externally, before major things get worked on.  We, users, have a certain set of expectations about features and when no one gets our expectations on record then we end up being overly disappointed in a feature we can’t/won’t use.  We, the users, are the biggest marketing arm of REAL Software.  Keeping us happy makes for happy reviews and comments in public forums.

6)  Don’t ignore beta feedback by saying we’re not the typical RB user.  Um…yeah, we are.  We care enough about the product to give you our time – for free.  Yes, we’re in it for our own interests but don’t dismiss our thoughts as not being representative.

Thoughts?  What would you change to the beta program, if anything?