We Are Part of the Problem

For some users, Xojo 2016 R1 hasn’t been a great release.  Some Windows users are experiencing frequent crashes.  For others (myself included), working in the WebPage Layout Editor is an exercise in frustration because it’s unusably slow.  We blame Xojo, but we, the beta testers, are culpable for these problems as well.

Before you start angrily sending in comments hold your horses.  R1 had a really long beta cycle.  Clearly, Xojo had issues with figuring out HiDPI for Windows and their work isn’t finished yet (hence the beta tag for HiDPI in Windows).  It happens and some times you have to ship an imperfect product.  We always have the option of NOT using the latest version of the product.

Some will say that Xojo shouldn’t have released R1 in the state that the Windows version was in.  That’s only partially true.  Xojo uses their own product all day long to create their product.  They, more than many of us, see the warts in their product and I know they try really hard to eliminate as many of those as possible.  But they use it differently than we do so I wouldn’t expect them to find every bug.  That’s where we come in.

The beta program exists so we can bang on the product using our own real-world projects.  It’s up to us to find and report the fringe cases and use the product in different ways that what the IDE expects.  I know that in this beta cycle I was swamped with consulting work so I only fired the betas up occasionally.  So whatever problems there is on me and on everyone else in the beta program that didn’t test.

I will grant you that the only incentive we have for participating in the beta program is a less buggy product.  For some that’s enough but when you’re busy trying to get your own products out the door there’s not much incentive to work even more and find bugs for someone else.  In my opinion, this has always been the problem with the beta program.

Some of use are also guilty of not starting testing until the Final Candidate releases.  This is actually the worst time to start testing because it means only show-stopper bugs will get fixed.  We really need to start testing with the early beta’s.  That’s when Xojo needs us the most to bang on their product.  Is it fun?  No.  Is it necessary? You bet because waiting until Final Candidate is worthless to them.  To us too, because those bugs might not get fixed for 3 months.

Did Xojo help us out in this cycle?  No.  There were times they were putting out a new beta every day.  Sometimes with only a couple of changes from the previous version.  I would download a beta with the intention of working with it and before I could get to it a new beta would be announced in the next couple of days.

I don’t know about the rest of the people in the beta program, but the high frequency of beta releases meant I looked at them less.  I don’t know why but it was.  I saw the announcements rolling in and I shrugged in apathy.  Perhaps I felt the frequent releases didn’t properly value my time.  I only have so much time to devote to beta testing and the frequency was off-putting.  I like to use a version for 3 or 4 days (or until I find a major bug a report it).  With so many releases I just didn’t test.

I propose that with the next cycle Xojo limit their public builds to once a week (maybe a Thursday or Friday release).  This respects my time a bit more since my only incentive is to make it a better product.  Frequent public releases says that they’ve not vetting their product enough before giving it to us.

I also think there needs to be something in place that identifies what type of beta tester we are.  Are you mostly a Windows user?  Then Windows specific changes get highlighted for you because you’ll be most likely to see them.  Same with Mac OS X, iOS, and Linux users.  Desktop, console, and web app users might also be a designation.  I realize this doesn’t fit in with the current generic “hey, we have a new beta” announcement.  But it’s clear from R1 that the generic approach didn’t work.

We have too many major bugs in this last release to call the beta program a success.  It’s not doing what we need it to do.  So how do we structure is to so that our time is respected and we actually report bugs before Final Candidate stage?

What about you?  What suggestions do you have to help us and Xojo out at the same time?

REALbasic Beta List Conundrum

I’ve been part of the REALbasic Beta List for a few years.  I got involved  back in the early days of the new IDE because I was tired of finding bugs in new features, and in a few cases, new features that never made it out into the public release (though documented as such).  So I make a point to kick the tires of the new features and I generally do this by writing them up as a quick review article that I’ve been posting on the ARBP site.

I’m now testing the 2009 R4 release and I realize that I started feeling guilty about not testing more (a new feature in R4 is one I’ve been asking for a long time).  After talking it through with another developer I came to the conclusion that I’m donating my time to Real Software.  Yes, donating because, as a consultant, I get paid good money to do coding and testing for clients and since I’m not compensated from RS in any way, it is pretty much a donation.

I guess one could argue that I get compensated with a less-buggy, better new features, product.  That’s not very motivating to me, somehow, mainly because I’m paying really good money for that product for the privilege to do that!

Then I came to the conclusion that if I were to be compensated in some way I’d do more testing because it’s worth more to me.  Now I’m not suggesting a free copy of REALbasic because that wouldn’t be economically feasible.  But it does strike me as somewhat unfair that I’m spending time beta testing a commercial product.

I’m not sure what would make it seem more fair.  A discount on upgrades?  A higher support priority?  A t-shirt or coffee mug?  Public recognition?  I dunno.  Certainly the problem with any sort of compensation is how to enforce it and qualify it.  If I log one bug is that enough?  What if that bug is really a feature request?  If I report a hard crash is that more important than a misspelled word?  See what I mean?  It’s a slippery slope at the very least.

Do you think the beta program works?  How do open source projects recognize their big contributors?  Do participants need some compensation?  If so, how do you compensate the one-time reporter versus the consistent reporters?  What are your thoughts?