For some users, Xojo 2016 R1 hasn’t been a great release. Some Windows users are experiencing frequent crashes. For others (myself included), working in the WebPage Layout Editor is an exercise in frustration because it’s unusably slow. We blame Xojo, but we, the beta testers, are culpable for these problems as well.
Before you start angrily sending in comments hold your horses. R1 had a really long beta cycle. Clearly, Xojo had issues with figuring out HiDPI for Windows and their work isn’t finished yet (hence the beta tag for HiDPI in Windows). It happens and some times you have to ship an imperfect product. We always have the option of NOT using the latest version of the product.
Some will say that Xojo shouldn’t have released R1 in the state that the Windows version was in. That’s only partially true. Xojo uses their own product all day long to create their product. They, more than many of us, see the warts in their product and I know they try really hard to eliminate as many of those as possible. But they use it differently than we do so I wouldn’t expect them to find every bug. That’s where we come in.
The beta program exists so we can bang on the product using our own real-world projects. It’s up to us to find and report the fringe cases and use the product in different ways that what the IDE expects. I know that in this beta cycle I was swamped with consulting work so I only fired the betas up occasionally. So whatever problems there is on me and on everyone else in the beta program that didn’t test.
I will grant you that the only incentive we have for participating in the beta program is a less buggy product. For some that’s enough but when you’re busy trying to get your own products out the door there’s not much incentive to work even more and find bugs for someone else. In my opinion, this has always been the problem with the beta program.
Some of use are also guilty of not starting testing until the Final Candidate releases. This is actually the worst time to start testing because it means only show-stopper bugs will get fixed. We really need to start testing with the early beta’s. That’s when Xojo needs us the most to bang on their product. Is it fun? No. Is it necessary? You bet because waiting until Final Candidate is worthless to them. To us too, because those bugs might not get fixed for 3 months.
Did Xojo help us out in this cycle? No. There were times they were putting out a new beta every day. Sometimes with only a couple of changes from the previous version. I would download a beta with the intention of working with it and before I could get to it a new beta would be announced in the next couple of days.
I don’t know about the rest of the people in the beta program, but the high frequency of beta releases meant I looked at them less. I don’t know why but it was. I saw the announcements rolling in and I shrugged in apathy. Perhaps I felt the frequent releases didn’t properly value my time. I only have so much time to devote to beta testing and the frequency was off-putting. I like to use a version for 3 or 4 days (or until I find a major bug a report it). With so many releases I just didn’t test.
I propose that with the next cycle Xojo limit their public builds to once a week (maybe a Thursday or Friday release). This respects my time a bit more since my only incentive is to make it a better product. Frequent public releases says that they’ve not vetting their product enough before giving it to us.
I also think there needs to be something in place that identifies what type of beta tester we are. Are you mostly a Windows user? Then Windows specific changes get highlighted for you because you’ll be most likely to see them. Same with Mac OS X, iOS, and Linux users. Desktop, console, and web app users might also be a designation. I realize this doesn’t fit in with the current generic “hey, we have a new beta” announcement. But it’s clear from R1 that the generic approach didn’t work.
We have too many major bugs in this last release to call the beta program a success. It’s not doing what we need it to do. So how do we structure is to so that our time is respected and we actually report bugs before Final Candidate stage?
What about you? What suggestions do you have to help us and Xojo out at the same time?