We Are Part of the Problem

For some users, Xojo 2016 R1 hasn’t been a great release.  Some Windows users are experiencing frequent crashes.  For others (myself included), working in the WebPage Layout Editor is an exercise in frustration because it’s unusably slow.  We blame Xojo, but we, the beta testers, are culpable for these problems as well.

Before you start angrily sending in comments hold your horses.  R1 had a really long beta cycle.  Clearly, Xojo had issues with figuring out HiDPI for Windows and their work isn’t finished yet (hence the beta tag for HiDPI in Windows).  It happens and some times you have to ship an imperfect product.  We always have the option of NOT using the latest version of the product.

Some will say that Xojo shouldn’t have released R1 in the state that the Windows version was in.  That’s only partially true.  Xojo uses their own product all day long to create their product.  They, more than many of us, see the warts in their product and I know they try really hard to eliminate as many of those as possible.  But they use it differently than we do so I wouldn’t expect them to find every bug.  That’s where we come in.

The beta program exists so we can bang on the product using our own real-world projects.  It’s up to us to find and report the fringe cases and use the product in different ways that what the IDE expects.  I know that in this beta cycle I was swamped with consulting work so I only fired the betas up occasionally.  So whatever problems there is on me and on everyone else in the beta program that didn’t test.

I will grant you that the only incentive we have for participating in the beta program is a less buggy product.  For some that’s enough but when you’re busy trying to get your own products out the door there’s not much incentive to work even more and find bugs for someone else.  In my opinion, this has always been the problem with the beta program.

Some of use are also guilty of not starting testing until the Final Candidate releases.  This is actually the worst time to start testing because it means only show-stopper bugs will get fixed.  We really need to start testing with the early beta’s.  That’s when Xojo needs us the most to bang on their product.  Is it fun?  No.  Is it necessary? You bet because waiting until Final Candidate is worthless to them.  To us too, because those bugs might not get fixed for 3 months.

Did Xojo help us out in this cycle?  No.  There were times they were putting out a new beta every day.  Sometimes with only a couple of changes from the previous version.  I would download a beta with the intention of working with it and before I could get to it a new beta would be announced in the next couple of days.

I don’t know about the rest of the people in the beta program, but the high frequency of beta releases meant I looked at them less.  I don’t know why but it was.  I saw the announcements rolling in and I shrugged in apathy.  Perhaps I felt the frequent releases didn’t properly value my time.  I only have so much time to devote to beta testing and the frequency was off-putting.  I like to use a version for 3 or 4 days (or until I find a major bug a report it).  With so many releases I just didn’t test.

I propose that with the next cycle Xojo limit their public builds to once a week (maybe a Thursday or Friday release).  This respects my time a bit more since my only incentive is to make it a better product.  Frequent public releases says that they’ve not vetting their product enough before giving it to us.

I also think there needs to be something in place that identifies what type of beta tester we are.  Are you mostly a Windows user?  Then Windows specific changes get highlighted for you because you’ll be most likely to see them.  Same with Mac OS X, iOS, and Linux users.  Desktop, console, and web app users might also be a designation.  I realize this doesn’t fit in with the current generic “hey, we have a new beta” announcement.  But it’s clear from R1 that the generic approach didn’t work.

We have too many major bugs in this last release to call the beta program a success.  It’s not doing what we need it to do.  So how do we structure is to so that our time is respected and we actually report bugs before Final Candidate stage?

What about you?  What suggestions do you have to help us and Xojo out at the same time?

22 thoughts on “We Are Part of the Problem

  1. Maybe they should accept *some* non-Pro users that might be motivated to test it.

    Speaking for myself I, for a profession, test quite a bit of my time. Not meant to brag about myself, but I think I’m quite good at seeking edge cases. I’m using Xojo as a hobbyist (at work we only use VS .NET) and like it. I would love to help them. Problem is I’m not a Pro user (only iOS license).

    What I’m trying to say is that Xojo must consider other users based on other criteria than only Pro users. Pro users might have less time then people like me.

    I know that testing is important. In my profession testing the products *I* make has low priority. “Sorry, no time”. It’s frustrating to see users complaining on you product that you worked so hard on. Complaining because they found bugs that could have been fixed if it was tested properly.

    If Xojo needs a motivated tester than they can contact me.

  2. I like that idea of what type of a user. I tend to look at the release notes to see how the update affects Web and iOS. That takes time though as you have to download it to see the notes rather than it listed in the forum post.

    I also like it when the Engineers ask us to test things in particular.

    • I forgot to say that I have no problem to use the Beta version for real production work. So, I really will use it instead of poking a bit around *when I have some time*. Although I’m a hobbyist user with regard to Xojo, my projects are not small. Some have made it to the App Store.

      In contrast to a professional Xojo user I would not have a problem if I goto stuck on some Beta bug that would stall my project for a week. I just then do something else with my free time until the bug is fixed …. 😉

    • I like it when they ask us to test particular things too. It helps me out in focusing on what to test with my limited time.

      They’ve tried to do that with focus areas. But, like you, I end up having to read the release notes and try to figure out what needs to be tested. Sometimes the Feedback title isn’t very helpful.

      Again, part of it is us, part of it is the Release Notes, part of it is Feedback. I sort of feel there is something missing. Maybe that’s having specialities within the beta program and we get emailed specific cases that our specialty area indicate we should test it.

      If I got an email like that saying, please test case X I sure as hell would make time for that test. On the rare occasions one of the engineers ask me, personally, to test something I will do that (partially because it’s so rare).

  3. I don’t know the number of people in the program. Hundreds? If each one did an hours worth of testing and reported one bug Xojo would be better off. However, I’m sure that a vast majority of beta program people file no bugs.

  4. I’m not trying to incite anyone or anything, but “It’s our fault” reminded me of the sermons I heard in church (when I still believed that stuff).

    2015R4 and R4.1 are unusable as far as I’m concerned (split bug). It still hasn’t been fixed in 2016R1 (which has more serious problems). There were a few more releases with show-stopping bugs (for me).

    I see the current situation as the logical progression that a few users warned about when the Rapid Release Cycle was introduced. The emphasis on new features (because feature requests are by nature cumulative, eg many users ask for iOS support) rather than bug fixing (bug reports are by nature distinct as users tend to be affected differently) has led to a situation where bugs were introduced at a faster rate than they were fixed … resulting in an increasingly buggy product.

    Just look at the increasing number of dot releases that are required because of major bugs (one release had FOUR dot releases). Dot releases have become the norm rather than the exception. 2016R1 will surely have a dot release soon too.

    It is somewhat ironic that this happens when Xojo NEEDS to concentrate on new features: 64bit, HiDPI. They neglected their core business (cross-platform desktop) and now they have to deal with a major transition WITH A BUGGY PRODUCT. We all know what a nightmare it is to implement major new features and get everything working when the basic features aren’t working.

    It’s not “our fault”. It is the consequence of Xojo’s policy.

    When the desktop transition is done then Xojo really needs to reconsider their emphasis. What’s the point of an app with lots of features that don’t work well? You wouldn’t tolerate it in your word processor or your spreadsheet, would you?

    • Well, 2015 R4.1 has worked well for us and our clients. I’m sorry that are experiencing some show stopping bugs, but to claim your experience is widespread seems a bit of hyperbole. I’m sure I would have heard more grumbling if that release had more issues.

      Since I know you’re part of the beta program did you participate in 2016 R1? Did you submit any bug reports before Final Candidate?

      Is your solution simply to do away from the Rapid Release Program?

        • Desktop needs to finish its transition. 64bit, HiDpi, new framework, .Net.

          For iOS Xojo needs a lot more controls.

          Those are the priorities in my opinion.

          But then the emphasis must be on bug fixing. You don’t need to get rid of the Rapid Release Model, but make two or three release each year bug fixes ONLY.

    • “2015R4 and R4.1 are unusable as far as I’m concerned (split bug). It still hasn’t been fixed in 2016R1”

      Do you mean the 64-bit split issues or is there something else?

  5. I used to code with Xojo both on a Mac (at home) and a PC (at Work) but when my work PC died, they gave me a old Mac instead so I no longer code on a PC… Though most what I write has to RUN on PC’s or both.

    That said this cycle it seemed was largely about retina… something that right now is not relevant for me… No machine I use is retina capable and no machine my code needs to to run on at work has a retina screen nor are the PCs HDPI…

    That said I did report a beta bug on an edge case… It happened to be one that matters a lot to me, as it killed my listbox subclass… Thank goodness it was fixed before release!

    • Excellent point. I would imagine that even among the Mac OS X user base there is only a minor percentage of Retina users. In Windows it’s *just* starting to become a ‘thing’. That also helps explain why more beta users didn’t test: it just didn’t apply to them.

        • Fair enough. What I mean by being a ‘thing’ is that walking through a BestBuy looking at computers it wasn’t until recently where HiDPI monitors were advertised on the cheap computers the masses are buying. I’m sure for some it’s been important for a long time.

          • That’s certainly fair; high DPI on Windows was for “displays” in the general sense, so it also included anything you could render to, such as printers. I don’t think high resolution displays gained a whole lot of traction on the Windows market until much more recently (compared to XP).

  6. I didn’t enjoy this beta cycle. For one thing, the betas were so flaky that they felt more like alphas in progress being passed off as betas and I couldn’t do any proper testing in them (i.e. actual work). I’ve asked them in the past to focus their betas more – give us less betas and give us more information on what they really want us to test. I want to actively contribute in testing each release but I’m so busy that unless they help me, I can’t help them the way I want to.

    Any testing that I did do was done on a Retina MBP but I tried the new release on a recent Air yesterday and it was painfully sluggish.

    Ultimately I’m guessing the main problem with this release might be quite simple: they’re stretched too thin. Major kudos for what they do actually achieve but it just seems that there aren’t enough engineers.

    One more thing: I’m not an Android guy personally but from a business point-of-view I would like Xojo to be able to build Android apps. But I now hope they ignore the calls on the forums and get iOS 1.0 out the door first.

  7. I am to blame like the rest of us. I didnt test as much as I should of. Between the many many releases and being busy with work it was hard for me.

    Now if Xojo had a cli compiler, I would test with every (ok I cant say every but almost every) release against my code base. I would write a script to make a git branch of the production code, run the beta against it, then move onto the next one (repeating through the code bases). Looking at the output of the script to see which code bases failed (if any) and then I can see why they failed and open the appropriate cases. Think of Christian that has 1000s of sample programs. He cant check even a fraction of them with each GA release let along each beta. He could script the beta compiles and see what breaks.

    I know that the “cli compiler” is “scheduled” but who know what release. I would kill for that feature.

    I try to be a good beta tester but it doesnt always work out. Xojo, sorry I am not a better tester. I do try.

    • To be fair, they didn’t help us by putting out beta’s of questionable quality and usefulness with such high frequency. Our time is valuable and they didn’t respect us this cycle.

      • very true Bob.

        I know that I (and many others including Xojo) could do some much more testing if we had the command line compliler. although it is scheduled I am not holding my breath.

Comments are closed.