Opening Up the Beta Box

There is a thoughtful blog post from Mattias Sandström on the Association of REALbasic Professionals site.  (Direct Link)

I think it fits in rather well with my post back in May about the REAL Software beta process.  Mattias’ standpoint as a user of a lot of other IDE’s is well taken.  I really like the idea of polling the beta users about the release.  I wish it was possible to ask about every change but I know that’s not feasible but the number of bugs that were marked as fixed but were obviously not in the past couple of releases is insane.  There has to be a better way!

I would agree with Mattias regarding the beta mailing list.  I find it to be waste of time and energy.  I prefer forum style discussions.

Here is my prediction though, if RS tried to switch to the forums:  Instead of using the forum, the RB “old-timers” will insist upon the mailing list because they feel it’s better.  How do I know this?  When RS tried to kill the NUG mailing list a few years ago the “old-timers” threatened to create their own mailing list and many threatened to stop using RB altogether!  In the long run RS backed down and I lost a lot of respect for RS and the RB community as a whole.  They were being held hostage by a very select and vocal group of users.

For the beta program to get better, RS has to do what’s best for RS – not necessarily for us users.  If the beta program isn’t doing what it’s supposed to be doing then it’s time to kill it and start over with testers that can do it right.

10 thoughts on “Opening Up the Beta Box

  1. We’ve toyed with the idea of moving the betas to the forums, but like you’ve so noted, it probably wouldn’t fly. I was also one of the “old-timers” that was quite adamant about leaving the NUG alone, but that was before I started with REAL. Being on the other side of the table now, I understand the desire to move away from the mailing list.

    I will say this, if anything were to happen, something new would replace both. Neither the forum nor the lists are fantastic options. That is not a promise or announcement that we’re doing anything.

    As for the testing itself, I think we’re seeing Cocoa muddy up the testing of our existing frameworks. Mac testers outnumber our Windows testers at least 2-to-1, and I believe most Mac testers are testing Cocoa. While we absolutely need that, we continue to need testing of the Carbon framework, and that has begun lacking. More than likely, our smaller group of Windows testers are the ones finding the bulk of the non-Cocoa bugs these days.

    I also think there is a lot of benefit to using beta builds on a daily basis. I know we tell our testers not to, but there isn’t a better way to find bugs than to really use it day-in, day-out. We use the beta builds ourself, and it has helped us catch bugs because of it. Downloading the beta, running a project, then shelving it is not testing and I think that’s what a lot of our testers do. I know I did before I started with REAL.

    Though reading that paragraph back, by no means am I trying to place the blame on our testers. I do think the “blame” could be split between the testers and REAL. Maybe the 90 day release cycle is too short, maybe 30 days of beta testing isn’t enough, maybe we all need to eat more pie – I don’t know.

    I agree there is room for improvement. But then again, I’m just the new guy 😉

  2. I can tell you that I use the forums more since I have my notification list to get emails for new posts. With notifications on replies, it’s quite like a mailing list, but it could be better.

  3. Considering that you can subscribe to a forum topic and get an email for any new topic and then get an email when you’ve posted into a topic their argument seems kind of specious. The forums thrive, IMO, because of the email notifications.

    Thom, I would argue that testers *are* part of the problem. The download and forget doesn’t help RS out.

    You are correct, though, that when it comes to testing, neither email or forums holds up very well. What is really needed is an integrated bug tracking, email, forum type thingy. In the year I used TestTrack Pro (http://www.seapine.com/ttpro.html) that’s exactly what it did. I was very, very happy with it and I was in charge of the testing department.

    The drawback is that it ain’t cheap. But how much do bugs cost in the long run? If it reduces bugs and increases quality then the prices isn’t that bad.

  4. We’ve talked about integrating the community into Feedback (which would suggest a rename at that point) but we don’t have endless time and resources. Geoff and I evaluated that option and determined it would be absolutely fantastic, but we have bigger fish to fry for the foreseeable future.

    So for now at least, our community will remain segmented because we cannot get one part to join the other.

  5. After Thom’s post I thought about the differences between RB and the Netbeans IDE that I discussed and was using betas in the daily work. The difference is that I program in JavaScript or PHP using Netbeans where the files are purely text-based making it possible to easily open the files in different versions and also use a versioning system in case things go wrong. I am one of those RB beta-testers that are not using betas on a daily basis on my main projects as a new version will update the files to the new version. In a complex project with multiple shared classes and libraries linked in as externals it could be pretty messy (I have multiple subversion project for the main application and the support libraries) testing new versions, especially if you stick to RB’s binary fileformat.
    With pure text-files not much can go wrong, but I have came across bugs with the code formatters in the IDE making the code useless but with an easy recovery of the individual file from the versioning system.

  6. I guess the biggest problem I see with the beta mailing list is the fact that nothing you post on the mailing list, no matter how many times you post it, nor how many people (even RS employees) reply to it/discuss it, is is any way officially regarded as feedback by RS. If the bug isn’t entered in the system (feedback) then there is no bug. That’s my biggest problem.

    PS: Sorry to say I disagree with you on forum/mailing list. I hate forums with a passion (search doesn’t ever work the way I want, information is by topic,…), but love seeing the (non moderated) flow of information on every kind of topic (and yes i’ve learned things that way that I otherwise never would have) on mailing lists (I also have my own local archive of mails going back to at least 2005). I am pretty sure if the mailing list was abandoned, we would again see an alternative pop up within days.

    • So basically you want a better search for forums and you’d be cool with it?

      I hate the mailing list with a passion. So you and I nullify each other. 😉

  7. I was one of the old timers, was involved in the alternative mailing list, and I did stop using Real Basic.

    For example. I did ‘not’ come and have a look at this discussion because I happened to be looking through your blogs Bob. I came here via an facebook notification which is pretty close to an email mailing list and because I appreciate getting your views on the current state of Real Basic – I may buy another RealBasic license in the future.

    I stopped using Real Basic because of the short release cycle and the buggy releases. I believe Real Basic is now much better. I found I was spending way to much time sorting out bugs, with RealBasic, only to repeat the process every 90 days. I work on a product that has a 45 day release cycle. Our users are starting to run into the same issues that I had with RealBasic, buggy releases. Our team pretty much ‘blames’ the beta testers … I know the testing process has to improve … in an automated way, people will only test what they are interested in, or what they are focused on. For the product I am working on I have decided to automate every thing I possibly can. Each time a user raises a bug we create a test first that verifies the bug exists, then we fix it, then we check the test works, then we build that test into our automated tests.

    Regards

    Leith

  8. @Bob Keeney
    Having a better search would be nice, but it would not completely fix the problem. It still does not solve my second problem, which is that information is entered by topic. So for instance if I am a new Realbasic user and have a problem with a TAB control on Windows, do I enter it under “Getting Started”, under “General” of under “Windows”. Depending on the user it could be entered in either. To me it makes the information feel like it’s all over the place instead of in one place. I for instance would not usually look at getting started, which means I would not see and therefore not help people with questions there, since I don’t have a need to be there. With the information in one place, I would see the question and answer it if I know the answer.

    PS: I promise not take take you forum away if you don’t take my mailing list ;-P

  9. @Dirk Cleenwerck
    Well, we’re talking about a beta specific forum so there wouldn’t be (I’m assuming) any sub topics. Although I could certainly see subtopics based on each Alpha/Beta release.

    You do know that you can search the entire forums, right? Enter the search term in the upper right and it will search all posts in all topics. I rarely search in a specific topic area simply because you never know where people might post their questions.

    You do also know that you can view new and/or unread posts, right? In the upper left (assuming you automatically log in), you have View Unanswered Posts, View unread posts, View new posts, and View active topics. I use this every day and answer a fair number of posts.

    My problem with the beta mailing list is there’s so much activity I have to use digest mode. And even then there’s an awful lot of chatter that doesn’t interest me. So I end up ignoring it. So why am I subscribed to a mailing list I don’t use? Because that’s the only way of getting that information. It’s not online like the NUG so it’s not like I can get online and search for a relative term.

    The forum lets me refine my search in ways that an email can’t. I search them all the time and think it’s the best resource for RB around. Heck, the forum posts show up in Google search results (though one would think you wouldn’t want the beta posts showing up on Google search results).

    In the long run, RS has to figure out what works best for them. Maybe they *like* the email list – I dunno (though Thom above said it’s not ideal from *their* perspective). All I know is that from *this* beta testers perspective the email list doesn’t help me help RS.

Comments are closed.