Test Data

At XDC 2019 my session was titled Xojo Design Mistakes (the alternate but way longer title was ‘Thankfully time travel doesn’t exist or my future self might travel back and murder my younger self for the stupid coding mistakes I’ve made’).  These are things that I’ve discovered over the years, both in my own projects and in other peoples projects that are just plain wrong or less than ideal.  This will be an on-going series since I had well over 50 slides and left 150 out.  So when I get bored I’ll bring these topics up.

Nearly all of our consulting projects are database driven applications.  It’s why we’ve created the tools to help with these projects like ARGen, which simplifies our interactions with the database, and BKS Shorts, which is our own reporting tool.  These tools are invaluable in getting our work done in a timely matter.

In a database application it’s typical to have a List of something.  A common example of this is a Customers list.  In that list the client typically wants the ability to Create, Read, Update, and Delete (or CRUD) a customer with varying degrees of rules behind it (like do they have permissions to add or delete a customer?).

During development we get the List form going, add the controls to be able to add a new record.  Then we create the Add/Edit form that allows us to test those capabilities.  We create a few, update a few, delete a few customers and then move on.  Maybe the client wants search capabilities so we add that to the List window and when we’ve tested it against our half dozen or so records we move on to the next task.

There is nothing wrong with this process.  It works and it’s fairly efficient as far as it does.  However, there’s one thing we’ve skipped that’s really important but also difficult to achieve.

So far we’ve test with *maybe* a dozen records.  What happens when the client adds 10,000, or 100,000 Customer records?  Does the list form take a long time to load?  Does the search function take a long time?  What about the Customer popup menu’s that you’ve scattered throughout the project – are those now slow, unwieldy, and unusable?

Unfortunately, with the way we implemented the project we don’t know how any of this works since we only have a dozen records.  So it’s really important to have adequate amounts of test data.  Creating 10,000 new customers using your new interface would take a long time.  So what can you do?

There are tools out there that will help generate data sets.  These tools allow you to create thousands, even millions of rows of realistic data.  Randomized male and female first names along with a last names is a great way to generate customer names.  Many tools allow you to add random dates in a range, random IP addresses, random values from a  list you provide and so on.  The sky is the limit when it comes to what sort of data developers need.

Now, when you do your testing you see how your application reacts with a lot of data.  I almost guarantee that it will act different.  Do you need to switch to a data-on-demand listbox?  Do you need to put an index on a common searchable field to speed up indexing?  Do you need to implement Full Text Search in your database?  Having a huge amount of data will answer these questions for you.

I once worked on an accounting application in VB6 where the original database designer using an Access database and did an account balance on the fly iterating through bills, checks, journal entries, etc. With a few thousand rows of data in each table this process took a second or two for all balances on a local machine. When this database was accessed over the network it took 5 to 7 seconds. When we converted our first client database it took 30 to 40 seconds for EACH account! Obviously this was not acceptable performance from an accounting application meant to be used daily by general contractors with hundreds of employees and tens of thousands of customers. The solution was to have a current balance value that was stored and then updated when a transaction occurred. We could have saved ourselves hundreds of hours of rushed development time (and much stress and heartache) if we had tested with large amounts of data much earlier in the process.

I mentioned adding an Index to a field earlier. One word of caution on this: it’s tempting to add an index to every field you’re searching on. Don’t do this! Only added indexes to the most important fields in a table. For a customer maybe the two most important fields are phone number and name even though you search on City and things like that. Indexing is extra work for the database so performance can take a signifiant hit with indexing a field.

Since the toolI’ve been using to create test data is no longer being sold I’m curious what you’d recommend.  Do you have a favorite tool?  Or is this a tool that would be of use to the community?

Happy Coding!

Exception Handling

At XDC 2019 my session was titled Xojo Design Mistakes (the alternate but way longer title was ‘Thankfully time travel doesn’t exist or my future self might travel back and murder my younger self for the stupid coding mistakes I’ve made’).  These are things that I’ve discovered over the years, both in my own projects and in other peoples projects that are just plain wrong or less than ideal.  This will be an on-going series since I had well over 50 slides and left about 150 out.  So when I get bored I’ll bring these topics up.

Xojo has evolved over the years.  Many of the global framework (the classic framework) classes set an error bit or error code when an error occurred and the developer has to check to see if there is an error and deal with it accordingly.  Many developers don’t check – mainly out of ignorance, and that’s a problem.

The newer (but soon to be deprecated) Xojo framework (and we’ve been told the API 2.0 framework as well) throws Runtime Exceptions to report errors.  The exceptions definitely get your attention because, well, an exception happened.  The problem is that many Xojo developers don’t catch exceptions anywhere in their application.  This results in dialogs like this:

Users hate seeing this and it’s unhelpful to the developer because it tells them NOTHING about the cause of the error.  And, their application quits making the user really unhappy.

The Application object in every Xojo project has an UnhandledException event made to catch any exceptions not caught anywhere else.  It’s declaration is this:

By returning True you can keep the application from quitting.  Just returning true is just as bad, perhaps worse, than the first dialog because the app had an exception and the app might now be in an unknown state and the user has no idea that it happened.  At this point what happens is anyone’s guess.  Silent errors are a bad thing.  Never do this.

A better way is to use the App.UnhandledException event to report that something happened so the user can decide what to do.  Ideally, you’d like to get some information from them so the developer can get an idea of what’s caused the exception.  What were they doing when the exception happened?  What is the stack trace?

If you are are unaware of what the Stack Trace is this is the call stack your app is current in when the exception occurred.  Maybe you called the Pushbutton1.Action that called ClassA.Method1 which called Global.MethodB which called Global.MethodC.  This information (although not as neatly) is in the stack trace.  It is sometimes invaluable in helping find and fix bugs.

BKeeney Software created a generic error reporting system years ago that when put in the App.UnhandledException shows the user a dialog like this when an exception occurs.  You can tailor the message to suit your needs.

We generally don’t quit apps when exceptions are raised, but it is something that some clients want.  The Report Error button then takes them to another dialog asking for more information.  The user can easily bypass this section by pressing OK.  Hopefully your users are nice and they’ll report the error.

If they select the E-Mail Bug Report or Save Text File button without putting anything in the TextField we present a dialog begging them for more information.  Sometimes this works but not always.

Regardless, the next step in the process creates an email or text file.  Sending an email requires the use of the computers built-in email client and we find this has advantages in that we get the users email address.  A sample email goes something like this:

In our email we get the type of exception, the time, the method location, basic system information, the user description (hopefully), and the stack trace.  A vast majority of the time that’s enough to find a bug.  Sometimes it’s not but at least you can email them back.  The text file still requires them to send it to us via email so we helpfully include the support email address.

If you want to play around with this example, please download it from https://www.dropbox.com/s/3uxyqvjf4wvjrpg/Exception%20Handler%201.0.zip?dl=0.  Feel free to use in your own code however it is provided as-is with no warranty.  We cannot provide support.

You need to implement something similar to this exception handling strategy.  It will help your customers provide feedback to you so you can fix bugs.  This seems like the least you should do.  With the upcoming API 2.0 with API’s that throw exceptions rather than setting error codes this will be more important than ever.  I’m sure I’ll talk more about Exception handling strategies once API 2.0 is released.

What other types of things do you do for error reporting?

Don’t Overuse Variants

At XDC 2019 my session was titled Xojo Design Mistakes (the alternate but way longer title was ‘Thankfully time travel doesn’t exist or my future self might travel back and murder my younger self for the stupid coding mistakes I’ve made’).  These are things that I’ve discovered over the years, both in my own projects and in other peoples projects that are just plain wrong or less than ideal.  This will be an on-going series since I had well over 50 slides and left about 150 out.  So when I get bored I’ll bring these topics up.

Variants are a very powerful datatype in Xojo. It allows you do set it to anything including null. It’s a wonderful thing to use in ListBox row, column, and cell tags where the you, the developer, could be anything into it.

Variants are evil too if you use them in ways you shouldn’t. Take the example below:

This is truly a made up example but it shows how subtle Variants can stab you in the back. Just adding the three Variants together where one is a string, the other an integer, and the other a double, can change depending on the order they’re used. Frankly, I don’t care how the compiler came up with the value because in both cases it’s ‘wrong’. Wrong in the sense that it depends on order and no programmer should have to depend on order.

How did we get here? Variants have implicit conversion. So if you try to add a variant to an integer it attempts to convert it to an integer. If you try to add it to a string it converts it to a string. What happens if you have a number inside the string and it really should be a string? The answer is that it depends.

That’s just a poor use of the language, in my opinion. Xojo is a strongly typed language and I like that it won’t let me willingly add an integer to a string. I like that I have to explicitly convert from one datatype to another. Variants break that rule in that they’ll implicitly convert from one to the other and the compiler will never tell you.

The variant datatype allows the Xojo developer to query what it is. For example you can use the Type method to see what variable type is it. You can see if it’s an array using the IsArray method. You can see if it’s a Numeric value using IsNumeric but you get into the same problem where assigning an integer 8 and a string “8” will both return true from IsNumeric. So it’s a bit of a problem because it could be either. We should rename it Schrödinger’s datatype because it could either either value until the compiler asks for it.

Auto was introduced in the Xojo framework (deprecated now?) to replace Variant. It didn’t do any implicit conversion for you. In fact, it wouldn’t even tell you what datatype was in it and the only way to do was to use Introspection. It certainly solved the problem of implicit data conversion but replaced it with a harder to use datatype. For what it’s worth I didn’t think it was hard to come up with your own “what is your type” methods using the extends keyword but it was something that many Xojo developers weren’t comfortable doing and frankly seemed silly to rely on Introspection to do that work.

I turned down a project many years ago because the original developer had used Variants for everything. Didn’t matter what (local, object, global), they used variants and they wanted me to tell them why their project wasn’t working right. When I told them I wouldn’t do it and asked why they had used variants for everything they simply said, “Because they could be anything. Why not use them?”

I know that many languages are not as strictly typed as Xojo and perhaps that’s where they came up with that mentality. I find some comfort in knowing that if I try to put a string into a numeric value the compiler won’t let me unless I explicitly convert it. I like the certainty of knowing that my data can’t change on my unless I tell it too.

Variants are good for storing things temporarily. Stashing values and objects in a Tag for use later is convenient and useful and a good use for them. For everything else it makes sense to use a datatype. If you finding yourself using a lot variants you should rethink what you’re doing because they can subtly make your calculations change in ways you don’t expect.

Formatted Text Control 3.2.1

BKeeney Software has released version 3.2.1 of the Formatted Text Control, a canvas based word processing control for Xojo Mac OS X and Windows applications. The FTC is an alternative to the built-in TextArea control and allows for in-line graphics, hyperlinks, custom components, better RTF support and much more. The demo project has quite a few examples, including a Masked Text Field, XojoScript example, Embedded FTC (in your own canvas), as well as a word processor with page view.

The Formatted Text Control costs $150 and is 100% unencrypted Xojo code. Version 3 and above requires the use of the Text Input Canvas plugin (included in package) to allow for proper text handling in Cocoa builds.

Download links and more information at http://www.bkeeney.com/formatted-text-control/ 

This version has a number of bug fixes reported by users since release. It is recommended for all users. Registered users should be receiving an email from our automated distribution system.

Bug Fixes:

  • Fixed issue where all StyleRuns after a hyperlink will also be marked as hyperlink
  • Issue #4074: RTF clipboard ignores hyperlinks with umlauts and cuts them
  • Issue #4087: missing RTF space after “\kerning0” keyword
  • Issue #4130: FTXojoScriptField: Left part of the text is hidden by the section of the line number.
  • Issue #4135: Migrating texts from a TextArea to the FTC (Ich migrieren Texte aus einer TextArea in das FTC)
  • Issue #4196: Remove GOTO calls from FTC

Enterprise Code Signing for iOS in Xojo

There’s been a feedback report (47975) for several years asking for the ability to use enterprise code signing certificates for building our Xojo iOS applications.  So far nothing from Xojo but there is a way to create enterprise code signed iOS app for Xojo – it just takes some extra work.

Here are my steps to building an iOS enterprise app:

  • Build your app in Xojo with the Build for App Store switch turned off.
  • Build the built app into a folder named “Payload”
  • Compress that folder.  Change the name from Payload.zip to “yourappname.ipa”.
  • In your favorite text editor paste the following code.  Replace App_ID_Prefix and App_ID with the valid information for your app from the Apple Developer website:

<?xml version=”1.0″ encoding=”UTF-8″?>

<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0.dtd”>

<plist version=”1.0″>

<dict>

<key>application-identifier</key>

<string>App_ID_Prefix.App_ID</string>

<key>keychain-access-groups</key>

<array>

<string>App_ID_Prefix.App_ID</string>

</array>

</dict>

</plist>

  • Save this file as Entitlements.plist
  • Go to the Apple developer website and download your enterprise developer provisioning profile.
  • Download iReSign project from https://github.com/maciekish/iReSign (see note below)
  • Run the ReSign app
  • Drag unsigned .ipa file to the top box, or use the brown button.
  • Drag your mobileprovision file you downloaded from Apple to the second box, or use the browse button.
  • Drag your Entitlements.plist file you created earlier into the 3rd box, or use the browse button.
  • Select your name from Keychain Access List.  For example it might be, “iPhone Developer:  Firstname Lastanme (XXXXXXX) from the dropdown.
  • Click ReSign! and with.  The resigned filed will be in the same folder as the original with (Resigned) in the name.
  • Voila!  Move that resigned file into the appropriate place and remove the Resigned so it’s back to the yourappname.ipa” name.
  • Now you can deploy your enterprise code signed app to any iOS device.

Each iOS device may have to trust the certificate.  Go to Settings -> General -> Device Management.

For it to work properly in MacOS Sierra (and above) I had to recompile the ReSign app in Xcode (no code changes).

It seems like a lot of steps but once you get it working it’s not hard after that. One thing we learned the hard way is that enterprise certificates expire in a year so plan renewing the certificate and getting a new version out before it expires.

Hopefully someone will find this useful.

[Update]: My iOS Build settings for Team are set to “None”

Don’t Use GoTo

At XDC 2019 my session was titled Xojo Design Mistakes (the alternate but way longer title was ‘Thankfully time travel doesn’t exist or my future self might travel back and murder my younger self for the stupid coding mistakes I’ve made’).  These are things that I’ve discovered over the years, both in my own projects and in other peoples projects that are just plain wrong or less than ideal.  This will be an on-going series since I had well over 50 slides and left about 150 out.  So when I get bored I’ll bring these topics up.

Don’t use GoTo – ever.  GoTo is a holdover from the old BASIC days where code was unstructured and you needed to manage flow-of-control.  Old BASIC programs used a ton of GoTo statements because, well, you just had to.  It was the only way to get anything done.  Xojo uses a modern BASIC syntax but it’s fully object-oriented code and still has GoTo as a reserved keyword and it still functions.

Xojo has many ways of doing flow control.  There are multiple ways of doing loops with While’s, Do-Loop’s, For-Next’s, and we control those loops with keywords such as Continue, Exit, Return.  Since we have methods and functions too we can exit those by calling Return at any point in the method and the code resumes execution at that point.  GoTo just isn’t needed any more.

And yet GoTo still exists.  I recently ran across this code in a project we’re updating for a client.

As you can see we are in a fairly typical For-Next loop iterating through a ListBox.  The first line into the loop we check if we’re closing the window and if we are we call GoTo bale which takes us to where bail: is.  Bail is at the bottom of the method and we’re doing nothing afterwards (the last two lines are Exception handling in this method and aren’t called).

This is such a poor example of Xojo coding.  We can use Exit to accomplish the same thing since Exit will immediately exit the loop and since there would be nothing after the loop it would simply leave the method entirely by simply calling Return.  There’s no penalty for using either Exit or Return.

Honestly, I have no idea why the original developer did this.  I think they came from VB6 where code like this was pretty common, but even in that language there were much better ways to do the same thing.  So I will call this lazy coding because the developer didn’t use the best way in Xojo.

Using GoTo like this is potentially risky too.  Feedback case 24710 shows that using GoTo may cause a memory leak because the compiler may skip cleanup code.  I think the above code is safe but I could see if we loaded oNote before the GoTo exiting the loop it would be dangerous.  But even still, if GoTo is outdated, not recommend, and dangerous on top of all that why use it?  I can think of at least three ways to make this code safer and better.

At the end of the day, If you’re using GoTo – don’t.  Refactor your code so it makes use of the modern Xojo calls. It’s safer and the right way to code in Xojo.  Your future self will thank you.

(233)

Xojo 2019 R1.1

A dot release for Xojo 2019 Release 1 was released today.  Xojo 2019 R1.1 has several significant bug fixes.  This release is highly recommend for all users.

Perhaps the biggest bug fix is related to ListBox.CellbackgroundPaint and Listbox.CellTextPaint events.  They no longer leak memory 64 bytes of memory each call.

In Windows URLConnection received some attention.  They sped up socket requests and decreased the CPU usage.  Authentication no longer hangs when there is request content present.  The events for URLConnection are no longer re-entrant.  This is fancy way for saying that some events, like PageReceived would get called a lot rather than just once and could cause some issues depending upon usage.

Changing the case of label now changes the text in Windows.  For example if you set the label1.text = “lower” and then tried label1.text = UpperCase(label1.text).

Also in Windows, TabPanels embedded within another TabPanel no fires the Changed event multiple times.

One new thing made it into this build.  iOS projects now add default plist entries for NSLocationWhenInUseUsageDescription, NSLocationAlwaysUsageDescription (for iOS 10) and NSLocationAlwaysAndWhenInUseUsageDescription (for iOS 11+).   These can be overwritten by user plist files.

I am glad to see this dot release version of Xojo.  I know we like to beat up on Xojo Inc for various things but I think they generally do the right thing for the community.

(265)

Listbox.CellTextPaint and Listbox.CellBackgroundPaint Memory Leaks

There are bugs and then there are bugs.  Not all are created equal and certainly not all are worthy of a dot release.  But when a serious bug comes to light you have to take it seriously.

Feedback 55596 is a perfect example of this.  If you use Listbox.CellTextPaint or Listbox.CellBackgroundPaint your app will leak memory.  Leaking memory is bad to begin with but since the Listbox calls these events frequently.  Losing 64 bytes per call adds up quickly.

I was going to do a release today for a client using 2019 R1.  This was the first release where Windows drawing was as fast as older versions of Xojo.  Prior releases of this product had to be built with 2017 R2.  So now what do I do?  I’m damned if I do a build and I’m damned if I go back to 2017 since we have to retest everything.

Xojo Inc. doesn’t seem interested in doing a dot release.  On the forum Greg was quoted as saying, “…doing a point release is not an insignificant task and takes resources away from making progress on other things, never mind the additional stress that comes with it.”  Who cares if it’s work on their part?  Seriously, if I can’t use the current public release then why am I here?  It’s bad enough I’ve been chasing Xojo versions for nearly two years with Windows drawing.

This is part of a much bigger issue that many at the Xojo Developers Conference discussed a few weeks ago.  Xojo has always been a lean organization and many times I’ve felt that it’s been quite anemic.  But an important bug and a public release to fix it shouldn’t be a major stress inducer to the developers.  To me this says there are too few developers and too much work on their plate.

For what it’s worth, the bug is marked as fixed as of May 14th.  It’s still too early to tell if we’ll get a dot release for such an important bug fix.  Maybe R2 is just around the corner but I don’t have time for a 30 to 60 day beta period.  I need this fix now.

(175)

XDC 2019 Ruminations

The 2019 Xojo Developers Conference (XDC) wrapped up last Friday. It’s my favorite event of the year and while I’m sure many people think that I’m an extrovert I can only play one for shorts periods of time. I’m enjoying the peace, and more importantly, the quiet at home today hoping to recharge my social battery before having to attending a high school robotics meeting, mandolin orchestra rehearsal, and a mandolin concert this week. Oh, and I have to pick my sone up in Chicago. No pressure to get things done, no?

XDC is my favorite time of the year. While attendance was down this year (more on this in a bit) a lot of people from Europe made the trip over to Miami. It was good to see a lot of old friends and awesome to make some new ones. To me, it’s the connections you make and the stories you hear from other Xojo developers that is inspiring in so many ways.

The keynote address by Geoff was…underwhelming. It was the usual mix of how the community is doing, what they’ve done since last XDC and what they’re working on. As expected they’re working on Web 2.0, API 2.0, Interops, and Android. The best timeframe Geoff gave for anything was ‘soon’. At least we didn’t have the mental gyrations over ‘priority’ and ‘important’ like last years XDC.

It’s hard to show much for big ticket items like Web 2.0 and Android. These are big multi-year projects that you just can’t show off without having significant work done on them. For Web 2.0 the demo that Greg gave seemed pretty solid but then there weren’t a lot of details. It did seem solid and that was good news in my opinion. The Android demo, on the other hand, looked much more scripted and not nearly as smooth – it had the feel of ‘this is something we know works and we’re not going to do much outside the script’ feel to it. For API 2.0 there wasn’t much to show except some before and after code snippets.

So on one hand the keynote wasn’t all that exciting. On the other they didn’t have much to show because it’s work in progress. They threw some numbers out like 70% complete in which the pessimist in me said, “great, that means there’s still 70% of the work left to be done.” The people around me at the keynote had some similar thoughts as mine. To be fair, ‘completion percentage’ means something different to everyone.

I am very excited about Web 2.0. It’s a rewrite of the entire framework. You have to keep in mind the original framework was done ten years ago and the web has changed drastically since then. The work they’ve done on theming is going to make our Xojo web apps look fantastic and since more processing is done on the client side it should make apps a lot more responsive. We’ll have a LOT more controls to play with too.

To be honest, I’m pessimistic about Android. I’m sorry, but two years later we’re still waiting for a release. With work still to be done on the Xojo framework items (think FolderItem) and the debugger I just don’t see that happening by the end of the year. Prove me wrong, Xojo.

Jim Meyer’s presentation on using AWS and Google Machine Learning API’s with Xojo was fascinating. Using their API’s it’s easy to do text transcription from text and video, recognize images, do facial recognition and much more. It was by far my favorite session I attended. I can’t wait to get the example projects and try them out on a few things.

I didn’t attend this session but attendees were buzzing about Monkeybread Software’s DynaPDF plugin. The big news was that Unicode support will be added by the end of the year and that we’ll be able to create PDF’s by drawing directly into the Graphics object provided by the DynaPDF class. All good news.

Michael Dettmer showed an interesting way to quickly create database applications for Xojo using an Apache Velocity and his own open source software called CAPP, or Computer Aided Program Production. Essentially you create templates for the various things you want and it creates a project for you. The drawback is you have to use Apache Velocity and Java to do some of the coding. I’m not doing the demo justice but it did look a bit more complex than many Xojo developers might be capable of. However, consultants might find this tool essential since you do a little work up front to create a database application. You can read more at https://capp.systems

Yousaf Shah had a great session on how to get happy customers, successful projects, and live stress free. Yousaf said that developers aren’t always the most empathetic people and it’s hard to put yourself in the client position. When mistakes are made you don’t have to admit a mistake but saying, “I’m sorry you feel that way,” is important. He also spent some time showing how we can go from features to tasks (wrong word) so that you can get a better feeling for what the client really wants while also letting them be in control driving what’s important to them.

One session that I thought was fairly depressing was the Ask the Engineers session. Greg, Travis, William, and Paul did a good job of answering questions from the audience but it was shocking to see that few of developers working on the product. They’re one heart attack away from major portions of the project to be delayed. If you didn’t know, Paul has been moved from his Evangelist role to an engineering role. Since last XDC Norman was let go and Joe (compiler) has moved on to another job. Geoff can say all day long that they have enough resources but I just don’t buy it. More engineers means more stuff gets done.

Not giving us any timeframe for releases is one reason why I think attendance was down this year. If you don’t think there will be any big news and hands-on with cool, new, stuff then what’s the point of the attending? I disagree with the reasoning but I understand it when it’s a significant cost to attend.

No XDC in the United States next year. They are looking at something overseas. No word on where they’re looking but since MBS has conferences in Germany that are well attended I suspect some place other than Germany. But who knows? Like everything else they’ll tell us when they tell us and BKeeney Software will most likely be there. After all, it wouldn’t be a Xojo Developer Conference with us being there! See what whenever and wherever the next one is held.

Thoughts on XDC 2019?