The newest version of ARGen

ARGen is no longer a BKS product but I’m still a user and I’m very excited about the things that Tim from Strawberry Software has done with it in the new release. Besides an obvious facelift there are some very nice features that have been added.

API 2.0 is now supported along with classic API. This should come as no surprise since Xojo moved to API 2 that ARGen would eventually support it and was a highly requested feature. If you select API 2 all code will switch over to the appropriate syntax for you including all of the hidden Xojo database classes that ActiveRecord uses. If the project type doesn’t support API 2 (like Web and iOS projects) it’s not an option.

Multiple database connections/classes supported in one project. Way back when we originally created ARGen and ActiveRecord we never envisioned that we’d ever want more than one database in a project. Thinking back that was a silly assumption but in ARGen 4 you can now have multiple databases, or namespaces, and each one does its own ActiveRecord thing.

Console project type added. Again this is a silly oversight on our part long ago but since since we didn’t do many console projects it makes sense that we didn’t have it. This should make adding helper apps much easier.

A new Verify class function detects deleted fields. ActiveRecord has always been able to detect newly added fields when running in the debugger that warned the developer if the Xojo class was missing a property for a database column. The new Verify class does the opposite: does your class contain a property that at one point we thought was a database column but no longer exists in that table? Before this version that was a silent failure as it didn’t detect it so at least now you can take care of it right away. When working on a big project with multiple developers touching the database this will be a huge help.

Foreign keys that are nil are inserted as NULL values. This was always a huge user request that ARGen handles gracefully. Whereas before it inserted zero’s or blanks (depending on primary key field type) it will now put NULL’s where appropriate. This allows better relational integrity and plays better with relational databases.

The User Interface generation has been cleaned up. No longer do you have the choice of a regular window and/or container for lists and add/edit UI – everything is a container control and this simplifies some choices that I think most users will really like.

Remember that ARGen is all about getting your Xojo database project going as quickly as possible that the tedious coding part is done for you. This version of ARGen takes that to the next level with many of these features. Obviously Tim will tackle Web 2.0, iOS API 2 and Android whenever Xojo releases them to the public. The future of ARGen is bright!

There are little changes throughout the product and the resulting project that I think you’ll be happy to see. A new ARGen license costs $99. Existing ARGen (from BKS) users receive a 20% discount when upgrading. All existing users should have received an email but in case you haven’t please send an email to workshop@timi.me.

For more information on ARGen 4 please visit https://strawberrysw.com/argen/.

Databases and Xojo API 2.0 And Why I’m not Using it Yet

Now that API 2.0 is more or less been through two releases (2019 R2.1 and 2019 R3) many Xojo developers are starting to poke around with API 2.0 in their database applications.  There are many things to like about the changes to the database classes and our own ActiveRecord class does many of these things.  Things like throwing exceptions when there are errors definitely get your attention whereas in API 1.0 unless you check for the error you may not know anything bad happened.  The built-in Prepared Statements are a super nice change too.

Despite all the good things with the db classes in API 2.0 I recommend holding off until one, very important bug, is fixed.  Feedback case 58739 is such a huge risk that you simply cannot use the new API 2.0 classes in your Xojo projects yet.

The essential detail of that case is that Xojo is not releasing objects as it should.  Norman did a test case and showed that when using API 1.0 his test project had 24 objects at the end of the test run.  Doing the same thing in API 2.0 had over 43 million still unreleased objects!  This means there is a huge use of memory in the application and in 32-bit applications it will simply crash once it consumes 4 GB of RAM.  64-bit applications will last longer but still bring your computer to its proverbial knees.  

The Feedback case talks about SQLite and in-memory databases but other reports are that it’s in other databases as well including ODBC and MS SQL Server (case 57978).  The good news is that 58739 is marked as fixed as of January 7th, 2020 (57978 has not been updated).  But unless Xojo issues a point release to 2019 R3 we won’t see this fix until 2020 R1 is released and there is zero evidence of that happening any time soon.

Several people have contacted us about an updated ARGen that generates API 2.0 code.  We are holding off until we feel API 2.0 is rock solid.  So until this is fixed we are not using or advocating for API 2.0.

Have you found anything else in API 2.0 that should discourage Xojo developers from using it?

Test Data

At XDC 2019 my session was titled Xojo Design Mistakes (the alternate but way longer title was ‘Thankfully time travel doesn’t exist or my future self might travel back and murder my younger self for the stupid coding mistakes I’ve made’).  These are things that I’ve discovered over the years, both in my own projects and in other peoples projects that are just plain wrong or less than ideal.  This will be an on-going series since I had well over 50 slides and left 150 out.  So when I get bored I’ll bring these topics up.

Nearly all of our consulting projects are database driven applications.  It’s why we’ve created the tools to help with these projects like ARGen, which simplifies our interactions with the database, and BKS Shorts, which is our own reporting tool.  These tools are invaluable in getting our work done in a timely matter.

In a database application it’s typical to have a List of something.  A common example of this is a Customers list.  In that list the client typically wants the ability to Create, Read, Update, and Delete (or CRUD) a customer with varying degrees of rules behind it (like do they have permissions to add or delete a customer?).

During development we get the List form going, add the controls to be able to add a new record.  Then we create the Add/Edit form that allows us to test those capabilities.  We create a few, update a few, delete a few customers and then move on.  Maybe the client wants search capabilities so we add that to the List window and when we’ve tested it against our half dozen or so records we move on to the next task.

There is nothing wrong with this process.  It works and it’s fairly efficient as far as it does.  However, there’s one thing we’ve skipped that’s really important but also difficult to achieve.

So far we’ve test with *maybe* a dozen records.  What happens when the client adds 10,000, or 100,000 Customer records?  Does the list form take a long time to load?  Does the search function take a long time?  What about the Customer popup menu’s that you’ve scattered throughout the project – are those now slow, unwieldy, and unusable?

Unfortunately, with the way we implemented the project we don’t know how any of this works since we only have a dozen records.  So it’s really important to have adequate amounts of test data.  Creating 10,000 new customers using your new interface would take a long time.  So what can you do?

There are tools out there that will help generate data sets.  These tools allow you to create thousands, even millions of rows of realistic data.  Randomized male and female first names along with a last names is a great way to generate customer names.  Many tools allow you to add random dates in a range, random IP addresses, random values from a  list you provide and so on.  The sky is the limit when it comes to what sort of data developers need.

Now, when you do your testing you see how your application reacts with a lot of data.  I almost guarantee that it will act different.  Do you need to switch to a data-on-demand listbox?  Do you need to put an index on a common searchable field to speed up indexing?  Do you need to implement Full Text Search in your database?  Having a huge amount of data will answer these questions for you.

I once worked on an accounting application in VB6 where the original database designer using an Access database and did an account balance on the fly iterating through bills, checks, journal entries, etc. With a few thousand rows of data in each table this process took a second or two for all balances on a local machine. When this database was accessed over the network it took 5 to 7 seconds. When we converted our first client database it took 30 to 40 seconds for EACH account! Obviously this was not acceptable performance from an accounting application meant to be used daily by general contractors with hundreds of employees and tens of thousands of customers. The solution was to have a current balance value that was stored and then updated when a transaction occurred. We could have saved ourselves hundreds of hours of rushed development time (and much stress and heartache) if we had tested with large amounts of data much earlier in the process.

I mentioned adding an Index to a field earlier. One word of caution on this: it’s tempting to add an index to every field you’re searching on. Don’t do this! Only added indexes to the most important fields in a table. For a customer maybe the two most important fields are phone number and name even though you search on City and things like that. Indexing is extra work for the database so performance can take a signifiant hit with indexing a field.

Since the toolI’ve been using to create test data is no longer being sold I’m curious what you’d recommend.  Do you have a favorite tool?  Or is this a tool that would be of use to the community?

Happy Coding!

ARGen 3.0.3

BKeeney Software Inc. is proud to announce an update to ARGen, our ActiveRecord Generator utility for Xojo developers. This minor update includes dark mode support, speed improvements, and important updates for generated projects. Updating to 3.0.3 is recommended for all ARGen users.

ARGen is available for macOS and Windows. It can be used for free in limited mode, and is priced at $99.95 to unlock all features. Existing version 2.x users will automatically be provided an upgrade opportunity when launching version 3.

3.0.3 Release Notes:

Changes:

  • Added Dark Mode support
  • Simplified manual relationship management
  • Selecting a different SQLite database now clears the password field
  • kMaxReturn is now a protected constant for cleaner code
  • DBUpdates module code is now cleaner
  • Improved instructions in some locations
  • Base project templates optimized
  • Preferences module no longer writes to SpecialFolder.Preferences
  • iOS Create Data Sources defaults to true
  • Updated links to Xojo documentation
  • Generated localization module constants are now protected

Fixes:

  • DBUpdates.SetDBVersion no longer uses a BKS extension synonym for str()
  • Fixed return statement for iOS apps using 2018r2
  • Projects with empty name now have default save name
  • BKS Created/Modified overrides no longer generate properties that fail to Register
  • Corrected minor UI bug on Windows
  • Project listing loads faster
  • Speed improvements throughout the software
  • Projects created but never saved are no longer retained when closed
  • Checking for updates at launch now works
  • Preferences window will show the last update check time

Pricing, examples, and more details can be found at the project homepage at https://bkeeney.com/argen/ 

Updating Records with Transactions

In my last blog post I talked about inserting records into a SQLite database and found that using transactions makes them incredibly fast.  Does the same hold true for updating an existing record?  We’ll find out.

In this set of tests I inserted 100,000 records and then randomly selected 10,000 records to update with our same random name, age, and date values.  In theory this means I have records distributed all over the range of the database.

In the first test we use a simple RecordSet which means we have to query for it first.  Then call its Edit method, change the values and call update.  It’s a two step process.

For i As Integer = 0 To ari.Ubound
   Dim rs As recordset
   rs = db.SQLSelect("Select * from t_table where table_id = " + Str(ari(i)))
   If db.error Then
      If db.Error Then
         MsgBox "DB Error: " + db.ErrorMessage
         Return
      End
   End
   rs.edit
   rs.Field("Name").stringvalue = GetName
   rs.Field("Age").integervalue = GetAge
   rs.Field("TheDate").DateValue = New date
   rs.Update
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

Unsurprisingly, this method took about 49 seconds for 10,000 records.

In test two I switched to using a PreparedStatement to update the record.

Dim sql As String = "Update t_table Set Name = ?, Age=?, TheDate =? WHERE table_id = ?"
Dim ps As SQLitePreparedStatement = db.Prepare(sql)
ps.BindType(0, SQLitePreparedStatement.SQLITE_TEXT)
ps.BindType(1, SQLitePreparedStatement.SQLITE_INTEGER)
ps.BindType(2, SQLitePreparedStatement.SQLITE_TEXT)
ps.BindType(3, SQLitePreparedStatement.SQLITE_INT64)
For i As Integer = 0 To ari.Ubound
   ps.Bind(0, GetName)
   ps.bind(1, GetAge)
   ps.Bind(2, New Date)
   ps.bind(3, ari(i) )
   ps.SQLExecute
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

When run for 10,000 records this takes about 47 seconds so there is a little bit of time savings by not querying the database for the data to begin with.  It’s also possible that the PreparedStatements aren’t nearly as efficient as one would hope.

I then repeated the same tests within a Transaction.  In both cases updating records effectively took no time at all!  If I increase the number of Updates to 10,000 both the RecordSet update and the PreparedStatement takes about 8 seconds.  

I’m surprised at these results because you would think that doing the query required for the RecordSet would take more time but it’s no different than using the PreparedStatement SQL Update.  Why is that?  I have a couple of theories.  First, the table isn’t very complex with only 4 columns and I’m only ever querying and updating based on the primary key.  I’m sure a more complex query would slow things down.  We also have no indexes on any the columns which can cause the database to do more work behind the scenes.

Regardless, we learned a few things today.  First, database transactions are critical to large scale database manipulation in a timely manner.  Second, it doesn’t matter if you use RecordSet Edit/Update or if you use PreparedStatements as either one is fast enough.

What sorts of questions do you have about databases that I can test for you?

Database Transactions in Xojo

Every now and then someone on the Xojo forum wonders why inserting data into an SQLite database is so slow.  Databases are designed to hold billions of records and do it fast, right?  So why is their application so slow?  The answer is they are relying upon the built-in transaction in Xojo.

By default SQLite databases do an automatic transaction for you.  This means that as soon as you attempt to insert, update, or delete data the work to write that change to disk happens as soon as possible.

For example, let’s take the following bit of code to insert data into a table:

For i As Integer = 1 To kMax
   Dim dbr As New DatabaseRecord
   dbr.Column("Name") = GetName
   dbr.IntegerColumn("Age") = GetAge
   dbr.DateColumn("TheDate") = New date
   db.InsertRecord "t_table", dbr
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

GetName gets a random name from an array that has 26 names in it and adds a random integer between 1000 and 10000.  GetAge returns a random integer between 1 and 74. 

This is pretty simple insert and if you run this 10,000 times it takes roughly 49 seconds saving to a desktop SQLite file on my 5k iMac.  And in so doing the application is locked up for that entire time because I’m in the tight loop.  This is simply unacceptable.

I’m sure someone is screaming why are you using the DatabaseRecord!  It’s slow!  It’s inefficient!  You should be using PreparedStatement’s!  Okay, so using pretty much the same logic:

Dim sql As String = "Insert into t_table (Name, Age, TheDate) Values(?, ?, ?);"
Dim ps As SQLitePreparedStatement = db.Prepare(sql)
ps.BindType(0, SQLitePreparedStatement.SQLITE_TEXT)
ps.BindType(1, SQLitePreparedStatement.SQLITE_INTEGER)
ps.BindType(2, SQLitePreparedStatement.SQLITE_TEXT)
For i As Integer = 1 To kMax
   ps.Bind(0, GetName)
   ps.bind(1, GetAge)
   ps.Bind(2, New Date)
   ps.SQLExecute
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

This takes about 48 seconds.  No big time savings there.  Obviously that’s not the improvement we need.

What we need to do is put these inserts in a database transaction using the following bit of code before the loop starts:

db.SQLExecute("BEGIN TRANSACTION")
If db.Error Then
   MsgBox "DB Error: " + db.ErrorMessage
   Return
End

And then at the end use:

db.Commit

Using the DatabaseRecord method takes a whopping 1 second with 10,000 records.  Using the prepared statement is so fast that my measured elapsed time in seconds is effectively 0.  

If I up the number of records inserted to 1,000,000 I get an interesting result.  The DatabaseRecord method takes 27 seconds where the PreparedStatement method takes 47 seconds.  And if I declare the PreparedStatement inside the loop it now takes 55 seconds.

What have we learned in this blog post?  First, using a database transaction is considerably faster than using the default transaction behavior.  Second, using DatabaseRecord is pretty fast and depending upon the number of record inserted it might be considerably faster.  Honestly, I didn’t expect this.  

In the next blog post I’ll look at Record updates and what are the best methods.

Nil Recordset

The question on how to deal with nil RecordSets is OFTEN asked in the Xojo forums.  Either the original poster or a well meaning responder will do something like this:

dim rs as recordset 
rs = db.sqlselect(sql)
If rs <> Nil Then
     If Not rs.EOF Then
       // No Records in Table or bad querry?
     End If
End If

It works, but it only masks the real issue.  Why is rs nil in the first place?  The answer is because the SQL statement to get the RecordSet had an error!  What is that error?  We have no idea because the code doesn’t ask for the error.

Most of the time the database will give you an error message that will point you in the right direction.  How do you tell that the database had an error?  You check the Error property.

dim rs as recordset 
rs = db.sqlselect(sql)
if db.error then
     //Do something
end


The database can give you an error code but this varies from database to database so I don’t find it exceptionally useful.  However, there is an ErrorMessage property that gives you a human readable message.  

dim rs as recordset 
rs = db.sqlselect(sql)
if db.error then
     MsgBox “DB Error: “ + db.ErrorMessage
     Return
end


In this example I’m showing this to the user but this is probably not a great idea for most uses because you don’t want the average user to know what happened.  However, it’s great while debugging.

The error you get back might be a little mysterious.  SQL is a language specific to databases and the error message returned expects you know a little bit about databases.

The second part of my message to you is that if you’re using SQLSelect or SQLExecute in your code you should look at PreparedStatements.  They’re safer in that PreparedStatement have some security features and does the heavy lifting of taking the input data and putting it into the right format.  What do I mean by this?

In a normal SQL statement you would have an SQL statement like this:

dim sql as string
sql = “Select * from users where last_name = ‘Keeney’;”


This selects all users whose last name is ‘Keeney’.  If you were going to allow the user to search for whatever name they want you would end up doing something like this:

dim sName as string
sName = txtSearchField.Text
sql = “Select * from users where last_name = '“ + sName + “’;”


Note that I had to manually put the single quotes in before and after the name since the database column is text.  This works great until you enter an Irish name like O’Neil.  The database will choke on this because it has an extra apostrophe in the statement.  You can escape this data yourself but what a pain.  Instead, use a PreparedStatement.

dim ps as SQLitePreparedStatement
dim sql as string
sql = “Select * from users where last_name = ?”
ps = db.prepare(sql)
ps.bindtype(0, SQLitePreparedStatement.SQLITE_TEXT
ps.bind(0, txtSearch.text)
dim rs as Recordset
rs = ps.SQLSelect
if db.error then
     MsgBox db.errormessage
     return
end


Nowhere in the code do I have to put in apostrophe’s or escape any data.  It’s just handled and this is true if you have database columns that are text, date, or numeric.

PreparedStatements have a number of advantages besides the automatic formatting.  They help with security and SQL Injection attacks.  If you don’t know what those are just do a web search to learn more.  You can reuse PreparedStatements in code but in my testing it doesn’t save much time as it appears as if the database plugin saves them internally.  Regardless, there are so many reasons to be using PreparedStatements over straight SQL.

To recap:  A nil recordset always mean a database error has happened.  You need to check for for the error after every operation whether that be a Select, Insert, Update, or Delete statement.  Basically after every SQLSelect and SQLExecute statement.  Ideally you want to start moving all of your code to use PreparedStatements because of all of its advantages.

The Xojo Community is Awesome

Have I told you how much I love the Xojo community?  I’ve been part of it for fifteen years and I’ve met hundreds of Xojo developers at developers conferences and probably exchanged emails with thousands more.  I am amazed at how much this community helps each other and I wish there was a way to promote that as a key feature of the product.  It’s a big deal.  Really!

If you’re just starting out using Xojo know that there are a bunch of people, myself included, that are willing to help out, if we can, on your journey.  Programming is hard.  Well, I don’t think it’s hard because I’ve been doing it for so long, but it is complex at times and that makes it hard.  Just ask your question in the Xojo forums and you’ll almost always get an answer within hours.

Even Xojo pros, such as myself, have need of help.  Xojo covers Mac, Windows, Linux desktop, console, and web apps.  It does iOS apps for iPhone and iPad.  It now does Raspberry Pi for heavens sake!  It works with dozens of different databases.  There is simply no way any one person is going to know everything there is to know about Xojo.  It just can’t happen.  So yes, I go to the forums, all the time, and ask for help.

Just the other day I asked for some help with WooCommerce.  Not Xojo related, really, but certainly related to a project we’re working on for a client.  Within a few hours I had half a dozen developers private message me saying they might be able to help.  Subsequent contact narrowed that list down a bit but the point is that I have probably shaved off several days worth of work simply by asking for advice.

I am biased towards Xojo, naturally, as it’s been my primary development language for fifteen years.  I think I’d be hard pressed to find such a friendly community.  I call many on the forums my friends even though I’ve never physically met them.  The few that I’ve met in person have lived up to their forum reputations and are really friends for life.

So maybe this is my belated Thanksgiving post.  I am thankful that so many years ago I jumped both feet first into the tool.  I asked questions – many of the silly and redundant.  I became more proficient and then made another jump to start blogging about it, making products for other developers, and training the next generation of developers.

So if you are in need of a cross-platform development tool I highly recommend Xojo.  It ain’t perfect but no development tool is.  If you jump in I think you’ll love the community.  I know I do.

What say you fellow Xojo developers?

Introducing ARGen 2.0

ARGen More Powerful Than Ever!

BKeeney Software is pleased to announce the release of version 2 of ARGen, our ActiveRecord generator utility.  The new release includes many enhancements.  Some of the highlights are:

  • Can now create User Interface elements.
  • Create entire projects for Desktop and Web projects with the proper database connections for each type.
  • Choose between standard database error reporting and a more robust version that BKeeney Software provides.
  • Can now create foreign key elements automatically.
  • Ability to create relationships without having to put them into foreign keys in the database.
  • Works with more databases.
  • Added ability to use database views.
  • Completely redesigned application!

Purchase Mac Version
Purchase Windows Version

Note:  Without the paid upgrade, you can still access the free version with nag screen.  You can use the free version with no time limitation.  The free version is limited to two tables at a time and will not create any User Interface elements.

If you are an existing user and did not receive an email containing an upgrade coupon code please contact us at support@bkeeney.com.

If you want to see ARGen 2.0 in action, please visit our new video at http://www.bkeeney.com/allproducts/argen/argen-2-0/

Product Home Page:  http://www.bkeeney.com/allproducts/argen/

XojoTalk 027 – Database Goddess

We at BKeeney Software are blessed in so many ways.  All of our employees bring a unique and interesting mix of talents and experiences.  It’s not just about one person and we often bounce ideas off each other to get the best possible result.

In the latest XojoTalk, Paul interviews our CEO, Carol Keeney, and how she uses Xojo and gets her thoughts on databases.    Carol has a ton of project management experience that makes running BKeeney Software easy since we often have a half dozen projects going at a time.

She has a lot of database experience too.  That has given her the “Database Goddess” nickname.  Again, that experience is so helpful for the programmers because, really, you don’t want your programmers designing the database.  We tend to do things the easy way which might not be the right way.

I thought it was an excellent interview.  You can find the XojoTalk at http://blog.xojo.com/2016/07/26/xojotalk-027-database-goddess/