Test Data

At XDC 2019 my session was titled Xojo Design Mistakes (the alternate but way longer title was ‘Thankfully time travel doesn’t exist or my future self might travel back and murder my younger self for the stupid coding mistakes I’ve made’).  These are things that I’ve discovered over the years, both in my own projects and in other peoples projects that are just plain wrong or less than ideal.  This will be an on-going series since I had well over 50 slides and left 150 out.  So when I get bored I’ll bring these topics up.

Nearly all of our consulting projects are database driven applications.  It’s why we’ve created the tools to help with these projects like ARGen, which simplifies our interactions with the database, and BKS Shorts, which is our own reporting tool.  These tools are invaluable in getting our work done in a timely matter.

In a database application it’s typical to have a List of something.  A common example of this is a Customers list.  In that list the client typically wants the ability to Create, Read, Update, and Delete (or CRUD) a customer with varying degrees of rules behind it (like do they have permissions to add or delete a customer?).

During development we get the List form going, add the controls to be able to add a new record.  Then we create the Add/Edit form that allows us to test those capabilities.  We create a few, update a few, delete a few customers and then move on.  Maybe the client wants search capabilities so we add that to the List window and when we’ve tested it against our half dozen or so records we move on to the next task.

There is nothing wrong with this process.  It works and it’s fairly efficient as far as it does.  However, there’s one thing we’ve skipped that’s really important but also difficult to achieve.

So far we’ve test with *maybe* a dozen records.  What happens when the client adds 10,000, or 100,000 Customer records?  Does the list form take a long time to load?  Does the search function take a long time?  What about the Customer popup menu’s that you’ve scattered throughout the project – are those now slow, unwieldy, and unusable?

Unfortunately, with the way we implemented the project we don’t know how any of this works since we only have a dozen records.  So it’s really important to have adequate amounts of test data.  Creating 10,000 new customers using your new interface would take a long time.  So what can you do?

There are tools out there that will help generate data sets.  These tools allow you to create thousands, even millions of rows of realistic data.  Randomized male and female first names along with a last names is a great way to generate customer names.  Many tools allow you to add random dates in a range, random IP addresses, random values from a  list you provide and so on.  The sky is the limit when it comes to what sort of data developers need.

Now, when you do your testing you see how your application reacts with a lot of data.  I almost guarantee that it will act different.  Do you need to switch to a data-on-demand listbox?  Do you need to put an index on a common searchable field to speed up indexing?  Do you need to implement Full Text Search in your database?  Having a huge amount of data will answer these questions for you.

I once worked on an accounting application in VB6 where the original database designer using an Access database and did an account balance on the fly iterating through bills, checks, journal entries, etc. With a few thousand rows of data in each table this process took a second or two for all balances on a local machine. When this database was accessed over the network it took 5 to 7 seconds. When we converted our first client database it took 30 to 40 seconds for EACH account! Obviously this was not acceptable performance from an accounting application meant to be used daily by general contractors with hundreds of employees and tens of thousands of customers. The solution was to have a current balance value that was stored and then updated when a transaction occurred. We could have saved ourselves hundreds of hours of rushed development time (and much stress and heartache) if we had tested with large amounts of data much earlier in the process.

I mentioned adding an Index to a field earlier. One word of caution on this: it’s tempting to add an index to every field you’re searching on. Don’t do this! Only added indexes to the most important fields in a table. For a customer maybe the two most important fields are phone number and name even though you search on City and things like that. Indexing is extra work for the database so performance can take a signifiant hit with indexing a field.

Since the toolI’ve been using to create test data is no longer being sold I’m curious what you’d recommend.  Do you have a favorite tool?  Or is this a tool that would be of use to the community?

Happy Coding!

ARGen 3.0.3

BKeeney Software Inc. is proud to announce an update to ARGen, our ActiveRecord Generator utility for Xojo developers. This minor update includes dark mode support, speed improvements, and important updates for generated projects. Updating to 3.0.3 is recommended for all ARGen users.

ARGen is available for macOS and Windows. It can be used for free in limited mode, and is priced at $99.95 to unlock all features. Existing version 2.x users will automatically be provided an upgrade opportunity when launching version 3.

3.0.3 Release Notes:

Changes:

  • Added Dark Mode support
  • Simplified manual relationship management
  • Selecting a different SQLite database now clears the password field
  • kMaxReturn is now a protected constant for cleaner code
  • DBUpdates module code is now cleaner
  • Improved instructions in some locations
  • Base project templates optimized
  • Preferences module no longer writes to SpecialFolder.Preferences
  • iOS Create Data Sources defaults to true
  • Updated links to Xojo documentation
  • Generated localization module constants are now protected

Fixes:

  • DBUpdates.SetDBVersion no longer uses a BKS extension synonym for str()
  • Fixed return statement for iOS apps using 2018r2
  • Projects with empty name now have default save name
  • BKS Created/Modified overrides no longer generate properties that fail to Register
  • Corrected minor UI bug on Windows
  • Project listing loads faster
  • Speed improvements throughout the software
  • Projects created but never saved are no longer retained when closed
  • Checking for updates at launch now works
  • Preferences window will show the last update check time

Pricing, examples, and more details can be found at the project homepage at https://bkeeney.com/argen/ 

Updating Records with Transactions

In my last blog post I talked about inserting records into a SQLite database and found that using transactions makes them incredibly fast.  Does the same hold true for updating an existing record?  We’ll find out.

In this set of tests I inserted 100,000 records and then randomly selected 10,000 records to update with our same random name, age, and date values.  In theory this means I have records distributed all over the range of the database.

In the first test we use a simple RecordSet which means we have to query for it first.  Then call its Edit method, change the values and call update.  It’s a two step process.

For i As Integer = 0 To ari.Ubound
   Dim rs As recordset
   rs = db.SQLSelect("Select * from t_table where table_id = " + Str(ari(i)))
   If db.error Then
      If db.Error Then
         MsgBox "DB Error: " + db.ErrorMessage
         Return
      End
   End
   rs.edit
   rs.Field("Name").stringvalue = GetName
   rs.Field("Age").integervalue = GetAge
   rs.Field("TheDate").DateValue = New date
   rs.Update
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

Unsurprisingly, this method took about 49 seconds for 10,000 records.

In test two I switched to using a PreparedStatement to update the record.

Dim sql As String = "Update t_table Set Name = ?, Age=?, TheDate =? WHERE table_id = ?"
Dim ps As SQLitePreparedStatement = db.Prepare(sql)
ps.BindType(0, SQLitePreparedStatement.SQLITE_TEXT)
ps.BindType(1, SQLitePreparedStatement.SQLITE_INTEGER)
ps.BindType(2, SQLitePreparedStatement.SQLITE_TEXT)
ps.BindType(3, SQLitePreparedStatement.SQLITE_INT64)
For i As Integer = 0 To ari.Ubound
   ps.Bind(0, GetName)
   ps.bind(1, GetAge)
   ps.Bind(2, New Date)
   ps.bind(3, ari(i) )
   ps.SQLExecute
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

When run for 10,000 records this takes about 47 seconds so there is a little bit of time savings by not querying the database for the data to begin with.  It’s also possible that the PreparedStatements aren’t nearly as efficient as one would hope.

I then repeated the same tests within a Transaction.  In both cases updating records effectively took no time at all!  If I increase the number of Updates to 10,000 both the RecordSet update and the PreparedStatement takes about 8 seconds.  

I’m surprised at these results because you would think that doing the query required for the RecordSet would take more time but it’s no different than using the PreparedStatement SQL Update.  Why is that?  I have a couple of theories.  First, the table isn’t very complex with only 4 columns and I’m only ever querying and updating based on the primary key.  I’m sure a more complex query would slow things down.  We also have no indexes on any the columns which can cause the database to do more work behind the scenes.

Regardless, we learned a few things today.  First, database transactions are critical to large scale database manipulation in a timely manner.  Second, it doesn’t matter if you use RecordSet Edit/Update or if you use PreparedStatements as either one is fast enough.

What sorts of questions do you have about databases that I can test for you?

Database Transactions in Xojo

Every now and then someone on the Xojo forum wonders why inserting data into an SQLite database is so slow.  Databases are designed to hold billions of records and do it fast, right?  So why is their application so slow?  The answer is they are relying upon the built-in transaction in Xojo.

By default SQLite databases do an automatic transaction for you.  This means that as soon as you attempt to insert, update, or delete data the work to write that change to disk happens as soon as possible.

For example, let’s take the following bit of code to insert data into a table:

For i As Integer = 1 To kMax
   Dim dbr As New DatabaseRecord
   dbr.Column("Name") = GetName
   dbr.IntegerColumn("Age") = GetAge
   dbr.DateColumn("TheDate") = New date
   db.InsertRecord "t_table", dbr
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

GetName gets a random name from an array that has 26 names in it and adds a random integer between 1000 and 10000.  GetAge returns a random integer between 1 and 74. 

This is pretty simple insert and if you run this 10,000 times it takes roughly 49 seconds saving to a desktop SQLite file on my 5k iMac.  And in so doing the application is locked up for that entire time because I’m in the tight loop.  This is simply unacceptable.

I’m sure someone is screaming why are you using the DatabaseRecord!  It’s slow!  It’s inefficient!  You should be using PreparedStatement’s!  Okay, so using pretty much the same logic:

Dim sql As String = "Insert into t_table (Name, Age, TheDate) Values(?, ?, ?);"
Dim ps As SQLitePreparedStatement = db.Prepare(sql)
ps.BindType(0, SQLitePreparedStatement.SQLITE_TEXT)
ps.BindType(1, SQLitePreparedStatement.SQLITE_INTEGER)
ps.BindType(2, SQLitePreparedStatement.SQLITE_TEXT)
For i As Integer = 1 To kMax
   ps.Bind(0, GetName)
   ps.bind(1, GetAge)
   ps.Bind(2, New Date)
   ps.SQLExecute
   If db.Error Then
      MsgBox "DB Error: " + db.ErrorMessage
      Return
   End
Next

This takes about 48 seconds.  No big time savings there.  Obviously that’s not the improvement we need.

What we need to do is put these inserts in a database transaction using the following bit of code before the loop starts:

db.SQLExecute("BEGIN TRANSACTION")
If db.Error Then
   MsgBox "DB Error: " + db.ErrorMessage
   Return
End

And then at the end use:

db.Commit

Using the DatabaseRecord method takes a whopping 1 second with 10,000 records.  Using the prepared statement is so fast that my measured elapsed time in seconds is effectively 0.  

If I up the number of records inserted to 1,000,000 I get an interesting result.  The DatabaseRecord method takes 27 seconds where the PreparedStatement method takes 47 seconds.  And if I declare the PreparedStatement inside the loop it now takes 55 seconds.

What have we learned in this blog post?  First, using a database transaction is considerably faster than using the default transaction behavior.  Second, using DatabaseRecord is pretty fast and depending upon the number of record inserted it might be considerably faster.  Honestly, I didn’t expect this.  

In the next blog post I’ll look at Record updates and what are the best methods.

Nil Recordset

The question on how to deal with nil RecordSets is OFTEN asked in the Xojo forums.  Either the original poster or a well meaning responder will do something like this:

dim rs as recordset 
rs = db.sqlselect(sql)
If rs <> Nil Then
     If Not rs.EOF Then
       // No Records in Table or bad querry?
     End If
End If

It works, but it only masks the real issue.  Why is rs nil in the first place?  The answer is because the SQL statement to get the RecordSet had an error!  What is that error?  We have no idea because the code doesn’t ask for the error.

Most of the time the database will give you an error message that will point you in the right direction.  How do you tell that the database had an error?  You check the Error property.

dim rs as recordset 
rs = db.sqlselect(sql)
if db.error then
     //Do something
end


The database can give you an error code but this varies from database to database so I don’t find it exceptionally useful.  However, there is an ErrorMessage property that gives you a human readable message.  

dim rs as recordset 
rs = db.sqlselect(sql)
if db.error then
     MsgBox “DB Error: “ + db.ErrorMessage
     Return
end


In this example I’m showing this to the user but this is probably not a great idea for most uses because you don’t want the average user to know what happened.  However, it’s great while debugging.

The error you get back might be a little mysterious.  SQL is a language specific to databases and the error message returned expects you know a little bit about databases.

The second part of my message to you is that if you’re using SQLSelect or SQLExecute in your code you should look at PreparedStatements.  They’re safer in that PreparedStatement have some security features and does the heavy lifting of taking the input data and putting it into the right format.  What do I mean by this?

In a normal SQL statement you would have an SQL statement like this:

dim sql as string
sql = “Select * from users where last_name = ‘Keeney’;”


This selects all users whose last name is ‘Keeney’.  If you were going to allow the user to search for whatever name they want you would end up doing something like this:

dim sName as string
sName = txtSearchField.Text
sql = “Select * from users where last_name = '“ + sName + “’;”


Note that I had to manually put the single quotes in before and after the name since the database column is text.  This works great until you enter an Irish name like O’Neil.  The database will choke on this because it has an extra apostrophe in the statement.  You can escape this data yourself but what a pain.  Instead, use a PreparedStatement.

dim ps as SQLitePreparedStatement
dim sql as string
sql = “Select * from users where last_name = ?”
ps = db.prepare(sql)
ps.bindtype(0, SQLitePreparedStatement.SQLITE_TEXT
ps.bind(0, txtSearch.text)
dim rs as Recordset
rs = ps.SQLSelect
if db.error then
     MsgBox db.errormessage
     return
end


Nowhere in the code do I have to put in apostrophe’s or escape any data.  It’s just handled and this is true if you have database columns that are text, date, or numeric.

PreparedStatements have a number of advantages besides the automatic formatting.  They help with security and SQL Injection attacks.  If you don’t know what those are just do a web search to learn more.  You can reuse PreparedStatements in code but in my testing it doesn’t save much time as it appears as if the database plugin saves them internally.  Regardless, there are so many reasons to be using PreparedStatements over straight SQL.

To recap:  A nil recordset always mean a database error has happened.  You need to check for for the error after every operation whether that be a Select, Insert, Update, or Delete statement.  Basically after every SQLSelect and SQLExecute statement.  Ideally you want to start moving all of your code to use PreparedStatements because of all of its advantages.

Automating Xojo Database Development with ARGen

Database applications are relatively easy with Xojo. The native Database, DatabaseRecord, and Recordset classes make it easy to connect, add, edit, and delete data in your console, desktop, and web applications. What we’ve found over the years isn’t that it’s hard to create these applications but that it’s tedious and also the very nature of Xojo tends to make life more complicated. Let me explain.

The Xojo database classes are very generic. To insert data into the recordset you either use an SQL Insert statement or use the DatabaseRecord class. The SQL statement, by itself if fraught with peril as it’s very easy to mistype the statement and miss it in testing. Even using PreparedStatements (which you should be using whenever possible) you can still screw it up. Using the DatabaseRecord class is more object oriented and without a doubt easier to use but it has one big problem: It doesn’t check datatypes. At design time it will happily let you try to put a date value into an integer field (depends on how you try, of course). It is not until runtime that you may get any sort error which leads to the next issue.

Database errors, at least in the global framework, do not generate exceptions. This means that unless you, the developer, check for the error, you may not get the results you expect. On an Insert statement, either using SQL or the DatabaseRecord, your data may simply never be inserted and you’ll never know it unless you’re checking the Error property on the database connection. The Error must be checked after EVERY SQL action whether that be SQL Select, Insert, or Delete. Even checking after a Prepare statement is wise to do so.

The good news (maybe) is that Xojo for iOS and the iOSSQLiteDatabase class will throw an exception if it experiences a database error. It’s good that you don’t need to check for the error manually, but now you need to put in exception handling around your database code and deal with exceptions accordingly. It’s not hard, but it’s also not trivial code either.

The IDE knows next to nothing about the database and certainly when you code it you have to query the database for Table and Field schema’s. To make it worse, the Xojo database framework has no idea about the concept of Views requiring you to query the database specifically to find and learn more about them. The IDE has no idea about the field datatypes either and the compiler can’t warn you about mismatched datatypes. These types of errors are only caught at runtime which is usually way too late in the process (hopefully not by the client either).

The database frameworks are not very object oriented friendly. If you have a simple Add/Edit dialog you need to know if the record is new, or not, since this changes how you code saving the data. New requires the DatabaseRecord object and an existing record can be updated via the Recordset object (both can use SQL statements but the statements themselves require different statements).

Xojo is easy to use but the ability to stick code in control events tends to make big database applications unwieldy and full of spaghetti code. Changing a field name in a commonly used table, or even worse changing the field datatype, is often at your own peril because you might have SQL statements referencing it in hundreds of places in the code. Forget it one place and you have an error in the waiting.

Finally, there’s the hooking your user interface up to your database code. Many people have Load/Save methods in every window/dialog/container that needs it. Use a SQL statement to get the data, then use the Recordset to load the user interface, and then use a Save method (that knows the difference between new and existing records) to put it back into the database. Again, the IDE compiler can’t help you if you make either a spelling mistake in the SQL statement or Field names so you won’t find these errors out until runtime (which is often too late).

There are options, to all of these problems. Creating your own data classes is a step above the standard Xojo code, in my opinion. You encapsulate a record into a Xojo class and implement your own Load and Save methods. This is a better object and forces some compiler warnings into your code. It also tends to put your database code all in one folder and/or NameSpace. You can use field name constants for field names, and force all db code into one class or module. Unfortunately, if you have hundreds of tables it’s also tedious to do all this.

There are other solutions out there, but ARGen takes some of the tediousness out of the work of creating a Xojo database application and does some other nifty things too like creating a basic User Interface for you.. ARGen is short for ActiveRecord Generator. ActiveRecord is a set of classes that map the database into equivalent Xojo namespaced classes.

Say you have a table named Employee. ARGen would create a class named ‘Employee’ in the ‘Data’ namespace and you would reference it in code as Data.Employee. Each field in the table is then mapped to a property in the class. The FirstName text field would be mapped to a string property named “FirstName”. This property is referenced as Data.Employee.FirstName and since the compiler knows this property is a string it will complain if you try and put an integer into it. Since there’s only one place you would ever define FirstName in your project it becomes really easy to change the name or change the datatype and then have the compiler catch any errors. It also means that AutoComplete in the code editor works meaning you’ll never mistype a table or field name again without the compiler catching it.

When an ActiveRecord project first starts up and connects to the database we call a Register method that calls out which tables/views are added into ActiveRecord. ActiveRecord scans the schema of the table and then attempts to match up every field with a corresponding property on the class. If a field is missing from your class it will generate an error that’s caught only while debugging (a “table x is missing field y” type of message).

Each ActiveRecord instance is a complete record. It has built-in Save and Delete methods. It also has events for Before and After Create, Before and After Update, to let you do things globally for each record. One example we use all the time are CreatedBy, CreatedDate fields in the Before Create event and ModifiedBy and ModifiedDate fields in the Before Update event. One place, ever, to put that data into the database rather than the potential thousands of places it might. There is also a convenient IsModified method to figure out if data has actually changed or not.

The Save method knows if it’s a new record or an existing record and does the right thing behind the scenes for you. Using a transaction, using a PreparedStatement. I think this one feature alone saves us hundreds of hours of coding because it’s a simple Save call. Nothing more be done. It really takes the worry out of it.

There are other coding features that we could get into but it all revolves around making creating database applications in Xojo fast and easy. ARGen creates a bunch of helper functions like IsDuplicate, List, and so on to help make your life easier. ARGen has a free mode which allows you to get the ActiveRecord classes and up to two tables at a time done.

For many years this is all ARGen did and we used it with great success. However, we found ourselves spending a lot of time on the next step – creating windows, dialogs, and containers and hooking up the database classes to the User Interface. Practically every table you run across has a List and Edit form. In version 2 we added the ability create a rudimentary User Interface which saves even more time. You only get this ability with the fully paid version.

When generating a Xojo project ARGen puts in a number of #pragma error statements where the developer needs to look at code and uncomment code and/or fix code. It’s hard to guess and frankly we don’t want your newly generated project to compile without you having to fix some code. Trust me – it’s better this way. There are simply too many variables and ways of doing things. Because of the #pragma error statements I like to bring over classes and UI over from the generated project as I need them rather than all at once. No need to fix hundreds of pieces of code until you’re ready to look at them.

ARGen and ActiveRecord isn’t perfect and it’s not the only solution around. ARGen lets the compiler do some work for you and eliminates some of the very common, but trivial, mistakes. It also lets you save a ton of time when it comes to standard database code and building user interfaces. It isn’t a panacea for development but ARGen can save you a lot of time and effort.

If you’d like to learn more about ARGen and what it can do for you, please download the free/demo version from the product page at http://www.bkeeney.com/allproducts/argen/. There are several videos on the ARGen menu that show more details.

Update:  I did a Xojo webinar a while back that talks about some of this in detail.  It’s a little dated but worth watching:  http://developer.xojo.com/webinar-simplying-db-access-with-bkeeney-activerecord

The Xojo Community is Awesome

Have I told you how much I love the Xojo community?  I’ve been part of it for fifteen years and I’ve met hundreds of Xojo developers at developers conferences and probably exchanged emails with thousands more.  I am amazed at how much this community helps each other and I wish there was a way to promote that as a key feature of the product.  It’s a big deal.  Really!

If you’re just starting out using Xojo know that there are a bunch of people, myself included, that are willing to help out, if we can, on your journey.  Programming is hard.  Well, I don’t think it’s hard because I’ve been doing it for so long, but it is complex at times and that makes it hard.  Just ask your question in the Xojo forums and you’ll almost always get an answer within hours.

Even Xojo pros, such as myself, have need of help.  Xojo covers Mac, Windows, Linux desktop, console, and web apps.  It does iOS apps for iPhone and iPad.  It now does Raspberry Pi for heavens sake!  It works with dozens of different databases.  There is simply no way any one person is going to know everything there is to know about Xojo.  It just can’t happen.  So yes, I go to the forums, all the time, and ask for help.

Just the other day I asked for some help with WooCommerce.  Not Xojo related, really, but certainly related to a project we’re working on for a client.  Within a few hours I had half a dozen developers private message me saying they might be able to help.  Subsequent contact narrowed that list down a bit but the point is that I have probably shaved off several days worth of work simply by asking for advice.

I am biased towards Xojo, naturally, as it’s been my primary development language for fifteen years.  I think I’d be hard pressed to find such a friendly community.  I call many on the forums my friends even though I’ve never physically met them.  The few that I’ve met in person have lived up to their forum reputations and are really friends for life.

So maybe this is my belated Thanksgiving post.  I am thankful that so many years ago I jumped both feet first into the tool.  I asked questions – many of the silly and redundant.  I became more proficient and then made another jump to start blogging about it, making products for other developers, and training the next generation of developers.

So if you are in need of a cross-platform development tool I highly recommend Xojo.  It ain’t perfect but no development tool is.  If you jump in I think you’ll love the community.  I know I do.

What say you fellow Xojo developers?

Introducing ARGen 2.0

ARGen More Powerful Than Ever!

BKeeney Software is pleased to announce the release of version 2 of ARGen, our ActiveRecord generator utility.  The new release includes many enhancements.  Some of the highlights are:

  • Can now create User Interface elements.
  • Create entire projects for Desktop and Web projects with the proper database connections for each type.
  • Choose between standard database error reporting and a more robust version that BKeeney Software provides.
  • Can now create foreign key elements automatically.
  • Ability to create relationships without having to put them into foreign keys in the database.
  • Works with more databases.
  • Added ability to use database views.
  • Completely redesigned application!

Purchase Mac Version
Purchase Windows Version

Note:  Without the paid upgrade, you can still access the free version with nag screen.  You can use the free version with no time limitation.  The free version is limited to two tables at a time and will not create any User Interface elements.

If you are an existing user and did not receive an email containing an upgrade coupon code please contact us at support@bkeeney.com.

If you want to see ARGen 2.0 in action, please visit our new video at http://www.bkeeney.com/allproducts/argen/argen-2-0/

Product Home Page:  http://www.bkeeney.com/allproducts/argen/

XojoTalk 027 – Database Goddess

We at BKeeney Software are blessed in so many ways.  All of our employees bring a unique and interesting mix of talents and experiences.  It’s not just about one person and we often bounce ideas off each other to get the best possible result.

In the latest XojoTalk, Paul interviews our CEO, Carol Keeney, and how she uses Xojo and gets her thoughts on databases.    Carol has a ton of project management experience that makes running BKeeney Software easy since we often have a half dozen projects going at a time.

She has a lot of database experience too.  That has given her the “Database Goddess” nickname.  Again, that experience is so helpful for the programmers because, really, you don’t want your programmers designing the database.  We tend to do things the easy way which might not be the right way.

I thought it was an excellent interview.  You can find the XojoTalk at http://blog.xojo.com/2016/07/26/xojotalk-027-database-goddess/

Views are Your Friend

One of the challenges I see regularly in my work as a database administrator is finding a way to bridge the gap between the well-normalized physical implementation of a database behind an application and providing a way to make the data available in an easy-to-use (and understand) fashion for the end user.  For many clients this means letting them design their own reports.  Database views are my go-to method for making a complex database structure easy to use for end-users and even developers.

Views have a number of advantages.  For one, they are available in all SQL databases, including SQLite.  Views allow you to reduce the complexity of the end user experience by managing and limiting the data presented.  This might mean taking a very complex query joining multiple tables and presenting it in a single table with the end user (or developer) not needing to know the gory details of the SQL behind it.

This last point is important for the developers too.  Having the programmers figure out the complex joins for a query pulling from multiple tables is sometimes challenging and not always a good use of their time.  Having the Xojo application do all that work for a complex query is sometimes painfully slow.  Views not only speed up the development process but make the Xojo application more efficient since the database has already created the view and optimized it internally.

Views are more secure since you can limit the data shown and control who has the rights to view it.  By giving users and developers access to the underlying tables you may be exposing sensitive data.  Views are an easy way of sanitizing data.  We use views as read-only constructs so the user cannot update the data (note:  not all databases treat views as read-only).

Views can be created, modified and deleted via simple SQL statements.  The syntax of the statements (in particular for modifying views) varies a bit from database to database but the details can be easily found.  To create a view, the syntax is:

    CREATE VIEW viewname AS

SELECT (fill in your query here)

We recently used views implementing a large customer records management (CRM) system for an insurance broker.  Their database has around 100 tables containing information on customers, policies, related products, agent and commission data.  The data is normalized and it isn’t always straightforward to get related data.  The client does not have a dedicated IT staff and reporting is the primary responsibility of a part-time administrative employee, “Jane”.

Our goal was to provide Jane with the easiest means possible to write any report that was requested by management.  While Jane has some knowledge of databases and foreign keys, she isn’t technical and doesn’t have any training on the SQL language.

Our answer was to use our desktop reporting tool, BKeeney Shorts, customized to only show her views created solely for reporting.  (We did this via a simple naming standard in the database.)  She is presented with options such as:  Agent Commission, Prospect Contact and Client Policy Details.  If she needs an additional view, it’s easy to add it to the database and not impact the Xojo code behind the application. (It doesn’t require a recompile and redeployment of the software).

The view incorporates the calculations behind the data, so the reports are always consistent in terms of things such as how commission checks are calculated or who is the primary agent for an account.  By using views we ensure that consistent results are shown to the end user even if someone else is assigned to write reports

Views provide a level of abstraction between the database and the end user and we find  them very useful.  For the developers, it means they don’t have to figure out the complex queries and have the Xojo application create inefficient and complex queries.  Views are your friend!