Note: This document is for TGB versions 1.6.0 and greater.

Overview

File handling from the developer perspective should be as transparent as possible. That is to say, when handling files within script, only the game directory and file layout should be important. This means all paths should be relative to the game's root directory. And at no point in time should the developer have to deal with a full system path.

This is actually the way it has been with our engines in the past. But with the advent of the TGB Editor, and the complex OS requirements for directory access in both XP and Vista, changes were made that altered the behavior of file access. These changes were, in most cases, destructive to the nature of user transparent file handling and in large part the cause of many bug reports and complaints of the last product release.

With the release of TGB 1.6.0, effort was put forth to help separate the unique file handling needs of the tools from the strict file handling needs of the game. By doing so, this release will hopefully be brought back in line with how file access has been treated with all of our products in the past. It should be stated again, that at no point should the scripts for a game have file paths that are full system paths, and if at any instance this becomes the case, it should be regarded as a bug and reported. It should also be noted that scripts should never attempt to utilize full system paths when handling files as doing so is incorrect and will cause errors.

TGB 1.6.0 Changes

• Added setCompanyAndProduct console function
  • Takes two strings, (company, product)
  • This is a requirement to have in script
    • If using T2DProject template, this will be specified for them
    • See: tgb/T2dProject/common/gameScripts/properties.cs
    • Needs to be called as soon as possible, near beginning initial script execution
  • Make sure to edit commonConfig.xml to reflect your company and product when shipping
  • This function will populate the globals $Game::CompanyName and $Game::ProductName with appropriate values

• Added FileIOTest sample game
  • Contains simple file tests

• Removed the following console functions (for games). These functions will only be available for tools
  • getUserHomeDirectory
  • getUserDataDirectory
  • getPrefsPath
  • execPrefs

• Renamed $Game::GameName to $Game::ProductName
  • This is a required change, make sure any game scripts reflect this

• $Game::CompanyName and $Game::ProductName should be treated as read only values
  • Use setCompanyAndProduct (not necessary if using TGB Editor template scripts) and commonConfig.xml to set these values

• Changes to exec (for games)
  • Will now execute scripts in both the Game Directory and the Application Data Directory
  • When executing a script, will look for both the raw (*.cs) and the compiled (*.dso) before trying the Application Data Directory

• Changes to findFirstFile and findNextFile (for games)
  • Returns a file path relative to the Game Directory, eg: my_mod/some_folder/my_file.ext
  • Usage: it is incorrect to specify a search pattern which has a wild card at the beginning of a full relative path
    • example:
      • incorrect: */my_mod/some_folder/*.cs
      • correct: my_mod/some_folder/*.cs
      • correct: ~/some_folder/*.cs

• Changes to FileObject and SimXMLDocument (for games)
  • Will attempt to read from the Application Data Directory before trying the Game Directory
  • Will write only to the Application Data Directory

• Changes to GFont
  • Will attempt to read fonts from the Game Directory before trying the Application Data Directory
  • Will write cached fonts to the Application Data Directory if the font doesn't exist in the font cache of the Game Directory
  • The TGB Editor will now, on level save, populate all fonts used for the level and dump them into the font cache of the Game Directory

Important Locations

There are several important directories to keep in mind when developing and releasing a game based off of Torque Game Builder. However, for the game itself you only need to keep one directory in mind, and that is the game's directory. TGB attempts to build a sandbox around this directory. It does this by forcing the usage of relative directory paths. When creating your game scripts, keeping with these relative paths should be your only concern. Never attempt to use a full system path in the game scripts.

However, there are other directories that are important. These directories are:

• The Game Directory
• The Application Data Directory
• The Development Directory
• The Tools Directory

When shipping a product only the Game Directory and the Application Data Directory are important. When developing a product, typically the Game Directory and the Development Directory are the same. So what are these directories and why are they important?

The Game Directory

Is the location of the game executable and the game's data files. When developing your game, often the Game Directory will be the same directory as the Development Directory, or at least in close proximity. When shipping a product, the Game Directory will be whatever the user's of your game want it to be. This could be in the Program Files directory if they are on Windows, for instance. This means, you will have no control over this location and should never rely on a particular location. This is also why, when developing a game, you should never use full system paths to refer to game data files within your code and scripts.

The Application Data Directory

Is a special folder used to store application specific data. It is similar to My Documents in Windows, but the difference being My Documents is designed so a user can add or remove files from it at any time. Whereas the Application Data Directory is meant to service only the applications themselves. In other words, you should never directly change or delete files from this folder unless you know what you are doing.

On Windows XP this folder is located in %userprofile%\Application Data. However on Windows Vista this location was changed to %userprofile%\AppData\Local. On certain systems, these paths may be hidden from view. Refer to your OS documentation for instructions on how to find these locations.

One thing to keep in mind with this directory is that all files and folders need to be unique for each application that stores information here. Otherwise applications would be overwriting each other's data, which would be bad. Historically this is accomplished by storing the applications files in a directory naming the company and product. For your game, this is where the setCompanyAndProduct functionality becomes very important! If your game was generated by the TGB Tools template project, than to set your company and product information you should change the information contained in commonConfig.xml located in your game's /common/ folder.

I'm telling you these locations because you should be aware of them when developing your game. You may have to delete the files your game puts in there, for testing purposes, etc. However, from the scripting side, this directory is hidden from you in such a way that, even though the engine may be reading or writing from there, you should never know it.

The Development Directory

Is the location being used by a developer to store the game's files while he or she is working on the development of it. This could be anywhere the developer chooses. But never fall under the impression that the development directory of the game will be the same as the installed location on an end user's machine. From the developer perspective, it is usually the case that the development directory and the game directory are the same -- for that developer.

The Tools Directory

Is the location of the Torque Game Builder editor. Again, this location can vary quite a bit from developer to developer. This directory is quite complex. It holds the executable and data files for the editor itself. As well as example games and the project template used to generate a working skeleton game project. This directory should have no bearing for any of your game scripts and should not be referred to in any of your game scripts. This directory is also, obviously, not going to be shipped with your product.

Under the Hood

So while it is important that the game scripts remain oblivious to what is actually happening when reading or writing files, the developer should not be. Most of the functions exposed to script will now attempt to find files in two different locations, the Game Directory and the Application Data Directory. In addition, console functions which are used to write files will now only write to the Application Data Directory. The exception being the compiled (*.dso) scripts which are still written to the Game Directory.

The file routines responsible for reading files will attempt to read files from two locations. The Game Directory and the Application Data Directory. Some routines will try the Game Directory first, while others will try the Application Data Directory first. They were designed this way so that, in general, the routines tried the location most likely to contain the file first. For example, in the case of the console function exec, the engine will attempt to locate the raw (*.cs) and compiled (*.dso) script file in the Game Directory before attempting the Application Data Directory as the Game Directory is the most likely location for script files.

The file routines responsible for writing files will now only write files to the Application Data Directory. The one exception being the console function compile which will read a raw (*.cs) script file and write a compiled (*.dso) to the Game Directory. In the future, the reading and writing of script files will most likely change, especially in regards to (*.dso) files.

When writing files, the Game Directory is essentially 'mirrored' in the Application Data Directory. This is accomplished by appending the expanded relative path given to the console function onto the end of the Application Data Directory. In this way, the developer only has to refer to files via their relative path (like would normally be the case) and the engine can do the lookup automatically in both the Game Directory and the Application Data Directory without any additional effort from the developer.

There is one 'gotcha' about the console function exec. If two separate script files exist, and if each are named the same. And if one is located in the Game Directory, and the other is located in the Application Data Directory, but the relative paths are the same. The engine will never execute the file located in the Application Data Directory. This is because exec will first look for the file in the Game Directory. Since it will find it there, it will never attempt to look in the Application Data Directory. This same scenario also applies to FileObject, but in reverse. So keep this in mind when laying out the directory and file structure of your game.

The Idiosyncrasies of findFirstFile / findNextFile

The console functions findFirstFile and findNextFile can be used to find files matching a particular pattern. A wild card is used to represent the unknown portions of the pattern. These console functions will then use that pattern to match files which are then returned. By setting up findFirstFile and findNextFile in a for loop, it is possible to iterate over all possible matches found by the pair console functions. However, these two console functions can sometimes be a pain to get working correctly. Hopefully this section of the article will shed some light on the odd inner workings of these two functions.

One important detail about these functions is they are 'location' aware. What this means is the functions know which directory they are in, which is based on the script file with which it belongs. In fact, this is true for all the script functions, but can be particularly noticeable with these two functions. This is what makes it possible to expand relative paths containing the ' ./ ' characters.

One example of how this might become an issue is if you had written a utility function which wraps the functionality of findFirstFile and findNextFile into a single function to execute files matching a pattern and then called that function from a script in a different directory but used a file path pattern with the ' ./ ' characters. It would fail, and miserably so! This is because the ' ./ ' characters would get expanded relative to the script containing your utility function and not relative to the script calling your utility function. For this reason, I personally always make sure to pass findFirstFile and findNextFile full relative paths along with the search pattern.

Both of these console functions return a string, with a file path matching the pattern given. When you first call findFirstFile, internally it is going to remember the last match returned to script. It does this so that when you proceed to call findNextFile, it is able to use that last match to get the next match. And this continues with findNextFile until there are no more matches left, each call storing the last match returned to script. findFirstFile is special in that it does the setup work necessary for findNextFile to continue the process of finding the files. In other words, call findFirstFile once, and than call findNextFile repeatedly in a loop, to find all the files matching the pattern given.

Another detail about these two functions, if you want them to work correctly anyway. Never mix and match the file patterns given to findFirstFile and then findNextFile. If you start it off with a particular pattern, continue to use that pattern for the duration of the loop. Remember, these two functions are meant to be used as a pair. In all honesty, the findNextFile console function shouldn't let you change the pattern... perhaps a design flaw.

When specifying a pattern to these functions, you can use any number of wild cards. The two wild cards that are recognized are the ' * ' and the ' ? '. One thing to remember however, it is incorrect to specify a wild card at what would be considered the 'root' of a relative path. In other words if you have a mod folder named my_mod and you want to find all script files in a folder named some_folder a search pattern like so will not work: */some_folder/*.cs. The correct search pattern would be either: my_mod/some_folder/*.cs or ~/some_folder/*.cs. This has always been the case, but with the file changes that were present in the last release, the incorrect pattern mentioned above was being used to 'trick' these console functions into ignoring the full system path. Now that these functions no longer return a full system path, and instead return a relative path, that incorrect pattern will no longer work.

// dumps out all of the files in the current mod directory:
%search = "~/*";
for (%file = findFirstFile(%search); %file !$= ""; %file = findNextFile(%search))
   echo("File:" SPC %file);

// dumps out only the (*.gui) files located in common/ mod directory:
%search = "common/*.gui";
for (%file = findFirstFile(%search); %file !$= ""; %file = findNextFile(%search))
   echo("File:" SPC %file);

Summary

I think the single biggest thing to take away from this article is that, as developers, we have no control over where an end user will install our game. This means it is foolish to use full system paths when defining the locations of our game files. Another lesson to learn here, as well, is sand-boxing file access is important. By enforcing the relative paths rule, we prevent lazy development practices, which ultimately decreases the number of potential bugs.