TSE/Atlas/Overview
From TDN
| This article or section needs updating. Parts of this article or section have been identified as out of date. Reason: The Atlas2 updated changed the methods for importing terrain some. See Guide to Using Atlas2 for now in addition to this article. We need a unification of those two pages. If you are able to revise this information, please do so and remove this 'update' flag when finished. |
Contents |
Atlas Engine Workflow
Legacy terrain had the benefit of having a full set of interactive in-engine editors. This made it very easy to work with the terrain, tweak it, and so forth. Unfortunately, full in-engine editing support for the chunk terrain is slated for a future milestone. Currently, the Atlas terrain engine has a slightly more cumbersome, but more flexible tool pipeline associated with it.
A general note: when working with tree depths, bear in mind that 2treeDepth-1 is the number of nodes on a side that tree will be. This becomes important when working with the TQT generator, for instance, when you implicitly describe the size of the input texture by treeDepth and leafSize. This means that 2treeDepth-1 * leafSize = imageSize.
Generating a chunk geometry (.chu) file
Atlas terrains are very efficient because they simply stream data describing chunks of precalculated levels of detail from the disk. These files are generated in a preprocessing step which is done offline, as it can potentially take a bit of time to process a heightfield (a 4097x4097 heightfield recently took 5-10 minutes to process).
The geometry processing is done by a console function called generateChunkFileFromRaw16. This interface will be extended for later milestones (or you can refer to the code contained at the end of engine/atlas/chunkGen.cpp and write your own variant of the function), but currently only loads from a raw unsigned 16bit heightfield, such as Terragen produces.
The values from the .RAW file are signed 16 bit integers. They go from about -32767 to +32767. In whole numbers. Atlas works in whole meter units, so that means you get from -32km to +32km, in whole meter increments. This might be useful in some cases but for a terrain you walk on is way too much!
So we have a scaling factor. The raw integer values are multiplied by the scaling factor to convert to floating point values that are more useful for realistic terrains.
The error metric is applied after the scaling factor. So if you change the scaling factor you probably have to change the error metric. If the error metric isn't big enough it will crash. If it's too big you'll get an undetailed terrain. There is another parameter for horizontal spacing, the squareSize, but it just affects how values are written out, and won't affect processing time or crashes/accuracy.
The mesh generation works via a quadtree. Each vertex is descended and a delta is calculated from that vertex's real position to where (essentially) the mesh would be at that point if the vertex was present. The distance is divided by the error metric. The resulting number is rounded down to a whole number. That number is the LOD at which that vertex will show up. When the quadtree is generated, it works up from the lowest levels of detail.
(Sort of. This summary leaves out some steps that make it a little bit smarter than that, but that's the basic structure.)
So, the idea when you're setting up your LOD parameters is to get a good number of vertices on the highest detail chunks without maxing out (ie, exceeding the 65k limit) on the lower detail chunks.
The simplest way to ensure that is to set a really high error metric. What's high and low? Well, if you're multiplying all the incoming points by 1.0/256.0, you have a working range of heights from -128 to positive 128 meters (we multiply the min/max values for heights, -32767 and +32767, by the scale factor). Depending on how jagged your terrain is, you might need quite a high error metric to avoid maxing out. You might start with an error metric of 50 or 100 in this case, note that you get a very undetailed terrain, and start lowering it. Most of the time the error metric is going to be around 1% of your working range, but it depends on what sort of terrain you're feeding it.
// Example of calling generateChunkFileFromRaw16.
generateChunkFileFromRaw16(
"demo/data/terrains/test_16_4097.raw",
4096, 2.0, 1.0 / 256.0,
"demo/data/terrains/large.chu", 2.0, 6);
generateChunkFileFromRaw16(srcFile, size, squareSize, vertScale, destFile, error, treeDepth)
srcFile
Filename of the source raw 16 bit heightfield we wish to generate data for. Example: demo/data/terrains/test_16_4097.raw.
size
Size of the heightfield, which is assumed to be square. The heightfield will be expected to be ONE MORE THAN THIS. Integer. Size must be a power of 2. Example: 4096. (for a 4097x4097 Raw file)
squareSize
Spacing between sample points, in meters. The default in legacy was 8 meters. This is a floating point number. There are no limitations on this value, although high polygon density may result in reduced frame rates. Example: 2.0.
vertScale
Vertical scale factor. Incoming raw data comes in as a whole number between -32,765 and +32,765, and is scaled by this factor to convert to meters. Factors are commonly of the form 1.0 / 2.0n, n often being between 1 and 8. 1.0 / 256.0 gives you a workable range of about 256 meters in the vertical range.
destFile
File to which generated data is written. This file should end in .chu for TSE's resource manager to properly load it. Example: "demo/data/terrains/large.chu".
error
Floating point number indicating the acceptable amount of error for each LOD to possess compared to the previous. This directly and dramatically impacts the number of polygons that will be rendered. Example: 2.0.
treeDepth
Integer indicating how deep the quadtree of chunks is to be. Example: 6.
Generating a texture quad tree (.tqt) file
Similarly, textures are generated from a single large source texture. Consumer hardware cannot support textures much larger than 4096x4096, so it is necessary to slice the source texture into smaller pieces so that they may be loaded piecemeal to the card. In addition, detail levels are precalculated, and the textures are compressed (currently with DXT).
Bear in mind that 2treeDepth-1 * leafSize = imageSize.
This preprocessing step is accomplished by the console function generateTQT.
// Example of generating a TQT.
generateTQT("demo/data/terrains/tseSurf.jpg", "demo/data/terrains/large.tqt", 5, 512);
generateTQT(srcFile, destFile, treeDepth, leafSize)
srcFile
Filename of the source image to be processed. It must be a power of two in size. Currently only JPEGs are supported. Example: "demo/data/terrains/terrSurf.jpg"
destFile
File to which generated data is written. This file should end in .tqt for TSE's resource manager to properly load it. Example: "demo/data/terrains/large.tqt".
treeDepth
Integer indicating how deep the quadtree of textures is to be. The TQT can be deeper or shallower than the geometry. If the former, some levels of texture detail will not be shown, and if the latter, geometry will be paged in but not have unique texture data - instead, the nearest available texture data in the tree will be used. Example: 5.
leafSize
Size of texture chunks to be emitted. The size of the lowest level of the quadtree (ie, 2treeDepth) times leafSize should equal the size of the source texture. Example: 512.
Limits in the Atlas Engine
There are some limits in the current implementation of the Atlas terrain engine. These are summarized in this section, along with any plans or techniques to address them.
In general, this version of the Atlas terrain ships with some very basic editors. Future versions will feature some powerful in-game editing tools similar to what is supported by the legacy engine in TGE, although tailored to fit the unique capabilities of the Atlas technology.
Texture Limitations
Supports only JPEG import/Can only load 65,535x65,535 images
Currently, the TQT generation tool only supports loading a single JPEG image for processing. Due to limitations in the JPEG file format, this means the maximum imported texture size is 65,535 by 65,535 pixels. There is no such limitation in the runtime or the .TQT file format.
In an upcoming milestone, we plan on adding support for other file formats, as well as support for loading multiple source images for TQT processing. This will allow virtually unlimited texture sizes (bounded only by the storage resources available).
Supports only DXT export
Currently, the TQT format only supports DXT texture data. The average JPEG compression of a TQT texture chunk is approximately ten times smaller than a DXT version. However, it requires more CPU to load, as well as potentially imposing a further overhead if we wish to store it in a compressed format on the card.
In a future release, we plan on adding support for various storage methods in the TQT format, primarily for JPEG or DXT, although other formats (PNG) may be supported as well.
Supports only precalculated texture data Some games do not need unique texture data for every point on their terrain. Especially in the casual market, adding several dozen megabytes of texture data to a download is not feasible.
In a future release, we plan to add support for a texture blender similar to that which is used with the legacy terrain.
Geometry Limitations
Dynamic lighting
Currently, no dynamic lighting is supported on the terrain.
When we enable TSE for dynamic lighting, we will also add the relevant support in the terrain engine.
Occlusion Culling/PVS
Some game situations can benefit significantly from culling of non visible portions of the terrain.
In a future release, we plan to add support for precalculated visibility information.
What are the dependencies of an Atlas terrain?
The only dependency of an Atlas terrain is a custom shader. The shader is responsible for doing geomorphing, as well as texture co-ordinate generation. The shader's HLSL source is contained in example/shaders/atlasSurfaceP.hlsl and example/shaders/atlasSurfaceV.hlsl. The ShaderData is defined in example/water_terrain_demo/server/scripts/shaders.cs and it is called AtlasShader. The CustomMaterial is in example/water_terrain_demo/server/scripts/materials.cs and it is called AtlasMaterial.
You may want to use a detail texture on your terrain. The default detail texture is example/terrain_water_demo/data/terrains/details/detail1.png.
To use Atlas terrain in a new mod, you'll need to make sure these assets are available. There are parameters to control what exact texture/material an AtlasInstance uses; check the script reference for more information on these.
Creating Performant Atlas Terrains
The biggest factor to making performant, good looking terrains is to tune your data for the expected usage. The algorithm will deal well with almost any data set, but if you know how the landscape will be viewed, you can make some optimizations.
First, find the highest error metric that looks good. Pay attention to the polygon usage report from the chunk generator. You might need to up your error metric, or make the chunk quadtree deeper.
For texture quad trees, you need to find balance between depth of the tree and size of the textures. Larger textures will load more quickly, but may cause hitching when they upload. Smaller textures will load more smoothly but potentially take longer to detail in. Similarly, shallow trees load faster and take less memory. (The parent levels of a texture remain resident in memory while the child textures are loaded.) But deeper trees allow you to have more detail without having to increase your texture size. Node texture sizes over about 1024x1024 may run less well on older cards.
If you know that you're going to be up close to the terrain and moving relatively slowly (ie, an FPS), you can have a deeper, more detailed geometry data set. On the other hand, this is a waste for a flightsim - you can spend more resources on high texture detail, and cut back significantly on geometric detail.
Also be aware that the terrain collides exactly against whatever is rendered for the lowest level detail. Having too many polygons in a single chunk at the lowest level may cause some performance issues.
Atlas Engine Runtime Reference
This section discusses three major aspects of the Atlas engine runtime: the script interface, the runtime parameters, and the C++ code layout. Together, knowledge of these sections will result in familiarity with how the runtime operates.
C++ Reference
The Atlas engine resides in the Atlas sub directory. The header files are annotated with Doxygen-compatible comments, which fully explain the functioning of the various classes and components of the terrain. The terrain itself is self-contained, and should not require any modifications to existing objects to work properly.
If you should require a C++ reference, simply run doxygen on the TSE codebase and refer to the appropriate sections. Or, you can read the comments, which are what the Doxygen documentation is generated from.
Script Reference
There is only one class exposed to the script system, AtlasInstance. Beyond the position and rotation fields, these are the other important fields:
AtlasInstance Fields
tqtFile
Path to the .TQT file for this instance.
chunkFile
Path to the .CHU file for this instance.
detailTexture
Detail texture for the terrain.
materialName
Name of the CustomMaterial used to render this instance. "AtlasMaterial" by default, and likely best to leave it that way. Don't change this if you don't know what you're doing. Will not work except with custom shaders!
There is one method of note, which is purge(). This method causes all paged-in resources for the geometry and texture files to be unloaded, forcing the system to page everything in from scratch.
Runtime Parameters Reference
Runtime parameters can be access from the console to control various aspects of the runtime behavior of the Atlas engine. They have global, immediate effect, and are not networked.
$AtlasInstance::showChunkBounds
Render bounding boxes showing the extents of each currently loaded geometry chunk in the hierarchy. Color of the box ranges from yellow to red to black, indicating the relative amount of paging activity occurring on that node. Boolean, default is false.
$AtlasInstance::renderWireframe
Draw all Atlas geometry in wireframe mode. Boolean, default is false.
$AtlasInstance::synchronousLoad
If true, performs all loading each frame in the main thread. If false, performs all loading in a background thread. Boolean, default is false.
$AtlasInstance::lockCamera
Debug parameter. Locks the camera position for testing LOD and culling code. Boolean, default is false.
$AtlasInstance::collideChunkBoxes
Debug parameter. For raycasts, treats chunks as big boxes. Useful for debugging raycast code. Boolean, default is false.
$AtlasInstance::collideGeomBoxes
Debug parameter. For raycasts, treats the nodes of the collision quadtree in a chunk as big boxes. Useful for debugging raycast code. Boolean, default is false.
In engine/atlas/atlasTQTFile.cpp, there is a variable AtlasTQTFile::smDumpGeneratedTextures, which controls a debug feature of the TQT generation code. If set to true, it will cause every subtexture generated by the TQT code to be dumped to the current working directory of the game. This is useful for diagnosing problems with the generation code.
Putting Things In Game
Ok, you probably want to put these assets in game. Well, that's the simplest part of the whole process...
First, get your favorite text editor and open the mission (.mis) file.
Next, inside the MissionGroup SimGroup, add the following block:
new AtlasInstance(BigChunk) {
position = "0 0 0";
rotation = "1 0 0 0";
scale = "1 1 1";
chunkFile = "~/data/terrains/large.chu"; // Don't forget to tweak these if
tqtFile = "~/data/terrains/large.tqt"; // you named your files differently.
};
Note
Currently, Atlas does not deal well with changing the chunk or TQT file settings in-engine. The best way to get things to work is to exit the game, edit the .mis, and reenter. A future milestone will resolve this issue.
Save the mission file, and you're all ready to go. Launch the mission and enjoy your new terrain!
Getting Higher Resolution Terrain Data
You may have noticed that the default data in TSE CVS is fairly coarse, and has a big GG logo on it, as well as giant yellow text reading, "TEST TERRAIN - Manual Says Where To Get A Real Terrain."
This is because a densely texture Atlas data set can easily hit fifty or a hundred megabytes. We don't want to bloat the CVS repository with that much extra data, so we've put in a tiny test data set. It's fully functional, just a bit ugly.
Categories: Needs Updating | Atlas | TSE
