Second generation GPU client on ATI hardware (GPU2) FAQ

Table of Contents

A Brief History of FAH: From Tinker to Gromacs to GPU to GPU2

Introduction

Since 2000, Folding@home (FAH) has lead to a major jump in the capabilities of molecular simulation. By joining together hundreds of thousands of PCs throughout the world, calculations, which were previously considered impossible, have now become routine. FAH has targeted the study of protein folding and protein folding diseases, and numerous scientific advances have come from the project.

In 2006, we began looking forward to another major advance in capabilities. This advance utilizes the new, high performance Graphics Processing Units (GPUs) from ATI to achieve performance previously only possible on supercomputers. With this new technology, as well as the new Cell processor in Sony's PlayStation 3, we will soon be able to attain performance on the 100 gigaflop scale per computer. With this new software and hardware, we will be able to push Folding@home a major step forward.

Now in 2008, we have developed a second generation GPU core (GPU2). This core is much more sophisticated than the original, with higher reliability, ease of use, and much more scientific calculation capabilities.

Our goal is to apply this new technology to dramatically advance the capabilities of Folding@home, applying our simulations to further study of protein folding and related diseases, including Alzheimer's Disease, Huntington's Disease, and certain forms of cancer. With these computational advances, coupled with new simulation methodologies to harness the new techniques, we will be able to address questions previously considered impossible to tackle computationally, and make even greater impacts on our knowledge of folding and folding related diseases.

Folding@home debuts with the Tinker core (October 2000)

In October 2000, Folding@home was officially released. The main software core engine was the Tinker molecular dynamics (MD) code. Tinker was chosen as the first scientific core due to its versatility and well laid out software design. In particular, Tinker was the only code to support a wide variety of MD force fields and solvent models. With the Tinker core, we were able to make several advances, including the first folding of a small protein starting purely from sequence (subsequently published in Nature).

A major step forward: the Gromacs core (May 2003)

After many months of testing, Folding@home officially rolled out a new core based on the Gromacs MD code in May 2003. Gromacs is the fastest MD code available, and likely one of the most optimized scientific codes in the world. By using hand tuned assembly code and utilizing new hardware in many PCs and Intel-based Macs (the SSE instructions), Gromacs was considerably faster than most MD codes by a factor of about 10x, and approximately a 20x to 30x speed increase over Tinker (which was written for flexibility and functionality, but not for speed).

However, while Gromacs is faster than Tinker, it has limits to what it can do; for example, it does not support many implicit solvent models, which play a key role in our folding simulations with Tinker. Thus, while Gromacs significantly sped certain calculations, it was not a replacement for Tinker, and so the Tinker core will continue to play an important role in Folding@home (including a recent paper in Science). For these reasons, points for Gromacs WUs were set to be consistent with points for Tinker WUs, as both play an important role in the science of FAH. Moreover, we switched the benchmark machine to a 2.8 GHz Pentium 4 (from a 500MHz Celeron) in order to allow us to fairly benchmark these types of WUs (as the benchmark machine needed to have hardware support for SSE).

The next major step forward: Streaming Processor cores (September 2006)

Much like the Gromacs core greatly enhanced Folding@home by a 20x to 30x speed increase via a new utilization of hardware (SSE) in PCs, in 2006, Folding@home has developed a new streaming processor core to utilize another new generation of hardware: GPUs with programmable floating-point capability. By writing highly optimized, hand tuned code to run on ATI X1900 class GPUs, the science of Folding@home will see another 20x to 30x speed increase over its previous software (Gromacs) for certain applications. This great speed increase is achieved by running essentially the complete molecular dynamics calculation on the GPU; while this is a challenging software development task, it appears to be the way to achieve the highest speed improvement on GPU's.

In addition, through collaboration with Pande Group, Sony has developed an analogous core for the PS3's Cell processor (another streaming processor), which should see a significant speed increase for the science over the types of calculations we could previously do on a x86/SSE Gromacs core as well. Following what we did with the introduction of Gromacs, we will now switch benchmark machines and include an ATI X1900XT GPU in order to be able to benchmark streaming WUs (which cannot be run on non-GPU machines). This machine will also benchmark CPU units (which continue to be of value since GPUs work only for certain simulations) without using its GPU.

The second generation GPU core, aka GPU2 (April 2008)

After running the original GPU core for quite some time and analyzing its results, we have learned a lot about running GPGPU software. For example, it has become clear that a GPGPU approach via DirectX (DX) is not sufficiently reliable for what we need to do. Also, we've learned a great deal about GPU algorithms and improvements. One of the really exciting aspects about GPU's is that not only can they accelerate existing algorithms significantly, they get really interesting in that they can open doors to new algorithms that we would never think to do on CPUs at all (due to their very slow speed on CPUs, not but GPU's).

After much effort, we have taken all we've learned about GPUs from the first generation client and produced a second generation client. This new client appears to be faster, more reliable, and has more scientific functionality. The preliminary results so far from it look very exciting, and we're excited to now open up the client for FAH donors to run.

General instructions

This web page will serve as the FAQ and Release Notes for this new client, and we will update this page as more information becomes available.

The FAH GPU Client installer should do everything one needs. It installs the new v6.x SysTray style client, as well as DLL files used by this new client. Download the client from the High Performance Client Download Page for folding experts.

Basic Requirements:

• 2xxx/3xxx ATI Video Card, or newer
• ATI Driver v8.1+, v8.3 or newer recommended
• AGP GPU aperture size in the BIOS must be set to 128 MB or larger
• Microsoft .NET Framework 2.0, with updates recommended
• Windows operating system, XP or newer
• Processor with SSE2 support to feed the GPU

This is a beta release and we expect there will be bugs, flaws, problems, etc. To minimize problems, we have been testing the client and cores extensively in house and they run well there. However, it's our experience that running in the controlled setup in our lab and running "out in the wild" are very different situations.

As in the use of any beta software, please make sure to back up your hard drive, and do not run this client on any machine which cannot tolerate even the slightest instability or problems.

Frequently Asked Questions

What about visualization?

We are working to add visualization and the visualization code right now is out in limited testing. The picture on the right shows what the visualization looks like. Click here to see a movie (warning: 10MB download). Click on the figure to see a larger version.

Like the Folding@home PS3 client visualization, the GPU2 client shows a real time view of the protein during the simulation. Since the GPU2 client is quite fast, the protein does move around a fair bit (about 1000x more compared to say the classic client in some cases). Thanks to ATI for their help with this visualization, especially for the look, and to Adam Beberg for the main engine behind it.

What hardware does the new client/core support?

The client runs on all hardware supported by the AMD Compute Abstraction Layer (CAL), i.e. R6xx hardware, Radeon 2400 and above. The 3870 X2 has not been tested with both cores active, but running on 1 core does currently work.

What OSs does the new client/core support?

The client runs on Windows XP 32-bit/64-bit and Vista 32-bit/64-bit. Windows XP should have SP2 installed. XP and Vista use different CAL DLLs (see below), but the installer should install the correct ones for you.

What about hardware clocks?

On 3xxx hardware, 3D clocks will be set automatically when FAH runs and you can adjust the clocks rates for core and memory in Catalyst Control Center in the Overdrive panel. Note that stable clocks for graphics may not imply stable clocks for Folding, overclock at your own risk. On 2xxx hardware, setting to 3D clocks is not reliably automatic, so a third party tool like ATI Tray Tool can be used to adjust clocks. Once again user beware. The recommendation is to leave the settings alone and fold at the clocks set by the driver.

Can I use my CPU to do calculations too?

Dedicating 100% of a CPU core to a GPU client is strongly recommended. You will need one CPU core per GPU core to keep the GPU client adequately fed with data.

Folding performance on high-end GPUs is directly related to the performance of the CPU feeding the new client. Increasing the GPU core speed may not improve folding performance without sufficient CPU speed to support those higher GPU speeds. The reverse scenario may also be true. A very fast CPU helps to get the best GPU performance with this new client.

How about AGP vs PCIe slots?

Performance of the GPU client is best with the board in a PCIe x16 slot. An x8 or x4 slot will cause some degradation in performance as the communication path between the CPU and GPU will be slower. AGP hardware is supported, but overall performance will be lower than PCIe boards because of a slower CPU to GPU connection. PCIe v2.0 is only marginally faster than PCIe.

What about multi-gpu support and the -gpu switch?

Running multiple GPU2 clients, one client each on multiple GPU cards, is supported through the -gpu x command line switch. The setup is similar to running multiple SysTray CPU clients.

• Copy your \Application Data\Folding@home-gpu folder to a new folder \Folding@home-gpu2 (\AppData\Roaming\Folding@home-gpu in Vista)
• Create a new shortcut for the first client, and be sure to use the correct Target: and Start In: information.
• Edit the shortcut properties to add the -gpu 0 switch to the end of the Target: field.
• Create a new shortcut for the second client, and be sure to use the correct Target: and Start In: information.
• Edit the shortcut properties to add the -gpu 1 switch to the end of the Target: field.

Except for the different -gpu x switch, the Target: field in both shortcuts will point to the same FAH executable. The Start In: field for each client will point to the two different \Apps Data\FAH folders. The Target: and Start In: fields for a SysTray client are explained in more detail below.

The display must be active on the GPU card you plan to use, and –gpu 0 will select the first board, –gpu 1 will select the second board, -gpu 2 the third board, and so-on. You will need to disable crossfire for multiple boards to be detected. You will also need to use different Machine IDs for each client. Currently, only one client is supported on a 3850X2 or 3870X2.

Does the new GPU client run the same WUs?

No, this new second generation GPU client will run a different set of WUs specially constructed for the Fahcore_11.exe functionality. Fahcore_11 will not run with the first generation GPU client, and Fahcore_10 will not run with this new client.

How do you decide the credit value of GPU work units?

Points are determined by the performance of a given machine relative to a benchmark machine, similar to the CPU client benchmark process. Before releasing any new project (series of work units), we benchmark it on a dedicated computer with an ATI Radeon 3850 GPU (512 MB, 320 Stream Processors), running in a Dell Inspiron 531, with a 2.16 GHz dual core AMD 64 X2 4000+.

We plug the results of this benchmark test into the following formula:

Points = 1000 * (DaysPerWU)

where DaysPerWU is the number of days it took the benchmark hardware to complete the work unit. Note that the GPU client still relies on a fast CPU, so the CPU is an important part of this. The Points Per Day (PPD) given here assumes that a CPU is heavily needed, with a larger PPD to compensate for the use of that CPU.

Please note the very concept of a reference machine will mean that some WU performance will vary from the performance on your machine. Even between various GPU models, there are significant differences in architectures and memory speeds. Moreover, there are variations between WUs within a given project which can lead to speed differences.

Our goal is consistency within a given definition of a reference machine setup (described above), but beyond that, the natural variation from machine to machine and WU to WU will never allow any point system to perfectly predict what you get on your machine.

Why is the new GPU client important?

The purpose of the GPU client is twofold: to take advantage of the high-performance capabilities of Stream Processing, and to help develop a simulation architecture that will become one of the dominant FAH computing paradigms as multi-processor GPUs become an industry standard over the next several years. High-performance clients enable us to run types of calculations that would be impractical on our standard architecture--calculations that enhance our scientific capabilities, and your scientific contributions, significantly.

High-performance clients often require more computing resources. GPU clients typically run on dedicated systems, 24 hours a day, and use more processing power, more disk space, more network resources, more system memory, etc. Also, a major part of the scientific benefit is dependent on rapid turnaround of work units; hence we assign short deadlines for GPU work units. To reward those contributors for donating resources beyond the typical CPU client, for completing these work units very quickly within the short deadlines, and for contributing to the development of our next-generation capabilities, we currently set a benchmark value proportional to these demanding GPU work units. Without the GPU clients and your additional contributions, we would not be able to complete many important projects.

What's different between the old and the new FAH GPU client?

Scientifically, the new client introduces several new advances which makes it much more useful. It matches the advanced water models in the PS3 client and adds a new one (which will likely appear in a future PS3 client). These more advanced water models make this new GPU client very useful to us.

There are also many changes under the hood. The previous generation client proved to be problematic due to GPU-specific issues and we've fixed all of them (as far as we can tell) in this second generation client. An important part of these fixes is using ATI's CAL instead of DirectX (the previous generation GPU client highlighted several issues with using DirectX). A major upside to using CAL is that DirectX context switches no longer affect the client. Actions such as fast-user switching, or locking your computer have no effect on GPU processing. Remote desktop does still affect the GPU client and will cause the FahCore to fail when a connection is initiated. VNC does not have the same problem and can be used as an alternative.

Initially, this new client will be a SysTray style client only. A console version may follow later.

Can I still use my GPU when the client is running?

Yes. Unlike the original GPU client, which interfered with many operations that used the GPU, the new CAL-based client does not. Playing videos and playing games either have no effect on the action of the GPU client other than a slow-down in processing, or cause a temporary suspension of folding. The new client will automatically back off whenever an application requests exclusive DirectX mode, although it is not reported in the client logfile. DirectX programs that do not request exclusive mode will cause the GPU client to slow down, and may in some instances have a detrimental effect on application performance. Full screen video is unaffected by the GPU client.

Troubleshooting

This was working, but now all I'm getting was Early Unit Ends (EUE's). How can I fix this?

We've seen cases where playing GPU intensive games can leave the GPU in a weird state, leading to consistent EUE's (Early Unit End error messages). Restarting the computer has worked to resolve this problem. We are looking into a better solution.

The client displays an error saying that I do not have a supported GPU, but I do!!!

When the client displays the error At present, only ATI Radeon HD 2xxx/3xxx and ATI FireGL Vx6xx GPUs are supported the client did not recognize your GPU card. If you have a supported 2xxx/3xxx model GPU, this error is most likely caused by custom device drivers that do not work with the GPU2 client. Some OEM and some laptop vendors modify the Catalyst drivers that ship with their cards. Download and install the latest ATI Catalyst drivers.

At present, the primary video device must also be a supported 2xxx/3xxx model, or you may see this error as well. The error CoreStatus = FFFFFFFF (-1) is also a symptom. Changing the ATI card to be the primary display device will resolve this.

Hey, where did all of the GPU client data files go?

The new GPU client is a SysTray client, similar to the new v6.x CPU SysTray clients, and follows a standardized Windows installation procedure. This new client type is similar in nature to the previous GUI style client, but with notable changes, and a separate visualizations module (to come later).

The client executable can be installed to any directory you select. The default location is
C:\Program Files\Folding@home\Folding@home-gpu\.
In Windows XP, the remaining data files are installed to
C:\Documents and Settings\<user name>\Application Data\Folding@home-gpu\.
In Windows Vista, the remaining data files are installed to
C:\Users\<user name>\AppData\Roaming\Folding@home-gpu\.

Note: The client installer creates a shortcut to the data files in the Folding@home Programs folder. The installer also creates a shortcut in the Startup folder that launches the GPU client. The Startup shortcut points to the specific file locations in the Target: and Start In: fields. That shortcut cannot be edited to add client switches. To add a client switch, create a new shortcut with the correct Target: and Start In: fields, and then add the switch to the end of the Target: field. Replace the shortcut in the Startup folder if desired. Please take care if making a new shortcut to start the GPU client. Failing to match the client file locations might cause work to be lost. Also, there is no need to use the -local flag, even for multiple clients. The local directory is specified by the Start In: field.

The core can't find the DLL's!

We've been seeing some unusual behavior with virus scanners. We are looking into this. For now, give it a second try and it should work.

A DLL error dialog box is popping up -- what's up with that?

If the DLL error pops up, go to the installed location, C:\Program Files\Folding@home\Folding@home-gpu by default, and make sure the amdcalcl.dll and amdcalrt.dll ended up there with the FahCore_11.exe file and run the client from that location.

What's with all the new DLL's anyway?

We're using a new system (CAL) which uses a few DLL's. We are looking into the possibility of statically linking the whole thing to avoid DLL issues, but for now we've got DLL's.

Who did all of this anyway?

In alphabetical order:

• Adam Beberg (Pande Lab): client modifications, GPU's APIs under the hood
• Dan Ensign (Pande Lab): server setup, science, testing
• Mark Friedrichs (Pande Lab, Simbios): core science code updates, testing
• Mike Houston (AMD): testing, problem solving, GPU tuning
• Vijay Pande (Pande Lab): Project management, fitting square pegs through round holes, etc
• We would also like to thank the Folding@home Community Forum moderators for their help with this FAQ and some early beta testing of the software.

For More Information

• ATI Folding Page
• ATI Stream Computing FAQ
• FAH FAQ
• Folding Support Forum

Last Updated on May 06, 2008, at 07:29 AM