2 million particles CG simulation

I have finally created stable box 128 nm x 128nm x 18 nm with almost 2 000 000 CG particles. There is 50 176 DPPC (602 112 CG particles) molecules and 1 388 471 W CG particles.

Here you can see video of DPPC bilayer forming:




Few frames in high resolution:

More on GROMACS 4.0 scaling

Last couple of days I have run more performance and scalability tests for Gromacs 4.0 with CG simulations. I have tested four systems with different sizes:

System 1:
Total of 5 760 CG particles inside 10nm x 10nm x 10 nm simulation box - 1 Protein (555 CG) , 229 DPPC molecules (2748 CG), 9712 water molecules (2428 CG) + 29 Cl ions

System 2:
Total of 19532 CG particles inside 20nm x 20nm x 10 nm simulation box - 1 Protein (555 CG) , 659 DPPC molecules (7908 CG), 44172 water molecules (11043 CG) + 29 Cl ions

System 3:
Total of 78953 CG particles inside 40nm x 40nm x 10 nm simulation box - 1 Protein (555 CG) , 2748 DPPC molecules (32976 CG), 181 572 water molecules (45393 CG) + 29 Cl ions

System 4:
Total of 342 708 CG particles inside 80nm x 80nm x 10 nm simulation box - 1 Protein (555 CG) , 5304 DPPC molecules (63684 CG), 1 113 904 water molecules (278476 CG) + 29 Cl ions

I have run 3 iterations for each systems for 1, 2, 4, 6 and 8 cores with total of 60 short different CG simulations.

Blue line represents real performance in ns per day and red line is optimal linear scaling based on one processor core test.

System 1 results:

System 2 results:

System 3 results:

System 4 results:
Conclusions:

1. I can run CG simulations with 8 cores without almost any degradation of performance with Gromacs 4.0 (this was not the case with Gromacs 3.3)

2. I can achieve hundred nanoseconds molecular dynamics CG simulations with hundred thousands of CG particles in simulation system. This will enable me to insert tens of proteins of interest to the simulation box, not only one.

Other observations:

1. How you can observe, blue line representing real performance is overperforming red line that should represents optimal linear scaling. This is because in all test cases, simulations with 2 cores overperform more than twice performance with one core. For some reason it is more efficient to run simulations on more cores than one. It is little bit odd, but I will try to find explanation to this.

2. Even with biggest system with 342 708 CG particles I have free 14.5GB of memory from total of 16GB during simulation. So performance bottleneck here is processing power not memory. Next time I will have to buy hardware for MD simulations I will better spend it on top processors than on memory. Expensive lesson learned :(

3. Gromacs 4.0 is scaling with system size lineary as well.

In last graph you can see comparing relative performance for all systems. I have multiplied number of CG particles with performance in ms/day. If there is any degradation in performance we would see values decreasing, in better case linearly in worse case exponentionally. We see values increasing which means that we are getting better relative performance with bigger systems.

And last this is how biggest system looks like after 20 ns simulation:

We can see here multiple bicelles forming. There is too high ratio of water molecules to lipid molecules to form bilayer. Next I will try to create 1 000 000 CG particles simulation that will form bilayer sheet.

Ubuntu 8.10 Desktop Edition

I have finally decided to installed Ubuntu 8.10 Desktop Edition 32-bit version on my workstation. Only one problem I had was with wireless USB adapter that was not recognized by Ubuntu. I had to use NDISwrapper that enables to use Windows drivers for wireless cards on Linux. I have installed additional required software packages that I need for my work. Workstation is now up and running without any problems.

Simple disk speed test to compare with server:

jan@workstation:~$ sudo hdparm -tT /dev/sde

/dev/sde:
Timing cached reads: 2436 MB in 2.00 seconds = 1218.00 MB/sec
Timing buffered disk reads: 460 MB in 3.01 seconds = 152.92 MB/sec

Result for cached results are comparable to server but buffered reads are at least two times worse than server ones as expected.

GROMACS 4.0 scaling for short coarse-grained simulation

During this weekend I had only time to install Gromacs 4.0 and again run short (20ns) coarse-grained MD simulations with very similar (but not exactly same) system as with Gromacs 3.3.3 for one, two, four and eight cores.

First results look more than promising:

It was only one iteration so I need to do more rigorous scalability testing during next week, but with my hardware it is almost linear scaling for this small system ...

GROMACS 3.3.3 scaling for short coarse-grained simulation

I have run short (20ns) coarse-grained MD simulations where system contained:

- 1 Phospholamban pentamer protein (pdb:1ZLL) 255 amino acid residues
- 229 molecules of DPPC phospholipids
- 2454 molecules of water
- 3 Cl- counter ions

with 1, 2, 4 and 8 cores in parallel on my server.


Red line represents linear scaling in perfect world based on performance of one core simulation. Blue line represent polynomial regression line from results for 2, 4 and 8 cores simulations.

My server has two quad-core processors and it looks like that in this case it does not make sense to use more than 4 cores in parallel for one simulation.

If you are interested about result of this short simulation, this is how it looks like in starting point random state in 0 ns and how it looks like after 20 ns simulation.
Water molecules and hydrocarbon lipid chains are not shown for clarity. Color chains represent five amino acid chains of phospholamban protein. Lipid headgroups are represented as colored balls. How you can see lipid bilayer is nicely formed in 20 ns timepoint.

Before I am going to do more scalability tests how to best utilize server hardware I decided to upgrade to Gromacs 4.0 that suppose to have two orders of magnitude better scaling than 3.3 version ...

Software instalation

I had to install different software packages and libraries to make server usable for my needs:

Just to mention few of them: EMACS, RCS, GCC, LAM/MPI, LIBAIO, FFTW, GROMACS, RUBY, RUBY ON RAILS ...

I was able to run first parallel coarse-grained MD simulation with Gromacs .... yupi!

Ubuntu 8.04 LTS Server Edition

Installation of operating system for my server, Ubuntu 8.04 Server edition 64-bit version took exactly 10 minutes (!). I am impressed, it looks like it had recognized all hardware correctly and it is running now just fine.

Lets just do some disk speed test.

jan@server:~$ sudo hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 2790 MB in 2.00 seconds = 1395.58 MB/sec
Timing buffered disk reads: 1068 MB in 3.00 seconds = 355.58 MB/sec

It looks quite good, I will compare later with workstation ...

Now lets install Gromacs, molecular dynamics software package ...