Instructions for running MPSA

Step 1: Copy supporting files from MPSA directory

Step 2: Prepare raw particle images

Step 3: Generate random initial model and make initial 2D references

Step 4: 2D orientation search

The program accepts both raw images and references with different image size and pixel size, and can set them to the right scale during alignments.

Step 5: 3D reconstruction

The program accepts input particles with different image size and pixel size.

Output 3D map can have user specified volume and pixel sizes.

Step 6: Post data processing and make new 2D references

Iterate from Step 4 to 6 until the 3D model is converged

Step 1: Copy supporting files to your current directory

Run command: mp-copyfiles2local for copying the some change-required files from MPSA directory to your working directory

The files include mp-configure.ini, mp-2dalignment, mp-3dreconstruction and mpsa*pbs

These files can also be manually copied from previous working directory for the purpose of keeping your previous setting

Step 2: Data preparation

Copy all raw particle image files (prefer Imagic formation) into a subfolder (i.g. "data").

Set the ctf parameters into the header of the particle images

mp-setCTF.py "data/*.hed" (needs eman1 ctfparm.txt file)

If eman2's ctf is in header, manually fit an eman1 ctf parameters as a default in ctfparm.txt file; then apply mp-setSameCTF.py "data/*.hed"

Generate a list file with random orientation for all raw particle images, which can also be used for generating random initial 3D model in Step3.

mp-particle2list.py "data/*.img" all-particles.lst

Step 3: Generate random initial model and make initial 2D references

Submit PBS job: mpsa-ini-model.pbs

The pbs script needs to be modified according to different queue system.

The input is the particle list file with random orientations generated in step 2 (i.g. all-particles.lst)

The finished job will output the random 3D initial model: ini-model.mrc

Make initial 2D references from the initial model

project3d ini-model.mrc grid=# randomall out=ini-ref.img

The square of grid# decides the number of 2D projections to be generated; generally set grid to 6 as default.

Step 4: 2D orientation search

Set parameters for 2D alignment by editing mp-config.ini file

Partial of the parameters of the configure file are list as the follows.

[iter1-options] ;section name must include the interationName

resultBaseDir = result ;final result will be

sym=icos ; set particle symmetry: icos, oct, tet, c#, d# [icos]

rawimage = all-particle.lst, listfile

refimage = ini-ref.img ; 2D ref images

numOfRefImages= 7 ; # of reference images

apixRaw = 2.6 ; this option is only for experimental purposes

apixRef = 1.5 ; this option is only for experimental purposes

......More details are described in the last part of the instruction

The raw images can have different pixel size and image size with reference images

Run command: mp-2dalignment to search the orientations and centers of the raw particle images. The shell script, mp-2dalignment will perform two major functions (for non bash shell, the script may need to be modified).

Generate a command file (iter*.cmd) containing a set of command for each particle image

mp-gen2daligncommand.py iter1

Distribute each of the commands to the allocated nodes using client-server model for parallel computing.

mpirun -machinefile machine_nodes2d -np ## python-mpi mp=runcommands.py iter1-coarse.cmd

mpirun, python-mpi and mp-runcommands.py need be specified in full path

The job distribution command, mp-runcommands.py, is implemented through mpi and python-mpi. As a result of this requirement, the provided dependencies: mpich and python-mpi are required.

The executed commands will be saved to iter*.save, and the result will be saved under directory: result/iter*. The script, iter*.mrg can merge all individual result files of the particles of a CCD frame/micrograph into single files.

Run command: mp-checkprogress result/iter1 for checking the progress and the results of orientations and centers of the raw particle images.

Step 5: 3D reconstruction

Generate a list file, iter1-coarse.lst, from the result of 2D alignment

The first part of the shell script, mp-3dreconstruction (if non-bash shell, it may need to be modified accordingly), extracts all particles' centers and orientations from the result of current iteration and save them as a list, iter1-coarse.lst

Run 3D reconstruction command to build 3D density map from the list file, iter1-coarse.lst.

Edit the second part of shell script, mp-3dreconstruction, to set the reconstruction parameters properly.

mpirun -machinefile machine_nodes3d -np ## mp-3dbuilder iter1-coarse.lst out=iter1-coarse.mrc,2.6,700 apix=3.1 sym=icos mask=700A mpipkgsize=3000000 usecenter flipphase wiener3d=scattering.sf > log-3dbuilder.log

mpirun and mp-3dbuilder need be specified in full path

If no CTF parameter was set in image's headers, apix needs be specified; otherwise, the pixel size will be set to 1Å/pixel. If CTF has been set in headers, the program will ignore the apix and read the values from the headers.

Particle images can have different defocus and apix values, which can be specified in the input file, iter1-coarse.lst.

Format: particle# filename alt az phi centerX centerY defocus=xxx apix=yyy

Once defocus and apix are given in the input list file, the corresponding defocus and apix of the CTF parameters in a header will take no effect, but the CTF will be automatically adjusted to fit the new apix.

Output map's volume and pixel sizes can be specified by users.

out=iter1-coarse.mrc,2.6,700. Here, the pixel and volume sizes of the output map are specified as 2.6Å and 700 pixels.

mask=700A indicates the input particle images will be applied a soft circle mask with radius 700Å. The unit can also be specified as 'P' or 'p' for pixel. If neither P or A is specified, the default unit will be pixel.

The reconstruction command, mp-3dbuilder, is implemented with mpi. The provided dependencies: mpich is required.

The 2D alignment and 3D reconstruction commands have been integrated in a pbs script file, mpsa.pbs. It may need to be modified according to different queue system.

The finished job will output the reconstructed 3D map: iter1-coarse.mrc, which will be the reference for the next iteration after post processing in Step 6.

Step 6: Post data processing and make new 2D references

Filter and sharpening, and mask can be applied for post processing the reconstructed 3D map . Sample operations are listed as the follows.

Lowpass filter and set structural factor: proc3d iter1-coarse.mrc iter1-coarse-setsflp30.mrc apix=2.6 setsf=scattering.sf lp=30

Mask: proc3d iter1-coarse-setsflp30.mrc iter1-coarse-setsflp30-mask.mrc apix=2.6 automask2=300,1.3,20 imask=230,20

......

Generate new 2D reference images for next iteration

project3d iter1-coarse-setsflp30-mask.mrc grid=# randomall out=iter1-ref.img

The next iteration

Edit the configure file, mp-config.ini, for 2^nd iteration

[iter2-options];

refimage = iter1-ref.img ; update the reference to iter1-ref.img

radialStartPoint=3 ; can be increased as the map's resolution improving

radialEndPoint=20 ; can be increased as the map's resolution improving

(Only updated parameters are listed here)

Run mp-2dalignment for orientation search of iter2

Run mp-3dreconstruction for iter2

Post data processing and generate 2D references for the 3^rd iteration

Ready for the 3^rd iteration, ......

The reconstructed 3D density map should be converged in several iterations for viruses/protein complexes with coarse surface (feature rich) . For smooth surface, it may take more iterations.

Detailed description in setting up the configure file, mp-config.ini

[iter1-options] ;section name must include the interationName

resultBaseDir = result ;final result will be [resultBaseDir/iterationName/]

baseFileName = coarse ;3 file names will be generated .cmd .mrg .save

sym=icos ; set particle symmetry: icos, oct, tet, c#, d# [icos]

rawimage = all-particle.lst, listfile

refimage = ini-ref.img ; 2D ref images

numOfRefImages= 7 ; # of reference images (3 to 12 for icos symmetry)

;apixRaw = 2.59 ; this option is only for experimental purposes

;apixRef = 2.55 ; this option is only for experimental purposes

;overSample=2 ; this option is only for experimental purposes

rMask=120 ; mask radius for raw image in pixel [half image size]; mask center is the particle's physical center. It should be larger than the radii of the particle and smaller than half box size.

phasecorrected= ;if True, the particles phase has been flipped, [False]

scatteringFactorFile=scattering.sf

iniCenterOrientationMode=random ;[random], iniparmfile, headerfile

iniParameterFile= ;if iniparmfile is choosed as an option above, specify the file name here

; The following 4 options are used for the internal control of mpsa optimization algorithm

numOfStartConfigurations=7 ; number of paths/configurations

maxIteration= 2500 ; # of total iterations for this search

numOfRandomJump= 800 ; # of iterations for random jump

numOfFastShrink= 700 ; # of iterations for converging all paths/configurations to the best local minimal state/configuration

orientationSearchRange=35 ; in degree

centerSearchRange=20 ; in pixel

radialStepSize=1.0 ; Sampling space for extracting common line points (in pixel)

radialStartPoint=3 ; search range (in pixel) from lower-end resolution (apix*imagesize/radialStartPoint)

radialEndPoint= 20 ; search range (in pixel) to higher-end resolution (apix*imagesize/radialEndPoint). EndPoint should be gradually increased for later iterations when model converges and resolution improves

searchMode=0 ; 0-->coarse/global search, 1-->refinement, other modes only for experimental purposes

residualMode=2 ; (1-cos(diffPhase))/2, other modes only for experimental purposes

maxNumOfRun=7 ; # of repeating search for the purpose of consistency check

solutionCenterDiffCriterion=2.0 ; consistency criterion for center (in pixel)

solutionOrientationDiffCriterion=2.0 ; consistency criterion for orientation (in degree)

verbose=2 ; screen output level

updateHeader=0 ; update header or not. No need to update header file for using MPSA