Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configuration stanza for the Intel oneAPI compilers with generic MPI wrappers #250

Merged
merged 4 commits into from
May 13, 2024

Conversation

mgduda
Copy link
Collaborator

@mgduda mgduda commented May 9, 2024

This PR adds a new configuration stanza for the Intel oneAPI compilers, with compiler wrapper names appropriate for most MPI implementations.

Also included in this PR is an update to the logic in the configure script to print compiler version information for the ifx compiler.

mgduda added 2 commits May 9, 2024 14:39
The logic in the compile script to print compiler version information now
recognizes the Intel oneAPI Fortran compiler, ifx.
The MPI wrappers in the new stanza are generic MPI wrappers, mpifort and mpicc,
rather than those for the Intel MPI library.
@mgduda mgduda requested a review from islas May 9, 2024 21:13
@mgduda
Copy link
Collaborator Author

mgduda commented May 9, 2024

@islas One additional change that could be included in this PR would rename the existing Intel stanzas from Intel compiler to Intel Classic compilers just to make the distinction between the old compilers and the newer oneAPI compilers more obvious. Any thoughts?

@islas
Copy link
Contributor

islas commented May 9, 2024

Agreed, especially as that is the nomenclature that Intel uses.

I saw the discussion regarding mpif90 vs mpifort - when only loading the Intel env via setvars.sh (their recommended way of loading) no mpifort exists. While using mpif90 would use Intel's MPI library, I would also expect a DM configuration to work. Not sure what the solution is, but I think it's worth noting.

…aults

Throughout the arch/configure.defaults file, this commit changes occurrences of
"Intel compiler" to "Intel Classic compilers" to distinguish the use of the
"Classic" fort/icc compilers from the use of ifx/icx in the "Intel oneAPI
compilers" stanza.
@mgduda
Copy link
Collaborator Author

mgduda commented May 9, 2024

@islas I've just pushed a commit to change "Intel compiler" to "Intel Classic compilers" throughout the arch/configure.defaults file.

I suppose we could switch to using mpif90. If I understand correctly, that would allow those who have the Intel MPI library to use the new stanza, while others (e.g., those of us on Derecho) could usually get by with other MPI libraries like MPICH and OpenMPI. I think my original intent with mpifort was that it was perhaps a more modern name for the MPI Fortran wrapper. But I suppose compatibility wins over modernization.

@islas
Copy link
Contributor

islas commented May 9, 2024

Right, I think as long as the MPI implementation you want is first in your path it should work, assuming that implementation supplies an mpif90 wrapper.

The 'mpif90' wrapper seems to be more widely available than 'mpifort'.
@mgduda
Copy link
Collaborator Author

mgduda commented May 9, 2024

Right, I think as long as the MPI implementation you want is first in your path it should work, assuming that implementation supplies an mpif90 wrapper.

Ok, I've just pushed another commit to switch from mpifort to mpif90.

@islas
Copy link
Contributor

islas commented May 10, 2024

I am unfortunately seeing the same behavior @HathewayWill described where without specifying the serial compiler you run the risk of getting something else:

/home/aislas/dependencies/mpich/bin/mpif90 -fc=ifx -V
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.0.2 Build 20231213
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

GNU ld version 2.35.2-42.el9
ld: /home/aislas/intel/oneapi/compiler/2024.0/lib/for_main.o: in function `main':
for_main.c:(.text+0x19): undefined reference to `MAIN__'


/home/aislas/dependencies/mpich/bin/mpif90 -V
gfortran: error: unrecognized command-line option ‘-V’


echo $PATH
/home/aislas/tools/ninja/run/bin/:/home/aislas/intel/oneapi/vtune/2024.0/bin64:/home/aislas/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/bin:/home/aislas/intel/oneapi/mpi/2021.11/bin:/home/aislas/intel/oneapi/mkl/2024.0/bin/:/home/aislas/intel/oneapi/itac/2022.0/bin:/home/aislas/intel/oneapi/inspector/2024.0/bin64:/home/aislas/intel/oneapi/dpcpp-ct/2024.0/bin:/home/aislas/intel/oneapi/dev-utilities/2024.0/bin:/home/aislas/intel/oneapi/debugger/2024.0/opt/debugger/bin:/home/aislas/intel/oneapi/compiler/2024.0/opt/oclfpga/bin:/home/aislas/intel/oneapi/compiler/2024.0/bin:/home/aislas/intel/oneapi/advisor/2024.0/bin64:/home/aislas/dependencies/bin:/glade/u/home/aislas/bin:/glade/u/home/aislas/.local/bin/:/glade/u/home/aislas/tools/built/bin:/home/aislas/bin/:/home/aislas/bin/VSCode-linux-x64:/home/aislas/bin/go/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin

My path specifically puts the Intel compiler before the system gfortran...

@HathewayWill
Copy link

This also would apply to the wrf implementation @islas @mgduda

@islas
Copy link
Contributor

islas commented May 10, 2024

The WRF stanza does have those args used, for better or worse.

@mgduda
Copy link
Collaborator Author

mgduda commented May 10, 2024

I don't think the -fc and -cc arguments are standard, which is why I'd prefer to not include them.

@islas Out of curiosity, does the following subroutine compile with both mpif90 -fc=ifx -c <filename> and mpif90 -fc=gfortran -c <filename> on your system?

subroutine nvsend()

   use mpi_f08

   real, dimension(100) :: buf
   integer :: count, dest, tag
   type(MPI_Datatype) :: datatype
   type(MPI_Comm) :: comm
   type(MPI_Request) :: request
   integer :: ierror

   call MPI_Isend(buf, count, datatype, dest, tag, comm, request, ierror)

end subroutine nvsend

(Specifically, you should not load a different set of modules between attempts.)

My suspicion is that both won't work, as the mpi_f08 module was only built with the compiler that was used to build MPICH. This example would illustrate why -fc can't work in general. But perhaps I'm mistaken.

@HathewayWill
Copy link

This also would apply to the wrf implementation @islas @mgduda ??

@mgduda
Copy link
Collaborator Author

mgduda commented May 10, 2024

This also would apply to the wrf implementation @islas @mgduda ??

How we invoke the MPI compiler wrappers should apply equally to WRF and to the WPS.

@HathewayWill
Copy link

What can I test for you on my side? @mgduda

@islas
Copy link
Contributor

islas commented May 10, 2024

@mgduda Suspicion confirmed:

[mmm-mystic {127} test/mpi ]$ which mpif90 
/home/aislas/intel/oneapi/mpi/2021.11/bin/mpif90
[mmm-mystic {0} test/mpi ]$ mpif90 -fc=ifx -c simple.f90 
[mmm-mystic {0} test/mpi ]$ ll
total 8
drwxr-xr-x. 2 aislas aislas   40 May 10 10:53 ./
drwxr-xr-x. 4 aislas aislas   82 May 10 10:50 ../
-rw-r--r--. 1 aislas aislas  309 May 10 10:51 simple.f90
-rw-r--r--. 1 aislas aislas 1368 May 10 10:53 simple.o
[mmm-mystic {0} test/mpi ]$ mpif90 -fc=gfortran -c simple.f90 
f951: Fatal Error: Reading module ‘/home/aislas/intel/oneapi/mpi/2021.11/include/mpi/mpi_f08.mod’ at line 1 column 2: Unexpected EOF
compilation terminated.
[mmm-mystic {1} test/mpi ]$ pathmunge ~/dependencies/mpich/bin/
[mmm-mystic {0} test/mpi ]$ which mpif90 
/home/aislas/dependencies/mpich/bin/mpif90
[mmm-mystic {0} test/mpi ]$ mpif90 -fc=ifx -c simple.f90 
simple.f90(3): error #7013: This module file was not generated by any release of this compiler.   [MPI_F08]
   use mpi_f08
-------^
simple.f90(7): error #6457: This derived type name has not been declared.   [MPI_DATATYPE]
   type(MPI_Datatype) :: datatype
--------^
simple.f90(8): error #6457: This derived type name has not been declared.   [MPI_COMM]
   type(MPI_Comm) :: comm
--------^
simple.f90(9): error #6457: This derived type name has not been declared.   [MPI_REQUEST]
   type(MPI_Request) :: request
--------^
compilation aborted for simple.f90 (code 1)
[mmm-mystic {1} test/mpi ]$ mpif90 -fc=gfortran -c simple.f90 

So would the onus fall on the user / sys admin to ensure a compatible wrapper is supplied?

Also, while it did do what I wanted and expected (use the compiler I said to), the same cannot be said for OpenMPI:

 [mmm-mystic {0} test/mpi ]$ pathmunge ~/dependencies/openmpi/bin/
[mmm-mystic {0} test/mpi ]$ which mpif90 
/home/aislas/dependencies/openmpi/bin/mpif90
[mmm-mystic {0} test/mpi ]$ mpif90 -fc=ifx -c simple.f90 
gfortran: error: unrecognized command-line option ‘-fc=ifx’
[mmm-mystic {1} test/mpi ]$ mpif90 -fc=gfortran -c simple.f90 
gfortran: error: unrecognized command-line option ‘-fc=gfortran’

@HathewayWill
Copy link

@mgduda @islas

Not sure if this is helpful to you but I tested the compilers after enabling the intel one llvm environment

workhorse@workhorse-MS-7D91:~/Desktop$ source /opt/intel/oneapi/setvars.sh

:: initializing oneAPI environment ...
   bash: BASH_VERSION = 5.1.16(1)-release
   args: Using "$@" for setvars.sh arguments: 
:: advisor -- latest
:: ccl -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: intelpython -- latest
:: ipp -- latest
:: ippcp -- latest
:: mkl -- latest
:: mpi -- latest
:: tbb -- latest
:: vtune -- latest
:: oneAPI environment initialized ::
 
workhorse@workhorse-MS-7D91:~/Desktop$ icc --version
Command 'icc' not found, but there are 27 similar ones.
workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ icx --version
Intel(R) oneAPI DPC++/C++ Compiler 2024.1.0 (2024.1.0.20240308)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2024.1/bin/compiler
Configuration file: /opt/intel/oneapi/compiler/2024.1/bin/compiler/../icx.cfg
workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ ifort --version
ifort: remark #10448: Intel(R) Fortran Compiler Classic (ifort) is now deprecated and will be discontinued late 2024. Intel recommends that customers transition now to using the LLVM-based Intel(R) Fortran Compiler (ifx) for continued Windows* and Linux* support, new language support, new language features, and optimizations. Use '-diag-disable=10448' to disable this message.
ifort (IFORT) 2021.12.0 20240222
Copyright (C) 1985-2024 Intel Corporation.  All rights reserved.

workhorse@workhorse-MS-7D91:~/Desktop$ ifx --version
ifx (IFX) 2024.1.0 20240308
Copyright (C) 1985-2024 Intel Corporation. All rights reserved.

workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ icpc --version
Command 'icpc' not found, did you mean:
  command 'ispc' from snap ispc (1.23.0)
  command 'ifpc' from deb fp-utils-3.2.2 (3.2.2+dfsg-9ubuntu1)
  command 'icp' from deb renameutils (0.12.0-9)
See 'snap info <snapname>' for additional versions.
workhorse@workhorse-MS-7D91:~/Desktop$ icpc --version
Command 'icpc' not found, did you mean:
  command 'ispc' from snap ispc (1.23.0)
  command 'ifpc' from deb fp-utils-3.2.2 (3.2.2+dfsg-9ubuntu1)
  command 'icp' from deb renameutils (0.12.0-9)
See 'snap info <snapname>' for additional versions.
workhorse@workhorse-MS-7D91:~/Desktop$ icpx --version
Intel(R) oneAPI DPC++/C++ Compiler 2024.1.0 (2024.1.0.20240308)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2024.1/bin/compiler
Configuration file: /opt/intel/oneapi/compiler/2024.1/bin/compiler/../icpx.cfg
workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ mpiifort --version
ifort: remark #10448: Intel(R) Fortran Compiler Classic (ifort) is now deprecated and will be discontinued late 2024. Intel recommends that customers transition now to using the LLVM-based Intel(R) Fortran Compiler (ifx) for continued Windows* and Linux* support, new language support, new language features, and optimizations. Use '-diag-disable=10448' to disable this message.
ifort (IFORT) 2021.12.0 20240222
Copyright (C) 1985-2024 Intel Corporation.  All rights reserved.

workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ mpiifx --version
ifx (IFX) 2024.1.0 20240308
Copyright (C) 1985-2024 Intel Corporation. All rights reserved.

workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ mpif90 --version
GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ mpiicc --version
/opt/intel/oneapi/mpi/2021.12/bin/mpiicx: 1: eval: icc: not found
workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ mpiicx --version
Intel(R) oneAPI DPC++/C++ Compiler 2024.1.0 (2024.1.0.20240308)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2024.1/bin/compiler
Configuration file: /opt/intel/oneapi/compiler/2024.1/bin/compiler/../icx.cfg
workhorse@workhorse-MS-7D91:~/Desktop$ 
workhorse@workhorse-MS-7D91:~/Desktop$ mpifc --version
GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

@mgduda
Copy link
Collaborator Author

mgduda commented May 10, 2024

@islas I think your test results do illustrate that (1) -fc= is not portable (it's not supported by all MPI compiler wrappers) and (2) -fc= isn't generally viable (when using any MPI Fortran modules, one cannot just switch Fortran compilers and expect to use the rest of the MPI library compiled with different compiler).

I would say that it's the responsibility of the sys admin or user to ensure that the available serial Fortran compiler agrees with the mpif90 wrapper in the user's path.

@islas
Copy link
Contributor

islas commented May 10, 2024

Sounds good, I'll make a note to change the WRF stanza in the next update to follow suit.

@HathewayWill
Copy link

Sounds good, I'll make a note to change the WRF stanza in the next update to follow suit.

I just got this from the Intel developers. @islas @weiwangncar @mgduda @kkeene44

Screenshot from 2024-05-13 11-19-57

Looks like the old wrappers are going away

@mgduda mgduda merged commit c8ce545 into wrf-model:develop May 13, 2024
@mgduda
Copy link
Collaborator Author

mgduda commented May 13, 2024

@HathewayWill Thanks for the info. On the WPS side, don't think this will affect us, as we don't currently have any configuration stanzas that use the Intel MPI wrappers (mpiifort, mpiicc, etc.).

@HathewayWill
Copy link

HathewayWill commented May 13, 2024

@HathewayWill Thanks for the info. On the WPS side, don't think this will affect us, as we don't currently have any configuration stanzas that use the Intel MPI wrappers (mpiifort, mpiicc, etc.).

I do see this in the configure.defaults in the WRF. Not sure if it's in the WPS ones. @mgduda

#
#        By default, some files are compiled without optimizations to speed up compilation. Removing
#        respective makefile rules in the end of this file will result in longer compilation time, and, possibly
#        Out Of Memory messages, but might produce binaries which are substantially faster.
#
#        Please visit http://www.intel.com/support/performancetools/sb/cs-028607.htm 
#        for latest info on how to build WRF with Intel compilers.
#
#        If you got Out Of Memory message, there are several options:
#          1. Check your memory limits (ulimit -a), possibly increasing swap partitions size.
#          2. Remove any debugging flags (-g, -check, -traceback).
#          3. Force the problematic file to be compiled with less optimizations (see examples at the 
#             end of this file), try -no-ip compiler flag.
#
#        This configuration is aimed at accuracy. To improve performance (at the expence of accuracy) you might
#        consider removing '-fp-model precise' flag from FCBASEOPTS. This enables non value-safe optimizations.
#        Another option is to add '-ftz' flag, which flushes denormal results to zero when the application is in
#        the gradual underflow mode. It may improve performance if the denormal values are not critical to the
#        behavior of your workload. To further improve performance, add suitable vectorization options for your
#        processor to FCOPTIM (see ifort manpage).
#
#        If you have Intel MPI installed and wish to use instead, make the
#        following changes to settings below:
#        DM_FC  = mpiifort
#        DM_CC  = mpiicc
#        and source bin64/mpivars.sh file from your Intel MPI installation
#        before the build.
#
#        Suggestions for timing improvements from Craig Mattocks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants