Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update install prereqs python fix #1782

Merged
merged 16 commits into from
Jun 22, 2023
Merged

Conversation

TedThemistokleous
Copy link
Collaborator

-Updated install_prereqs when using 22.04
-Added new Dockerfile for Ubuntu 22.04 used for testing
-Kept removed items commented in new dockerfile for visibility in this PR, can remove later.

Tested and ran with Navi10 and MI250 servers and make check passes

Needed to run containers with 22.04
Updated dockerfile to use ROCm 5.5 and Ubuntu 22.04 for use with building MIGraphX
Able to run make -j$(nproc) check successfully with this
@TedThemistokleous TedThemistokleous added enhancement New feature or request bugfix Fixes a bug found in the code. dependencies Pull requests that update a dependency file labels May 25, 2023
@codecov
Copy link

codecov bot commented May 25, 2023

Codecov Report

Merging #1782 (58919ed) into develop (226da49) will not change coverage.
The diff coverage is n/a.

❗ Current head 58919ed differs from pull request most recent head e176eb4. Consider uploading reports for the commit e176eb4 to get more accurate results

@@           Coverage Diff            @@
##           develop    #1782   +/-   ##
========================================
  Coverage    91.55%   91.55%           
========================================
  Files          419      419           
  Lines        15503    15503           
========================================
  Hits         14193    14193           
  Misses        1310     1310           

# Setup ubsan environment to printstacktrace
ENV UBSAN_OPTIONS=print_stacktrace=1
ENV ASAN_OPTIONS=detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1
RUN ln -s /opt/rocm/llvm/bin/llvm-symbolizer /usr/bin/llvm-symbolizer
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we keep this dockerfile in sync with the other dockerfile? This looks like a duplicate of Dockerfile. Also docker files should be named Dockerfile or *.docker.

Copy link
Collaborator Author

@TedThemistokleous TedThemistokleous May 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The big add here was using the Pinned version Package via

# From docs.amd.com for installing rocm. Needed to install properly
RUN sh -c "echo 'Package: *\nPin: release o=repo.radeon.com\nPin-priority: 600' > /etc/apt/preferences.d/rocm-pin-600"

along with the updated keyring signing needed.

The only real difference I see here from 22.04, and 20.04 where

  • FROM statement
  • 22.04 breaks with g++-7 & clang-format-10
  • changes to preqs

I suppose we can migrate the specifics to the install_prereqs.sh script for 20.04 , and then pass in which target dist we want somehow for our FROM? Maybe use --build-arg? That would mean a smaller update to our CI for additional containers from changing FROM and updating prereqs

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose we can migrate the specifics to the install_prereqs.sh script for 20.04 , and then pass in which target dist we want somehow for our FROM?

The problem is not just the differences but the large amount that is copied. I dont think we can move everything into install_preqs.sh(that also can destroy any docker layer caching).

Instead, we should either upgrade the current Dockerfile to use ubuntu 22.04(which seems like a seperate PR), or just drop this extra dockerfile thats not tested(and I would prefer not to increase CI jobs to test it though).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean I would agree with updating the currently docker to 22.04 but then preserving the older dockerfile in Dockerfiles/Docker_.dockerfile so if we need to revert back/testing would be a good historical record of what's worked / used by us

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean I would agree with updating the currently docker to 22.04 but then preserving the older dockerfile in Dockerfiles/Docker_.dockerfile so if we need to revert back/testing would be a good historical record of what's worked / used by us

We can get the old docker from git. No need to duplicate it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So should I just remove the 22.04 docker, or is the plan to migrate it in another PR at some target date?

Copy link
Member

@umangyadav umangyadav Jun 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is not just the differences but the large amount that is copied.

Docker files can be composed.

e.g. https://github.com/ROCmSoftwarePlatform/migraphx-benchmark-utils/blob/main/dockerfiles/Daily.Dockerfile

This Dockerfile takes base docker image as an argument during docker build. We could create a base docker image similar to that and customize on top it.

https://github.com/ROCmSoftwarePlatform/migraphx-benchmark-utils/blob/main/dockerfiles/Dockerfile

This one is the base docker image which again takes linux/ubuntu version and ROCM_RELEASE as the argument during docker build.

But that would mean two-step process of docker build inside the CI.

tools/install_prereqs.sh Outdated Show resolved Hide resolved
tools/install_prereqs.sh Outdated Show resolved Hide resolved
tools/install_prereqs.sh Outdated Show resolved Hide resolved
tools/install_prereqs.sh Outdated Show resolved Hide resolved
-use one protobuf version
-remove extra python3.8 installs from 3.10 case
Dockerfile_Ubuntu_2204 Outdated Show resolved Hide resolved
tools/install_prereqs.sh Outdated Show resolved Hide resolved
tools/install_prereqs.sh Outdated Show resolved Hide resolved
@migraphx-bot
Copy link
Collaborator

migraphx-bot commented May 26, 2023

Test Batch Rate new
9a9151
Rate old
41ba30
Diff Compare
torchvision-resnet50 64 893.27 893.54 -0.03%
torchvision-resnet50_fp16 64 5,302.14 5,290.87 0.21%
torchvision-densenet121 32 1,122.38 1,124.74 -0.21%
torchvision-densenet121_fp16 32 3,269.26 3,269.98 -0.02%
torchvision-inceptionv3 32 591.72 591.71 0.00%
torchvision-inceptionv3_fp16 32 2,510.06 2,492.21 0.72%
cadene-inceptionv4 16 328.79 328.77 0.01%
cadene-resnext64x4 16 392.51 392.52 -0.00%
slim-mobilenet 64 7,118.13 7,117.28 0.01%
slim-nasnetalarge 64 159.59 159.52 0.04%
slim-resnet50v2 64 1,088.03 1,087.88 0.01%
bert-mrpc-onnx 8 716.85 717.72 -0.12%
bert-mrpc-tf 1 368.79 366.65 0.58%
pytorch-examples-wlang-gru 1 296.62 289.84 2.34%
pytorch-examples-wlang-lstm 1 300.53 300.37 0.05%
torchvision-resnet50_1 1 91.37 91.32 0.05%
torchvision-inceptionv3_1 1 128.40 128.56 -0.13%
cadene-dpn92_1 1 332.46 333.48 -0.31%
cadene-resnext101_1 1 235.99 235.46 0.23%
slim-vgg16_1 1 53.26 53.27 -0.02%
slim-mobilenet_1 1 1,478.01 1,459.25 1.29%
slim-inceptionv4_1 1 99.14 98.82 0.32%
onnx-taau-downsample 1 316.09 315.69 0.13%
dlrm-criteoterabyte 1 20.97 20.99 -0.13%
dlrm-criteoterabyte_fp16 1 38.64 38.66 -0.04%
agentmodel 1 5,559.54 5,594.31 -0.62%
unet_fp16 2 52.42 52.36 0.11%

This build is OK for merge ✅

@migraphx-bot
Copy link
Collaborator

migraphx-bot commented May 26, 2023


    :white_check_mark:bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

    :white_check_mark:bert-mrpc-tf: PASSED: MIGraphX meets tolerance

    :white_check_mark:pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

    :white_check_mark:pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

    :white_check_mark:torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

    :white_check_mark:torchvision-inceptionv3_1: PASSED: MIGraphX meets tolerance

🔴cadene-dpn92_1: FAILED: MIGraphX is not within tolerance - check verbose output


    :white_check_mark:cadene-resnext101_1: PASSED: MIGraphX meets tolerance

    :white_check_mark:slim-vgg16_1: PASSED: MIGraphX meets tolerance

    :white_check_mark:slim-mobilenet_1: PASSED: MIGraphX meets tolerance

    :white_check_mark:slim-inceptionv4_1: PASSED: MIGraphX meets tolerance

    :white_check_mark:dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

    :white_check_mark:agentmodel: PASSED: MIGraphX meets tolerance

    :white_check_mark:unet: PASSED: MIGraphX meets tolerance

tools/install_prereqs.sh Outdated Show resolved Hide resolved
tools/install_prereqs.sh Outdated Show resolved Hide resolved
Copy link
Collaborator

@CharlieL7 CharlieL7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, testing ubuntu2204 dockerfile on my Navi21 system just in case

@CharlieL7
Copy link
Collaborator

LGTM, testing ubuntu2204 dockerfile on my Navi21 system just in case

Looks to work

@kahmed10 kahmed10 merged commit c5cd87c into develop Jun 22, 2023
10 checks passed
@kahmed10 kahmed10 deleted the update_install_prereqs_python_fix branch June 22, 2023 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix Fixes a bug found in the code. dependencies Pull requests that update a dependency file enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants