Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 13 Regressions on 1/23/2025 10:40:01 PM +00:00 #111913

Open
performanceautofiler bot opened this issue Jan 28, 2025 · 3 comments
Open

[Perf] Linux/x64: 13 Regressions on 1/23/2025 10:40:01 PM +00:00 #111913

performanceautofiler bot opened this issue Jan 28, 2025 · 3 comments
Assignees
@amanasifkhalid
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
10.0.0

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline ba72b9787d6c591557b4fc60d3e2a602f2b18dd3
Compare 0a477a86d0fea6234ada63e660c0faa310fe121d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.10 ns 8.80 ns 1.24 0.34 False
18.37 ns 19.90 ns 1.08 0.10 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives&lt;Single&gt;*'

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Single>.Add_Scalar(BufferLength: 128)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Single>.AddMultiply_Vectors(BufferLength: 128)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline ba72b9787d6c591557b4fc60d3e2a602f2b18dd3
Compare 0a477a86d0fea6234ada63e660c0faa310fe121d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
5.12 ns 20.38 ns 3.98 0.12 False
2.96 ns 4.11 ns 1.39 0.20 False
4.02 ns 5.12 ns 1.27 0.21 False
7.76 ns 9.56 ns 1.23 0.16 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

System.Memory.Span<Byte>.Reverse(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Byte>.IndexOfValue(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Byte>.StartsWith(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Byte>.SequenceEqual(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline ba72b9787d6c591557b4fc60d3e2a602f2b18dd3
Compare 0a477a86d0fea6234ada63e660c0faa310fe121d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.02 ns 10.95 ns 1.56 0.22 False
18.24 ns 20.11 ns 1.10 0.09 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives&lt;Int32&gt;*'

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Int32>.Max_Scalar(BufferLength: 128)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Int32>.AddMultiply_Vectors(BufferLength: 128)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline ba72b9787d6c591557b4fc60d3e2a602f2b18dd3
Compare 0a477a86d0fea6234ada63e660c0faa310fe121d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.13 ns 5.41 ns 1.31 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

System.Numerics.Tests.Perf_BigInteger.Equals(arguments: 259 bytes, Same)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline ba72b9787d6c591557b4fc60d3e2a602f2b18dd3
Compare 0a477a86d0fea6234ada63e660c0faa310fe121d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.58 ns 25.63 ns 2.68 0.25 False
15.51 ns 19.84 ns 1.28 0.08 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

System.Memory.Span<Char>.Reverse(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.SequenceEqual(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline ba72b9787d6c591557b4fc60d3e2a602f2b18dd3
Compare 0a477a86d0fea6234ada63e660c0faa310fe121d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Base64Tests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
57.87 ns 63.18 ns 1.09 0.12 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*'

System.Buffers.Text.Tests.Base64Tests.ConvertToBase64CharArray(NumberOfBytes: 1000)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline ba72b9787d6c591557b4fc60d3e2a602f2b18dd3
Compare 0a477a86d0fea6234ada63e660c0faa310fe121d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tensors.Tests.Perf_FloatingPointTensorPrimitives<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
167.64 ns 187.86 ns 1.12 0.18 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_FloatingPointTensorPrimitives&lt;Single&gt;*'

System.Numerics.Tensors.Tests.Perf_FloatingPointTensorPrimitives<Single>.Distance(BufferLength: 3079)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Jan 28, 2025
@performanceautofiler performanceautofiler bot mentioned this issue Jan 28, 2025
Closed
@LoopedBard3
Copy link
Member

LoopedBard3 commented Jan 28, 2025 •
edited
Loading

Potentially related to #111736, FYI @amanasifkhalid, some of the tests also may be noise.

Related regressions:

@LoopedBard3 LoopedBard3 transferred this issue from dotnet/perf-autofiling-issues Jan 28, 2025
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 28, 2025
@LoopedBard3 LoopedBard3 added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jan 28, 2025
@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jan 29, 2025
@dotnet-policy-service Dotnet Policy Service
Copy link
Contributor

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

@amanasifkhalid amanasifkhalid added this to the 10.0.0 milestone Feb 3, 2025
@amanasifkhalid amanasifkhalid removed untriaged New issue has not been triaged by the area owner needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Feb 3, 2025
@amanasifkhalid amanasifkhalid self-assigned this Feb 3, 2025
@amanasifkhalid
Copy link
Member

I looked at the largest regressions from this as part of #113108, and I don't think these regressions (or at least most of them) can be attributed to #111736. That change affects only methods with exception handling, and looking at the hot path of, say, System.Memory.Span<Byte>.Reverse(Size: 512) (which regressed 4x on Viper Ubuntu), there are no EH regions. For that particular benchmark, I looked at the JIT dumps of every method optimized in the benchmark's call stack using the baseline and diff JITs, and I don't see any meaningful differences, so I think we can rule out #111726 and #111653 as well. None of the other PRs in the diff look suspicious either.

@LoopedBard3 do you know if any changes were made to the benchmark machines around this time period? Some of these regressions were substantial and have persisted, but I don't see any codegen changes we can point to. Thanks!

@amanasifkhalid amanasifkhalid mentioned this issue Mar 21, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amanasifkhalid amanasifkhalid

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
10.0.0
Development

No branches or pull requests
3 participants
@jeffschwMSFT @LoopedBard3 @amanasifkhalid