Skip to content

C++: Reduce memory pressure from getInstruction #13207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 26, 2023

Conversation

MathiasVP
Copy link
Contributor

Since we merged #12900 the Foundations teams internal performance measurements have been OOM'ing when analyzing Wireshark because they're running their experiments with ~6GB of RAM (unlike our DCA analysis of Wireshark which uses a much larger runner).

The analysis was OOM'ing because of the use of the shortestDistances HOP. This use was introduced in 8368c37 to get around an issue where Eclipse was OOM'ing because the log file was too large 😅.

This PR reverts 8368c37 and replaces the transitive closure of adjacentInBlock (which also was OOM'ing when I tried it locally) with an EquivalenceRelation-based approach. This succeeds even with 5GB of memory 🎉.

We do, however, now have 148647 iterations of getMemberIndex 😱. So we need to run DCA (and maybe even MRVA?) to check if this is a change we want to make.

MathiasVP added 2 commits May 17, 2023 13:46
…HOP in 'getInstruction'. This reduces the memory pressure when generating the CFG for Wireshark.
@MathiasVP MathiasVP requested review from a team as code owners May 17, 2023 12:57
@MathiasVP MathiasVP added the no-change-note-required This PR does not need a change note label May 17, 2023
@MathiasVP
Copy link
Contributor Author

MathiasVP commented May 17, 2023

Hmmm... Apparently, this breaks some of our tests 🤔. I'll investigate.

Edit: Fixed in 57cc316.

@rdmarsh2
Copy link
Contributor

Stage timings on DCA look pretty noisy. I'm OK with the code but we should look into that.

@MathiasVP
Copy link
Contributor Author

Stage timings on DCA look pretty noisy. I'm OK with the code but we should look into that.

Sure, we can do that. Note that the stage timings behavior is actually quite consistent with our observations. In all of the projects, the CppType::CppType#1#IRConstruction::Raw#14#Operand#3#SSAConstruction::Cached#7#TOperand::Internal#3 stage is the one with the largest regression (which is most likely the stage that evaluates getInstruction).

@MathiasVP
Copy link
Contributor Author

Yeah, the query-specific stage timing alerts flagged up here is just noise.

@rdmarsh2 rdmarsh2 merged commit 5bc844c into github:main May 26, 2023
MathiasVP added a commit to MathiasVP/ql that referenced this pull request May 31, 2023
…s-in-getInstruction"

This reverts commit 5bc844c, reversing
changes made to b2fb2aa.
@MathiasVP MathiasVP mentioned this pull request May 31, 2023
sashabu added a commit that referenced this pull request May 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C# C++ no-change-note-required This PR does not need a change note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants