# Slow and Steady: Measuring and Tuning Multicore Interference **Authors:** D. Iorga, T. Sorensen, J. Wickerson, A. F. Donaldson **Venue:** RTAS, 2020 **PDF:** [rtas2020.pdf](../rtas2020.pdf) --- Slow and Steady: Measuring and Tuning Multicore Interference Dan Iorga Imperial College London, UK d..iorga17@imperial.ac.uk Tyler Sorensen UC Santa Cruz, USA Princeton University, USA tyler.sorensen@ucsc.edu John Wickerson Imperial College London, UK j.wickerson@imperial.ac.uk Alastair F. Donaldson Imperial College London, UK afd@imperial.ac.uk Abstract—Now ubiquitous, multicore processors provide repli- cated compute cores that allow independent programs to run in parallel. However, shared resources, such as last-level caches, can cause otherwise-independent programs to interfere with one another, leading to significant and unpredictable effects on their execution time. Indeed, prior work has shown that specially crafted enemy programs can cause software systems of interest to experience orders-of-magnitude slowdowns when both are run in parallel on a multicore processor. This undermines the suitability of these processors for tasks that have real-time constraints. In this work, we explore the design and evaluation of tech- niques for empirically testing interference using enemy programs, with an eye towards reliability (how reproducible the interference results are) and portability (how interference testing can be effective across chips). We first show that different methods of measurement yield significantly different magnitudes of, and variation in, observed interference effects when applied to an enemy process that was shown to be particularly effective in prior work. We propose a method of measurement based on percentiles and confidence intervals, and show that it provides both competitive and reproducible observations. The reliability of our measurements allows us to explore auto-tuning, where enemy programs are further specialised per architecture. We evaluate three different tuning approaches (random search, simulated annealing, and Bayesian optimisation) on five different multicore chips, spanning x86 and ARM architectures. To show that our tuned enemy programs generalise to applications, we evaluate the slowdowns caused by our approach on the AutoBench and CoreMark benchmark suites. Our method achieves a statistically larger slowdown compared to prior work in 35 out of 105 benchmark/chip combinations, with a maximum difference of 3.8×. We envision that empirical approaches, such as ours, will be valuable for ‘first pass’ evaluations when investigating which multicore processors are suitable for real-time tasks. I. INTRODUCTION Multicore processors have become thoroughly mainstream, and are now routinely used even in safety-critical settings [31]. Yet, in a real-time domain, the greatest asset and defining feature of multicore architectures – that they allow work to be distributed among multiple processing cores working simultaneously – begets a significant weakness: interference between cores. This interference arises from contention between cores for shared resources (e.g. caches, buses, and main memory), and it can have a significant and unpredictable effect on the worst-case execution time (WCET) of programs [24]. This undermines the suitability of multicore processors in scenarios that involve real-time constraints. This paper is concerned with empirical techniques for estimating upper bounds on how much interference a program running on a multicore processor could be subjected to. Although empirical techniques cannot promise to identify the absolute worst case, they are nonetheless valuable for making rapid comparisons between candidate processors. The key idea is to design enemy programs that access shared resources in just the right way to maximise the contention with a program running on another core [24], [21], [12]. Prior work has designed several enemy programs, demonstrating substantial slowdowns on commercial off-the-shelf (COTS) processors of more than 300× [3]. The first contribution of this paper is a method for reliably estimating how much interference these enemy programs re- ally cause. We propose a two-pronged measurement strategy that directly mitigates against factors that can be controlled (e.g. frequency throttling and thread migration) and uses statistical methods to mitigate against factors that are more difficult to control. We apply our measurement strategy to enemy programs proposed in prior work, and show that while there is no doubt that very large slowdowns are possible, it is difficult to reproduce the same magnitudes of slowdown once the various potentially-confounding factors are taken into account. Thus, armed with a method for reliably measuring multicore interference, our second contribution is to investigate the extent to which auto-tuning techniques can be effective at gen- erating better enemy programs. To this end, we have designed and implemented an open source tool that automatically tunes the parameters of a set of ANSI-C enemy programs, aiming to maximise the interference they cause on shared resources. Our tool supports three different auto-tuning strategies – random search, simulated annealing and Bayesian optimisation – and we compare the effectiveness of each. We have applied our tool to create enemy programs for five different chips, listed in Table I. We evaluate our approach using benchmarks from the CoreMark [6] and AutoBench [2] real-time application suites. Our auto-tuning framework is able to create enemies that cause slowdowns up to 3.8× larger than the slowdowns caused by enemies studied in prior work that were hand-tuned by experts. More generally, our results show that auto-tuning for about 10 hours can lead to statistically significant improvements in TABLE I: Development boards used to evaluate our approach Name Short name SoC Arch Cores Raspberry Pi 3 B Pi3 BCM2837 Arm A53 4 DragonBoard 410c 410c Adreno306 Arm A53 4 Intel Joule 570X 570x Atom 4 Nano-PC T3 T3 S5P6818 Arm A53 8 BananaPi M3 M3 A837 A7 8 observed slowdowns for 35 out of the 105 benchmark/board combinations that we tried. We attribute the effectiveness of auto-tuning here to its ability to uncover parameters that exploit architectural features that are not immediately obvious and are thus unlikely to be discovered even by experts. A system developer can use these enemies with instrumentation and measurement tools (e.g. performance counters), to gain more insights into the system. An added benefit of our methodology over expert-tuned enemies is that it is portable: it can be used to automatically tune effective enemies across multiple, distinct architectures. We emphasise that auto-tuning is only enabled once our reliable measurement methodology is in place. Without this, the slowdowns associated with dif- ferent enemy program configurations cannot be meaningfully compared. In summary, our main contributions are: • a set of principles for how enemy programs can be rigorously evaluated, with emphasis on reproducibility (Section II), • an auto-tuning framework that can automatically config- ure enemy programs to maximise interference without the need for expert knowledge (Section III), and • an experimental study instantiating our auto-tuning framework with our reliable measurement method and using it to tune enemy programs that have been proposed in prior work, obtaining enemies that can be up to 3.8× more effective at provoking interference in real-time applications (Section IV). Our tool is open source and available online.1 II. MEASURING INTERFERENCE RELIABLY We first present our approach for measuring the interference caused by an enemy program in a reliable manner, which underpins our practical instantiation in Section IV of the auto- tuning approach that we present in Section III. As in prior work [24], [3], we are interested in evaluat- ing commodity hardware using a regular (i.e. non-real-time) operating system, as this is a realistic usage scenario for early evaluation of COTS processors. But unlike this prior work, which pays little-to-no explicit attention to measurement reliability (i.e. variance and uncontrolled interference), we go to great lengths to stabilise the system environment. This significantly lowers the amount of measurement variance, which in turn allows us to produce results that have high statistical confidence. To this end, we rely on two principles: 1https://github.com/mc-imperial/multicore-test-harness (1) controlling as many sources of interference as possible, and (2) acknowledging that not everything can be controlled and using statistics to account for any remaining interference. Section II-A describes our unsuccessful attempt to reliably replicate prior work in an uncontrolled environment. Sec- tions II-B and II-C then show how these results can be made more reproducible by following our first principle, and our first and second principles, respectively. The net effect of applying our approach is to “tame” the slowdowns achieved by the enemy programs of prior work so that they yield smaller, but still impressive, slowdowns that can be reliably reproduced, and that are thus more likely to be genuinely attributable to the interference effects of enemy programs. We attempted to reproduce the highest slowdowns reported in recent interference work by Bechtel and Yun [3]: namely, that their BwWrite enemy program can cause a slowdown of more than 300× on a synthetic memory-intensive piece of software (on a Raspberry Pi 3 B chip). The architectural intuition behind these results is that the BwWrite program exploits the internal structure of non-blocking shared caches by filling the miss-status holding register and write-back buffer. The section is structured around a discussion of Figure 1, which has six bars, (a) through (f). Bar (a) shows the results for the BwWrite enemy program reported by Bechtel and Yun in [3]. Bars (b) through (f) show results for our efforts to reproduce their results under successively more controlled conditions, with (b) using an uncontrolled environment at one extreme, and (f) a controlled environment combined with statistical techniques. A. Uncontrolled environment The experiment described in [3] was performed on the Pi3 developer board, with one core executing the synthetic memory-intensive software and the other cores executing instances of the BwWrite enemy program. For simplicity, we shall henceforth refer to a setting where a program under test is running on one core and the other cores are executing instances of the BwWrite enemy program as the Bechtel and Yun Environment (BYE). The exact operating system used in [3] is not specified, so we installed a clean version of Raspbian Buster Lite on the Pi3– the minimal install available for this development board. Since Bechtel and Yun [3] do not discuss efforts to control the environment, we also did nothing in this regard at this stage. We are interested in configurations that produce high inter- ference only under specific timing conditions, and thus might produce such high interference only rarely. Because of this, we need to measure the same configuration multiple times. Bar (a) of Figure 1 shows the slowdown reported in [3], while bar (b) shows the results we obtained trying to replicate this result in our freshly installed (uncontrolled) environment. We ran each experiment 20 times, taking 25 measurements each time and recording the maximum value. The error bars represent the variation of this maximum across the 20 runs. We were able 2 (a) (b) (c) (d) (e) (f) 200 250 300 400 500 Slowdown factor (log scale) Fig. 1: Slowdowns obtained using the BwWrite enemy pro- gram on the Pi3: (a) as reported by Bechtel and Yun [3], and then reproduced by ourselves (b) in an uncontrolled environ- ment, (c) in an environment where just frequency throttling is accounted for, (d) in an environment with controlled frequency throttling and frequency scaling, (e) in a fully controlled environment, and (f) using our statistical approach. The graph uses a logarithmic scale. Error bars for (b) to (e) show the range of maximum values observed. Error bars for (f) show the confidence interval around the percentile. to obtain results that are similar to those reported in [3], but with very high variance. Under the hypothesis that the environment in which ex- periments are performed can significantly affect performance measurements, we proceed to incrementally clean up (i.e. reduce interference) from the environment. B. Controlled environment Taking inspiration from previous work that measures mi- croarchitectural details through microbenchmarking [10] (but not in the context of real-time systems), we first consider mitigation of factors related to the hardware and operating system that might cause us to wrongfully conclude that ob- served slowdowns are due to the effects of enemy programs. 1) Hardware: Hardware mechanisms are generally de- signed to be transparent to the user and can be unpredictable. We take into account two factors here: frequency throttling and cache warmth. a) Frequency throttling: When the temperature of a processor increases beyond a certain limit, the processor can throttle its clock frequency to reduce the temperature. As an example, Figure 2 shows how frequency is affected by temperature on the Pi3. The data was gathered for a run of a cache-intensive workload (though in our experience, similar results can be obtained using any computationally-intensive workload) using the vcgencmd tool. The figure demonstrates that frequency throttling can cause serious fluctuations in performance measurements. To mitigate the risk of mistaking throttling effects for enemy-induced interference, we measure temperature reported by the operating system at the end of each experiment and discard the result if the reported 78 79 80 81 82 83 800 1,000 1,200 1,400 Temperature (°C) Frequency (MHz) Fig. 2: Frequency variation due to temperature on the Pi3. As the execution temperature increases, frequency throttling is activated, slowing down the core and making it difficult to determine the cause of the slowdown in execution time. temperature is above 80 degrees Celsius. We empirically found that using this temperature threshold works well on all the processors used in our experiments. Bar (c) of Figure 1 shows results for the BwWrite example when we control for frequency throttling only. This decreases variation by 56%, but has a small and not statistically signifi- cant effect on the maximum slowdown factor (the confidence intervals associated with bars (b) and (c) overlap). b) Cache warmth: We flush the cache at the beginning of each experiment to guard against the possibility that data left over from one experiment might affect the execution time of the next experiment. 2) Operating System: Another source of unreliability in execution time is the preemption mechanisms of modern operating systems. A preempted victim may have a longer execution time due to the interrupted time, as well as potential cache pollution by the interrupting task. This interference should not be attributed to the enemy processes. To maximally mitigate against measurement error due to preemption we would have to run experiments in a “bare- metal” fashion or via a proven real-time operating system. However, this would destroy the portability of our approach, and the manual effort required per processor would defeat the main intended use of our work as a early-stage method to help in the selection of COTS processors for real-time systems. We thus run experiments under Linux, taking account of: thread migration, preemption, scaling governance, and ensuring parallel execution. a) Scaling governance: We ensure that the operating system does not adjust the processor clock frequency by using the “powersave” non-dynamic governor. Bar (d) of Figure 1 shows results associated with the BwWrite enemy program when the governor is set to “pow- ersave“. Comparing bars (c) and (d) we see that this more steady frequency management setting decreases the maximum detected slowdown by 33%. b) Thread migration: To guard against the operating system migrating threads between cores, we pin the program under test (PUT) and enemy programs to specific cores using the taskset Linux command. c) PUT preemption: We run the PUT at the highest available priority level to reduce the risk that our measure- 3 0 50 100 150 200 1.5 2 2.5 3 Context switches Execution time[us] Fig. 3: The effect of context switches on the execution time on the Pi3 ments are affected by the cost of context switching. We now discuss an empirical investigation into the impact of, and our approach to reduce the number of, context switches. We use the getrusage command, which returns the number of context switches a process had during execution. First, we determine if context switches have a statistically significant impact on execution time. To do this, we measured the context switches for the experiment in Figure 1. We plotted the samples using the number of context switches as the x- axis and the execution time as the y-axis. Visually we can see a clear correlation, and this can be statistically confirmed using the Spearman’s rank-order correlation test which shows a strong positive correlation. Next, we investigate whether setting the scheduling of the PUT has a statistically significant effect on reducing the number of context switches. We ran the same experiment as the previous paragraph, however, this time the priority of the PUT is set to the maximum allowed value. To determine if this scheduling scheme meaningfully affects the number of context switches, we compare the results using the Wilcoxon rank-sum method [20], a non-parametric test to determine if one set is stochastically larger than another set. The test confirms that the number of context switches is significantly reduced, returning a p-value of less than .05. Concretely, the median number of context switches observed per PUT execution without priority scheduling is 88, and it decreases to 75 when the maximum priority is set. At the end of this subsection, we will return to Figure 1 and show that while context switching (and other environmental factors, discussed next) do correlate with execution time, their overall impact on slowdowns (and variance) is small compared to the previously discussed frequency throttling and power governance. d) Ensuring parallel execution: Our measurements will be meaningless if the enemies do not actually run in parallel with the PUT, so we need to take care with respect to the pro- cess startup latency. To evaluate the maximum startup latency on each platform, we used a latency evaluation framework [32] and discovered that the maximum startup latency is generally below 1 ms. Thus, we conservatively allow 10ms between launching enemy programs and launching the PUT. Bar (e) in Figure 1 shows the slowdown obtained when the environment is controlled as described above. We can see that smaller slowdowns are obtained, but that there is less variation between measurements. The values reported in prior work [3] are between the ones obtained in the uncontrolled environment and the controlled environment. This highlights how difficult it is to reproduce an experiment without knowing the exact configuration in which it was run. To summarise the findings in this section, frequency throt- tling and scaling are the key factors that influence the degree of slowdowns and the variation. While the other environmental factors, such as scheduling priority, do have a correlation to the PUT execution time, they are not statistically significant when compared to frequency throttling and scaling, as can be seen in Figure 1, bars (b)-(f). We have presented an accessible approach, e.g. that can be straightforwardly applied to COTS systems, for controlling the execution environment. However, as we note, we cannot account for all possible sources of interference, e.g. context switches, and they do correlate with execution time, e.g. see Figure 3. Our efforts to control such factors are impactful (see Figure 1), however, we still observe a high variance. Thus, we next present a statistical approach that aims to decrease the variance, i.e. increase measurement precision, even in the presence of uncontrollable environment factors inherent in COTS systems. C. Statistical approach If the remaining variance in experimental measurement illustrated by the bar (e) of Figure 1 was entirely due to the interference induced by the enemy programs, then taking the maximum of the interference measurements would be appropriate, as prior works have done. However, despite the stringent efforts described in Sec- tion II-B, it is not possible to fully eliminate nondeterministic aspects of the environment that might interfere with our measurements. For example, without extreme manual effort it is not practical to ensure that system daemons have been completely disabled. We thus need a statistical analysis over multiple measure- ments that does a reasonable job of estimating the maximum interference caused by enemy programs, ignoring outliers arising due to external system events that we have not been able to control. Figure 4 illustrates how outliers can affect the measurement of interference caused by enemy programs. The figure shows the execution speed of a vector addition program executing on the 410c (see Table I), running alongside various random 4 a b c d e f g h i j k l m n o p q r s t u 0.5 1 1.5 2 Configuration Execution time (s) Fig. 4: Each point shows the execution time of a simple vector addition running alongside various enemies on the 410c. 80 85 90 95 100 1 1.05 1.1 1.15 Execution time (s) Fig. 5: The 95% confidence interval for each percentile after 1000 measurements on the Pi3, when causing interference to a vector addition program. enemy program configurations.2 We ran the vector addition program 20 times in the presence of each enemy program configuration a through u. The 20 points above each configuration in Figure 4 show, for each run, the execution time that was observed. The two unusually high execution times associated with enemy configuration k are clearly suspicious. While it is possible that these high times really are due to rare, timing-related interference between enemy configuration k and the program under test, it is prudent to assume that these outliers are blips caused by uncontrolled external factors. If we would measure interference as the max- imum slowdown observed over all runs, we would certainly select configuration k as the most effective of the enemy pro- gram configurations in Figure 4, discarding the more readily- reproducible results associated with e.g. configuration o. To account for such outliers, we aim to measure interference with respect to a percentile that is as close to the 100th percentile as possible, but such that the effects of rogue outliers are eliminated. The qth percentile of a data set is defined as the value where q% of the data is below that value. We take inspiration here from prior work on comparing latency and end-to-end times of Linux schedulers [7]. To choose a reasonable value for the percentile, we ran a pilot experiment consisting of the previously mentioned vector addition application alongside three enemy programs. We measured the execution time 1000 times and calculate the 95% 2These configurations were produced by the random search auto-tuning strategy that we discuss in Section IV-B, but for the purposes of the present discussion all that matters is that each represents a distinct configuration of enemy programs whose interference effects we are interested in measuring. confidence interval for the 75th to 100th range of percentiles. The confidence interval gives an estimated range of values which is likely to include the true value, the estimated range being calculated from a given set of sample data. The results of this experiment are highlighted in Figure 5 where we can observe that as we increase the percentile, we also have to tolerate a wider confidence interval. At this point, we have to make a trade-off between wanting to be confident in the accuracy of our measurements and wanting not to ignore interesting cases. We ran the same experiment on the other boards at our disposal and have chosen the 90th percentile as a reasonable trade-off. Returning to the BYE, bar (f) in Figure 1 shows the results obtained after introducing our statistical approach. Comparing bars (e) and (f), the slowdown reported when outliers are removed is slightly more modest, but the confidence interval is substantially smaller. Such small confidence intervals allow us to reliably compare multiple enemy processes in terms of interference caused on a program under test. Summary: To recap again what Figure 1 shows with respect to the slowdowns reported by Bechtel and Yun [3] for their BwWrite enemy program: bars (a) and (b) indicate that we can sometimes reproduce the same order-of-magnitude slowdown, but with extremely high variance; bars (c), (d) and (e) show how we progressively improved the control of our environment; finally, bar (f) shows that, by using statistical methods to prune outliers, these more modest, yet still impressive, slowdowns can be observed with much lower variance. Armed with this rigorous method for measuring slowdowns, we now propose an automated technique for tuning enemy programs to maximise interference. III. TUNING HOSTILE ENVIRONMENTS We now describe our method for tuning a hostile en- vironment—a set of enemy programs, each of which will run on its own distinct core in parallel with a PUT—with the aim of developing a hostile environment that interferes maximally with the PUT. In this section we describe our approach in general terms, abstracting away specifics such as the shared resource with which interference is associated and the search strategy used to guide the tuning process. We provide a concrete instance of this framework, with detailed experimental results, in Section IV. Section III-A details the two-stage nature of our tuning approach. In the first stage, we use auto-tuning to search for enemy template parameters per resource. In the second stage, a more generic hostile environment is constructed from the previously-tuned enemy programs. In Section III-B we discuss the tractability of single- and multi-stage tuning strategies and various assumptions we have made. A. Our Two-Stage Tuning Methodology A user of our approach must first decide on the interference paths they wish to investigate, i.e. shared resources for which multicore contention might cause slowdowns. For each one 5 of these paths they must create: (1) a parameterised enemy template that will run in an infinite loop and stress the resource and (2) a victim program that performs a fixed amount of synthetic work, making heavy use of the resource. This is the main manual task associated with using the method we present, and the results obtained by our framework will only be meaningful if the user provides suitable enemy and victim programs. We detail example enemy and victim programs that we have found to work well for a number of shared resources in Section IV and others that did not work equally well in Appendix A. Our tuning methodology has two stages. In Stage 1 we automatically tune the parameters of each enemy template with the objective of maximising the effect of the enemy on its corresponding victim program. Our framework is parametric with respect to both the search strategy used and the specific notion of “effect” that determines the objective function. We discuss practical choices of search strategy in Section IV-B, and use the statistically-rigorous measurement approach of Section II-C as the objective function in practice. Because the victim program is vulnerable to interference on the target resource, the degree to which it is affected serves as a proxy for measuring interference on the associated interference path. Alternatively, the real PUT can be used as a tuning victim. However, this would only be practical if the PUT is (a) known in advance, and (b) sufficiently fast-running to be executed a large number of times during the tuning phase. In Section IV-D we present results for experiments that explore the trade-offs associated with PUT tuning. Stage 2 combines the resource-specific tuned enemy pro- grams found in Stage 1 to construct a hostile environment that aims to be effective across all considered resources. That is, because we aim to interfere with an arbitrary PUT, which may utilise multiple shared resources in complex ways, we search for a combination of tuned enemy programs that is effective across all victim programs. Rather than tune enemy program parameters across multiple victim programs (we argue in Section III-B that this is not tractable for large search spaces), we instead exhaustively search for a Pareto-optimal combination of individually-tuned enemy programs. 1) Stage 1: Tuning Enemy Programs per Resource: We now rigorously describe how, for a given chip, the parameters for an enemy template are tuned to maximise interference with the corresponding victim program. In what follows, let R denote the set of resources to be targeted. For every r ∈R, a victim program (Vr) and enemy template (Tr) must be provided. We note that while R can contain arbitrary sets of resources, it is up to the user to design these judiciously. When tuning for memory hierarchy interference as in this paper, we use R = {cache, memory}. A template Tr takes a set of parameters drawn from a parameter set Pr. For a given parameter valuation p ∈Pr, let Tr(p) denote the concrete enemy program obtained by instantiating Tr with p. Let us define an environment as a set of programs, each of which can be assigned to a core. If these programs are enemy programs, we say that the environment is hostile. For a victim program Vr and environment e, let perf(Vr, e) denote the performance associated with executing Vr on core 0, in parallel with e running across the remaining cores. For the sake of generality, we leave the specific definition of perf abstract here. In practice, it could measure wall-clock execution time, readings obtained from performance counters, or consumption of some other resource, e.g. energy. For a template Tr and parameter valuation p ∈Pr, let itf(Vr, Tr(p)) denote a numeric metric value for the interfer- ence associated with (1) executing Vr in isolation on core 0 (with all other cores unoccupied), compared with (2) executing Vr on core 0, in parallel with an instance of Tr(p) on every other available core. That is: itf(Vr, Tr(p)) = perf(Vr, Tr(p)) perf(Vr, nop) where nop denotes the absence of an enemy process. The aim in the first stage is to determine the parameters that maximise interference to the victim program as follows: arg max p∈Pr itf(Vr, Tr(p)) (1) Because Pr is too large to search exhaustively, search strate- gies can be used to approximate the maximum. In Section IV-B we evaluate several natural choices: random search, simulated annealing, and Bayesian optimisation. Let ptuned r denote the best parameter setting that was found via search using the chosen strategy. We refer to the set {Tr(ptuned r ) | r ∈R} as the set of resource-tuned enemies. 2) Stage 2: Tuning a Generic Hostile Environment: Stage 1 considers the same enemy running on every available core other than that occupied by the PUT. We now aim to devise a deployment of enemy programs that is effective at inducing interference across all resource types since we do not know the resource usage profile of the PUT a priori. We determine the best configuration of enemy programs by using a strategy similar to the one described by Sorensen et al. [27]. In particular, for an n-core processor where the PUT executes on core 0, we seek a hostile environment that maps each core i ∈{1, . . . , n −1} to a resource-tuned enemy.3 We now describe our strategy for choosing a suitable hostile environment from the set of |R|n−1 possibilities. First, for each resource r ∈R, we rank all possible hostile environments according to the extent to which they cause interference with Vr, i.e. according to itf(Vr, e), with the environment that induces the largest interference ranked first. Let RankedEnvironments(r) denote this ranking. While the number of possible hostile environments grows exponentially with the number of cores, for three resources it is feasible to exhaustively search the space of 8 and 128 candidate environments associated with 4- and 8-core chips, 3We also considered leaving cores unoccupied to account for the possibility that multiple enemies might interfere with each other, in which case interfer- ence might not always be maximised by filling up all cores. However, this does not seem to be the case in practice. 6 respectively. We discuss the use of approximations to scale to larger core counts in Section III-B. We then select a Pareto-optimal hostile environment. This is an environment e for which there does not exist an en- vironment e′ ̸= e such that for all r ∈R, e′ is ranked more highly than e in RankedEnvironments(r). Being Pareto- optimal, e may not be unique. In such cases, we break ties by selecting the environment that is ranked better in all but one of the RankedEnvironments(r). In our experimental results, there were no ties after this step. B. Assumptions and Tractability We now describe several assumptions the above tuning methodology makes about multicore interference and comment on some aspects of the tractability of the approach. 1) Multi- vs. Single-Stage Tuning: We have described a two-stage process, in which enemies are first tuned to cause slowdowns in their corresponding victims, and second, a Pareto-optimal combination is found across all victim pro- grams. It is possible to formulate these stages into a single bilevel optimisation problem where the combination of enemy programs is the outer optimisation task, and individual enemy template parameters are the inner optimisation task. While such a formulation is mathematically elegant, the parameter space quickly becomes very large. Additionally, as shown in Section IV-B, the search space in this domain is often unstructured, thus more intelligent searching techniques are not more effective than simple random sampling. Indeed, our pilot experiments using this approach failed to provide meaningful slowdowns after many hours of tuning. The two-stage tuning approach allows more precise explo- ration into individual interference paths, which are interesting even by themselves, before being merged into a generally effective hostile environment. 2) Tuning With All Cores Occupied: In the first stage of tuning, we assign n −1 enemy programs, all with the same parameters, across all cores except one (which is used to run the victim program). However, in the second stage, we select a mapping where the enemy programs that were tuned on n−1 cores may execute on fewer cores. Thus, it is natural to wonder whether enemies tuned on n −1 cores remain effective when run on fewer cores. Our decision to tune on n−1 cores is based on three factors. First, considering every possible core count for every resource during tuning would lead to a blow-up of the already very large parameter-tuning space associated with one core count. Second, hostile environments are noisy, making it challenging to measure interference reliably (as discussed in depth in Section II). As a result, we can reason more confidently about our measurements in cases where interference is most pronounced, i.e. when all available cores are occupied. Third, we have found that the following monotonicity property tends to hold in practice, where r ∈R is a resource, p, p′ ∈Pr are parameter valuations, and itf(Vr, Tr(p), m) is the same as itf(Vr, Tr(p)) except that Tr(p) is executed on m < n cores: (itf(Vr, Tr(p)) > itf(Vr, Tr(p′)) ∧0 < m < n) =⇒itf(Vr, Tr(p), m) > itf(Vr, Tr(p′), m) (2) That is, if parameter valuation p beats p′ when enemies are run on all available cores then p tends to also beat p′ when enemies are run on fewer cores. We tested this property over a subset of possible configurations by sampling the parameter space of a cache enemy program (described in Section IV) for one hour on the Pi3, and found the statement to hold true for 93% of runs. 3) Scalability: Stage 2 exhaustively searches combinations of core-to-enemies mappings. This is tractable for the 4- and 8-core chips examined in this work, core counts that we understand to be typical for processors being considered for use in real-time systems. Stage 2 could be straightforwardly adapted to larger core counts by approximating a Pareto- optimal hostile environment using a search strategy, similar to the way search strategies are used to approximate the global maximum when tuning enemies per resource in Stage 1. IV. AN INSTANCE OF OUR AUTO-TUNING FRAMEWORK We now describe how we have combined the auto-tuning methodology of Section III with the rigorous measurement approach of Section II to automate the process of creating a hostile environment that provokes interference on shared memory resources. In Section IV-A we explain how we have generalized the enemy programs used in prior work by Bechtel and Yun [3] to create a parameterized enemy template suitable for auto- tuning, with an associated victim. We outline several natural choices for the search strategy that can be used to guide auto-tuning in Section IV-B. In Section IV-C we present experimental results related to creating a hostile environment with respect to synthetic victim programs, comparing the effectiveness of different search strategies for this purpose. Having found a hostile environment that is particularly effective for each board, we evaluate the extent to which tuning with respect to synthetic victim programs translates to slowdowns on real-world benchmarks and compare to previous work in Section IV-D. Finally, in Section IV-E, we delve deeper into reasons for the effectiveness of our enemy pro- grams by presenting data obtained via hardware performance monitor counters. Appendix A briefly details alternative enemy programs that we experimented with, that either proved to be less effective than our generalisation of the BYE or were targeting resources that we ultimately considered beyond the scope of this paper. A. Creating enemy templates For our instantiation, we focus on the cache and the main memory as the interference paths. Previous work has shown how different configurations can be used to affect these paths [3], [28], [24]. The size of the memory being accessed as well as the operations performed on that memory are all 7 TABLE II: The enemy/victim code, the tunable parameters and their corresponding ranges. HEADER is “;;” for enemy programs and “int i=0;i