The 1955 printing of The Oxford Universal Dictionary that sits next to my desk defines benchmark (1864) as, “A surveyor’s mark, cut in rock, or other durable material, to indicate the starting or other point in a line of levels for the determination of altitudes over the face of a country. It consists of a broad arrow with a horizontal bar through its apex…”
Cool, right? And very pertinent to geography. But what does this have to do with a performance assessment tool for ArcGIS Pro?
Well, a more modern definition of benchmark offered by Lexico is, “A standard or point of reference against which things may be compared or assessed.” Or, “A test designed to evaluate or compare the performance of computer hardware or software.”
Ah yes, so now this is starting to make sense. The ArcGIS Pro Performance Assessment Tool (PAT) is a set of tests that assesses a computer’s ability to run key ArcGIS Pro workflows. You can use it to investigate a perceived performance issue on its own or with the help of Esri Technical Support. Here is what PAT looks like:
Now you will understand why I led with an image of a physical benchmark. With its medieval reference and geographic significance, I was hoping it would draw you in. But back to PAT…
Why PAT?
We have already explained what PAT is, so let’s move on to why it was created. Benchmarking is not a new practice in the hardware/software world, and when it comes to ArcGIS specifically, it is something that we have had requested periodically.
Performance is an important part of the user experience when it comes to working with software. Factors that contribute to the performance experience include things like hardware, data format, structure, and location, network bandwidth, display and visualization settings, and analysis complexity, among others. Performance can be affected both positively as well as negatively by changes in any one or combination of the above factors. Because of this complexity, before we can even ask, “Why is it slow?” we must first ask, “Is it slow?”
Running a scripted set of commands on common data allows us to provide users with a reproducible benchmark; that is, a point of reference against which things may be compared or assessed. Delivering these tests and data as a self-contained package that you can install and “click a button” to generate a result, allows you to compare ArcGIS Pro performance on your machine and in your environment to our published benchmarks. In summary, it gives you an easy way to understand whether ArcGIS Pro is performing optimally on your machine.
Who should run PAT?
Honestly, we think everyone should run PAT! However, if you’ve been working away in ArcGIS Pro and have never given performance a second thought, maybe there is no need for you to run PAT. But maybe you are working in a project and you just feel like things aren’t running as fast as they should be. Should it run faster? Is it sluggish? How would you know, without anything to compare it to, right? With PAT, now you have something to compare to.
How to use PAT?
Download PAT and follow the instructions in the PAT-README.pdf. PAT contains tests that run at three different levels, each with an average benchmark execution time and typical ranges. After you complete running a level on your machine, compare your summary log to the benchmarks published with the tool. For reference, here is a table describing the tests with each benchmark execution time.
You’ll undoubtedly notice that the ranges for Levels 2 and 3 are more variable than Level 1. This is partly due to the fact that those tests rely on the machine’s GPU and performance will vary accordingly.
There are three possible scenarios when running PAT:
- “I ran PAT and I hit all the benchmarks. I don’t have any performance issues when working with my own projects and data in ArcGIS Pro.”
Great! You’re good to go. Thanks for reading and tell your friends about PAT.
- “I ran PAT and I hit all the benchmarks. But when working with my own projects I’m not satisfied with performance.”
The fact that you hit all the benchmark execution times when running PAT is an indicator that the hardware is in good shape. It will be necessary to dig into specifics about the projects you’re working with. We recommend working through Troubleshooting Performance Issues in ArcGIS Pro to help narrow down the issue and try to isolate the performance bottleneck.
- “I ran PAT and my results are not hitting the benchmarks.”
If you ran PAT from a network location and didn’t hit the benchmarks, copy the ProPAT_and_PerfTools_vN folder to your local machine (C:) and re-run the tests. The difference in execution times could be attributed to network latency. For more information, see Application, project, and data location tips.
If you ran PAT locally, not hitting the benchmarks may indicate that there are issues with the hardware. We recommend working through the Hardware check section of Troubleshooting Performance Issues in ArcGIS Pro and continuing through the Antivirus check and Data location check sections if necessary.
My Personal Story
Before I let you go to try it for yourself, I wanted to share my personal story of how PAT helped me. I have worked with ArcGIS Pro since the beginning and never had any performance complaints until some point last year when things just started to “feel slow.” I wasn’t able to pinpoint any specific issue, my machine specs met the minimum requirements, and since other things on the machine were also a bit laggy, I simply suffered on. There would be points when I’d be demonstrating something to a colleague and we’d both comment, “Hmm, this shouldn’t take this long,” but then whatever it was we were waiting on would finish and we’d move on.
Without anything to compare against to determine whether ArcGIS Pro on my machine was performing as expected or not, I didn’t really have much to go on (try asking your IT department for a new machine because something “feels slow”)! Then, as part of the holistic performance work undertaken for ArcGIS Pro 2.8, we start work on this thing called PAT. A group of us all run an early version of PAT and everybody is reporting Level 1 numbers of just under 3 minutes, yet my machine was clocking in at between 5 and 6 minutes. This was finally the indicator that I needed to be able to work together with our IST department to dig deeper into what the problem was. After doing some performance logging we found that there was a problem with the cooling fans that was causing the CPU to be throttled down to a processing speed of about ¼ of its capacity. We fixed the hardware issue and now I’m seeing PAT results comfortably within the ranges shown above and ArcGIS Pro is performing well!
I wanted to share my own account to drive home the point of how PAT can be used, and set the expectation that while it is a valuable tool to assess Pro performance, it is just that. PAT will not fix performance issues, but will help to determine whether your machine in your environment is running key ArcGIS Pro workflows within typical ranges. This is a very powerful ability as part of troubleshooting workflows.
Conclusion
We hope that sharing benchmarks for ArcGIS Pro performance, along with a tool for you to compare your machine to these established standards, helps you assess where you stand compared to expected results. PAT can be used along with the troubleshooting document referenced above as a self-service tool to troubleshoot perceived performance issues, or if necessary, to provide valuable information to Esri Technical Support if further investigation is necessary.
Happy benchmarking!
Article Discussion: