apt_info: massive optimizations and rewrite

[anarcat]

This is a set of tweaks to the apt_info.py script to optimize collection and reorganize the codebase. It started with an attempt at optimizing the code and ultimately meant essentially rewriting the whole thing since it required refactoring the functions into a single one to optimize data structures.

The performance improvement on my laptop is kind of dramatic:

> hyperfine 'python apt_info-34f9073d8.py' 'python apt_info-69fe18add.py' 'python apt_info.py'
Benchmark 1: python apt_info-34f9073d8.py
  Time (mean ± σ):     10.081 s ±  0.318 s    [User: 5.796 s, System: 4.283 s]
  Range (min … max):    9.575 s … 10.832 s    10 runs
 
Benchmark 2: python apt_info-69fe18add.py
  Time (mean ± σ):      9.517 s ±  0.280 s    [User: 5.380 s, System: 4.136 s]
  Range (min … max):    9.101 s …  9.864 s    10 runs
 
Benchmark 3: python apt_info.py
  Time (mean ± σ):     780.7 ms ±  37.3 ms    [User: 741.2 ms, System: 39.3 ms]
  Range (min … max):   732.1 ms … 843.9 ms    10 runs
 
Summary
  python apt_info.py ran
   12.19 ± 0.68 times faster than python apt_info-69fe18add.py
   12.91 ± 0.74 times faster than python apt_info-34f9073d8.py

That's 10 seconds to 800ms!!

On a canary server of ours -- a minimal example -- the performance is less impressive, going from 1.6s to 500ms... But still a 3x improvement!

(Note that the filename above include commit hashes that do not exist here, but are from our own monorepo which includes this file, that we're trying to get back in sync with upstream.)

---

This a large number of commits that have accumulated on our end in the past 18 months. We have held from contributing those back because !234 got stuck, but seeing that it's now merged gives us hope that we can contribute again.

I'm really sorry about the state of this PR. I wish things were more orderly, but this is the best I could do with a couple of hours of work.

You'll note that the commit starts with a revert of dc297f1, which changed the metric names, because our patches were made without merging that change from !234 that I had completely overlooked. Reverting the patch, applying our patchset, and reapplying the fix was much easier than constantly resolving the conflicts over the rebase. If that's really a problem for the history, I could try rebasing the entire thing without commits, but given the massive rewrite, it's going to be a huge ordeal and I'm not sure I would go through it. I'll also note that

We still have a small delta after this. I've ran black over the file locally, and would love to do so here, but for the sake of readability, I have actually managed to port the patches without black. I'd love to hear if such a noop PR would be accepted as well, following this one, to remove our last delta.

Thanks!

[anarcat]

We still have a small delta after this. I've ran black over the file locally, and would love to do so here, but for the sake of readability, I have actually managed to port the patches without black. I'd love to hear if such a noop PR would be accepted as well, following this one, to remove our last delta.

... and it turns out flake8 is unhappy here so, as a stopgap, i just made a commit to please it, but it's exactly the kind of stuff black completely fixes without us having to think about it. :)

[anarcat]

rerolled with signoffs.