Run previously failed tests first

[christiangnrd]

This will prevent lots of wasted CI time when paired with --quickfail. Or with runs that are usually expected to fail (GPU backends on julia nightly for example)

Open to implementation feedback

[christiangnrd]

@giordano I haven't looked into how this'll interact with #125 but we may want to wait for that one to get merged and fix this rather than the other way around.

It's also possible that it would just work

[christiangnrd]

Marking this as draft since it seems to merge cleanly with #125 but doesn't run the failed tests first anymore. I'll fix it up after #125 is merged

[christiangnrd]

This feels too invasive to me... I think maybe just a new file with the test failures could work and then old versions can just ignore it

[giordano]

This feels too invasive to me...

I do like the idea though. I also recently started using --quickfail in CI to get quicker feedback, and running previously failing tests early after pushing a possible fix would fit into the scheme of getting quick feedback. Having to rename a test file looks more like a hack to me.

[christiangnrd]

Having to rename a test file looks more like a hack to me.

I meant create a new file that just contains the test failure that way the durations data structure stays a Dict{String, Float64}.

I've pushed my proposed new implementation. Now TestHistoryEntry is only ever used for sorting and stays a Float64 for everything else, which mean I don't have to define a whole bunch of interface functions (convert, ...) when all I really want the struct for is comparison.

[christiangnrd]

I'm done touching this PR until review so you can safely review without worry that I'll push

[giordano]

Looks good besides the last comment!