Ive created a js app that can be used to test solutions for the community. Certain ones people made for challenge #1 wont be able to be put through it since they accept user input through entering it while its running rather than command line arguments but ill be enforcing command line argument use going forward
Let me know if theres any issues you find with it. I tested with with javascript, python, and rust and seemed to be working great. There can be differences in runtime when you run the same thing multiple times so ill be taking the median of running it 50 times for the performance score in the challenge
So is the idea to run our own solutions through it locally and post the results alongside our submissions?
Does it permit providing all the test cases as a long argument list to do it all in a single invocation, or is that cheating?
I find hyperfine very nice for casual benchmarks, FWIW.
People can use it to test their solution locally but for final results it will be ran by the people running the challenge (since test cases are hidden + so people can’t fake a time)
You can enter as many test cases as you want into the txt file and it’ll run through them all. You need to make your own though
Thanks!
Regarding my second question, I’m thinking for some languages there’s a performance hit each time the program/function/runtime is launched, and that it’s more performant to handle all tests with a single invocation that loops through them all.
But I think, and am looking to confirm, that that form is not allowed for these challenges, so we should only have each solution take a single test case.
I have it coded where it will check how long it takes for a basic program to launch and subtract that from the time
But yeah solutions should only take 1 test case at a time