Looking for advice on a CI / regression testing platform

Hi all,

I'm looking for some advice regarding how to set up a basic CI regression / testing suite. This isn't my full time job, but a side project my group at work wants to spin up to... shall we say, give us a more real time monitoring of functionality and performance regressions coming out of the underlying software stack development (long story).

As none of us are particularly automation experts, I was looking for some advice from my fellow Tilderinos. Please forgive me if any of the below is obvious and/or silly.

A few basic requirements I had in mind:

1. Can handle different execution environments: essentially different versions of the software stack, both in docker form and (eventually) via lmod or some other module file approach (e.g., TCL), and sensible handling of a node list.

2. Related to one, supports using the products of builds as execution environments. Ideally we'd like to have a build step compile the stack and install it to a NFS from which we can load it as a module.

3. Simple to add tests. Again, this isn't our full time job -- we mostly want to add a quick bash script / makefile / source code or the like to the tests when we run into an issue and forgot about it.

4. Related. We should be able to store the entire thing as a git repo. I have seen this to some extent with Travis, but my experience with Jenkins was... sub-par (is there a history? Changelog? Any way at all of backing up the test config?).

5. Some sort of post-processing capabilities. At a glance we need to be able to see the top line performance numbers for 20-30 apps over the different build environment. Bonus points if there's a graph showing performance vs build version or the like, but honestly a CSV log file is good enough.

6. Whatever CI software we get has to be able to run this locally. Lots of these are internal only numbers / codes. FOSS prefered.

7. A webui for scheduling runs / visualizing results would be nice, but again this could be a bash script and none of us would bat an eye.

Any thoughts would be greatly appreciated. Thanks!