Hook up running make test with testport.

[mat813]

No description provided.

[bdrewery]

Thanks for working on this.

Note 'test' needs to be handled by the '-k' mechanism, implied by '-cat', so the error from 'test' doesn't immediately stop but is just noted as a failure in the end. - So plist/qa testing is still done.

[mat813]

It's going to make the patch a bit bigger 😄

[mat813]

Is that something like this that you had in mind ?

[bdrewery]

Yup, looks familiar (I don't have time to dive into this now).

Do we want 'test' to always run?

Note also that DEVELOPER is not passed down unless we allow it. Grep for DEVELOPER in common.sh and you'll see you need to do similar where the test framework needs it. We also need to fetch TEST_DEPENDS in list_deps

[bdrewery]

While I don't have a lot of time on this now, I'll merge what looks right and it can be put into -devel for wider testing.

[mat813]

test is a no-op if not enabled in the port's Makefile, so I would say yes.

I'll have a look at the rest later.

[AMDmi3]

test runs after install

[mat813]

Well, they don't depend on install, they depend on stage, so, I'm not certain it's true.

[AMDmi3]

Oh. Stage and install are messed up and install has different meaning from inside and outside the port. I assume the later is meant here, so the code is correct.

[mat813]

Well, maybe tests need to run after the port has been really installed, and not just staged, I don't know, I'm just saying this is coherent with what bsd.port.mk does :-)

[mat813]

Ok, now, it also builds TEST_DEPENDS.

[mat813]

So, there is another problem, that I'm not quite certain on how to fix, test depends are sometime circular, today, I got a:

[00:00:35] ====>> Cleaning the build queue
tsort: cycle in data
tsort: py27-py-1.4.30
tsort: py27-pytest-2.7.1
tsort: cycle in data
tsort: p5-Test-Warnings-0.021
tsort: p5-Test-Kwalitee-1.22

I'm not exactly sure on how to fix that.

[mat813]

ping ?

[bdrewery]

You said there was a circular dependency problem. Is that fixed?

Regards,
Bryan Drewery

On Nov 10, 2015, at 04:13, Mathieu Arnold notifications@github.com wrote:

ping ?

—
Reply to this email directly or view it on GitHub.

[AMDmi3]

Can we maybe only enable tests which have no TEST_DEPENDS as a first step?

[DragonSA]

Given that the make test target depends on make stage I would submit that any circular dependencies are a problem with the port and not a feature that needs to be supported by poudriere. My motivation for that is if one where to do make test and the port isn't installed it would (kinda) fail from ports. (Also, how common is the circular dependency? If minor cannot one error out and have something available than nothing at all)

[bdrewery]

I think what we need to do to fix the cyclic dependency problem is add separate items into the queue. One to build the package, and another to test it. The test one will install the test depends (extract it again), install itself, then handle running the tests. Granted I haven't looked at this in depth but I think this may ultimately be the need. #259 would be needed first.

[mat813]

If poudriere could detect that adding TEST_DEPENDS doesn't add a cyclic dependency, it could run make test at the same time as the rest, and only postpone make test if there is a cyclic dependency.

[bdrewery]

I'm going to make testport into a two-pass thing here. Phase 1. Nothing changes from today. Phase 2. Build dependency tree considering TEST_DEPENDS, then run the tests.

bulk will have to wait until later. I at least want some kind of testing support in here, at the very least for testport.

[bdrewery]

A fix for #259 is nearly ready to merge. I will try to get port testing into 3.2.

[bdrewery]

The queue needs to be smarter. Rather than everything in the queue being implicitly "build foo" it needs to be explicit. So "build foo deps bar" and "test foo deps foo baz". This avoids the double pass since the 1 queue can deal with it without reworking all of bulk/testport to deal with 2 passes.

[moufjenkins]

Can one of the admins verify this patch?

[bdrewery]

Ok the whole idea of testing separately is killed because 'make test' depends on 'make stage'. So it must happen at the same time or I need to tar the workdir. So it seems the simplest solution is going to be to drop testing of anything that has a circular dependency. Not sure if tsort will allow that or not.

[mandree]

ping

[bdrewery]

The queue needs to be smarter. Rather than everything in the queue being implicitly "build foo" it needs to be explicit. So "build foo deps bar" and "test foo deps foo baz". This avoids the double pass since the 1 queue can deal with it without reworking all of bulk/testport to deal with 2 passes.

I have this working in a local branch named generic-queue-job_type-nodir. I just need to add in the part of saving the wrkdir and take the build_port changes here as a base for a new test_port that picks up from the tarred wrkdir.

And for some ports we can likely combine the build and test jobs into 1 without the wrkdir tarring.

[ocochard]

ping

[bdrewery]

The queue work is done. The feature can be implemented now.

[svmhdvn]

I have some bandwidth to help push this forward now. Has anyone tested the current state of this PR, and what's remaining? This would really help with eventually bringing in CI to the Ports collection and doing testing of all ports on a monthly or so basis.