make gmp conditionally (build) depends on curl

[luigi-calori]

On some sites (mine) system curl fail to download from gmp url while spack - built succeed, see issue #895

[tgamblin]

@luigi-calori: we've seen the same thing at LLNL. I don't know why curl (and SSL in general) is set up so poorly at HPC sites.

I like this PR but can you make it more general? Ultimately, this is what MacPorts calls a "fetch dependency" but I don't think you need to go that far. It seems like this would be of use to more than just GMP, so you could put the logic here into do_fetch in package.py and have it recursively install curl if needed, then set spack.curl to the spack-installed one. You could add an option to spack itself or to spack install to trigger this...

[eschnett]

On my laptop (OS X), the system curl can download packages just fine, but the Spack-installed has problems. I think it doesn't find the system certificates, or has a similar problem. I don't understand the error:

Cloning into 'SimulationIO'...
fatal: unable to access 'https://github.com/eschnett/SimulationIO.git/': SSL certificate problem: unable to get local issuer certificate
Cloning into 'SimulationIO'...
fatal: unable to access 'https://github.com/eschnett/SimulationIO.git/': SSL certificate problem: unable to get local issuer certificate
==> Fetching from [git] https://github.com/eschnett/SimulationIO.git failed.
==> Error: All fetchers failed for simulationio-master-s36kmlzps2qziewfs6qyrtr6kqmdlzcl

Just pointing out that using a Spack curl doesn't seem to help in all cases...

[luigi-calori]

Tried to follow your suggestion. Now in gmp there is just the conditional dependency on curl.
On my platform,
spack install gmp ---> FAIL
spack install gmp+curl ---> OK

nevertheless note that
spack fetch gmp+curl ---> FAIL as well :

the message is incorrectly:
using spack built curl: path: /usr/bin/curl
It seems that fetch command does not inherit the environment from the dependencies...
And this make sense as curl installation is triggered by install in it's recursive form but probably not by fetch....

[adamjstewart]

Perhaps Spack could use wget if it is installed, else curl. The system curl is unable to download gmp, but wget works just fine.

[citibeth]

I've encountered both these problems; the curl/gmp problem happens first thing, when you try to install GCC. Here is how I solved them. I don't recommend modifying packages.

Curl Won't Fetch from https://

The problem here is an ancient openssl on your system that won't talk to modern https:// servers, for security reason. I don't know why wget worked for @adamjstewart but not curl, assuming they both use the same underlying OpenSSL (try ldd wget and ldd curl to verify).

The solution is to use Spack to bootstrap a new curl (making sure you build your own OpenSSL in the process). Once installed, you should put the appropriate "spack load" commands in your .bashrc (or better yet, the appropriate "module load" commands; see PR #721).

Install Git

Older system-supplied versions of git do not provide features that are
necessary today. You might wish to install the latest, greatest
version of git. Do this with::

spack install git+curl+expat

Once Git is installed, make it available to Bash via::

spack load git

Install Curl

Older system-supplied versions of curl may not work to download
some packages, particularly those using the https protocol. If this
is the case, you may use a newer Spack-installed curl instead::

spack install curl
spack load curl

Updated Binutils

On Ubuntu 12.04, an updated binutils may be required to build some packages (eg, py-numpy)::

spack load binutils

Certificate Problems with Curl

When you install your own curl, it doesn't really know where the system certificates are stored, and ends up having problems. I don't know where they are stored either; I suppose the right way is to set up the correct env vars / config files to give this info to curl. The quick-and-dirty way is to use the --insecure flag with curl. This should not be a problem because Spack checks checksums anyway.

To use --insecure, I use the bash script below; better would be to add some kind of option / configuration to Spack to add the --insecure flag when calling curl. So far, I see no need to use wget (it's simpler to use just one fetcher command inside of Spack instead of two).

#!/bin/sh
#
ROOT=/my/home/dir/spack/opt/spack/linux-x86_64/gcc-4.9.2/curl-7.47.1-g6ghmzdf5lk5bipfys6viwigvcs3yfp7
$ROOT/bin/curl --insecure "$@"

[luigi-calori]

@citibeth Thanks for the hints.
It is questionable weather these bootstrapping problems are worthwhile to be solved within spack ( like i' ve tried to do) or outside, like you are suggesting.could have solved gmp downloading.
In any case it would be good to have a way of sharing solutions.
I do not know if forthcoming support to architecture could help site customization.
The best thin

[luigi-calori]

Sorry ... wrong button pressed...
This PR try to be less invasive as possible by letting packagers to decide if it's worthwhile to require building curl or rely on system one.
Are there quick tests to decide if system provided curl is ok or not and how to properly configure spack built curl?
@tgamblin any suggestion on where to put bootstrap setup for things like curl, git and binutils?
Anyway, it could help to share examples of spack customization and module usage like #744 #721

[mathstuf]

I think this was mentioned at the last meeting, but building on #378 to add the idea of fetch dependencies might be the way to go here. Spack could also just embed knowledge that curl is required for basic things, git for cloning from git repos, etc. instead.

[luigi-calori]

Ok, I agree that having a fetch dependency would be probably the best approach but it seem to me a long shot, as #378 itself is not merged yet.
If the suggestion is non-merge, the suggested approach is, as @citibeth suggest, to
first install curl then load it ( probably better with modules) then install packages that require spack curl to fetch their stuff? It seems to me more work at install time but maybe it is cleaner.
Let me know so I can test the approach and eventually remove this PR ( so the number of PR decrease ;-)

[citibeth]

I'm not keen on the idea of a fetch dependency. What's the purpose, every
package needs curl to be fetched.

Why not take a simpler approach: start by explicitly listing (in English)
the packages that Spack needs installed in order to work. They would be
Python 2.7, git, curl, and a couple of others. Then make a way to tell
Spack where to find the things it needs; for example, tell it in a config
file to load certain modules. That way, users don't have to load those
modules themselves when they want to use Spack.

-- Elizabeth

On Wed, Jun 15, 2016 at 9:50 AM, Luigi Calori notifications@github.com
wrote:

Ok, I agree that having a fetch dependency would be probably the best
approach but it seem to me a long shot, as #378
#378 itself is not merged yet.
If the suggestion is non-merge, the suggested approach is, as @citibeth
https://github.com/citibeth suggest, to
first install curl then load it ( probably better with modules) then
install packages that require spack curl to fetch their stuff? It seems to
me more work at install time but maybe it is cleaner.
Let me know so I can test the approach and eventually remove this PR ( so
the number of PR decrease ;-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#902 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AB1cd5H5k8kSWKLEurzmzepG2-n8TDL-ks5qMAMygaJpZM4IX18I
.

[adamjstewart]

@citibeth makes a good point. I'm currently dealing with the problem that on our older cluster (CentOS 5), the system cURL is too old to fetch many packages. If I try to build it with Spack, the system cURL is actually too old to fetch a newer cURL! The only solution is to use yum to install a newer cURL.

I'm still thinking about how this should apply for CMake though. Some packages I've seen actually depend on a very new version of CMake, which isn't likely to be found on the system. It would be nice if we could specify which version of CMake is required.

[citibeth]

@citibeth https://github.com/citibeth makes a good point. I'm currently
dealing with the problem that on our older cluster (CentOS 5), the system
cURL is too old to fetch many packages. If I try to build it with Spack,
the system cURL is actually too old to fetch a newer cURL!

Ouch!

The only solution is to use yum to install a newer cURL.

There are other solutions as well:

1. Download the tarball manually for the latest curl, and place it in a
   local cache (not sure the right term, but I've done it). Same procedure
   used for proprietary products. Then install normally, and you're
   bootstrapped.
2. Extend Spack to use other programs in place of curl: wget, for example.
3. Build curl manually, and then use that to install the latest curl.

If yum doesn't work out, I would go with option (1), I think it's the
easiest.

I'm still thinking about how this should apply for CMake though. Some
packages I've seen actually depend on a very new version of CMake, which
isn't likely to be found on the system. It would be nice if we could
specify which version of CMake is required.

All packages that require cmake should depends_on('cmake@version:'), it
should not be considered a core Spack requirement. Then things will "just
work." Don't worry if Spack decides to install a few more versions of
CMake than was really necessary. The new build dependency functionality
should improve all of this.

Same thing for autotools: things that depend on autotools (i.e. the
configure script hasn't already been generated in the tarball) should
explicitly depend on it, and ideally provide minimum version numbers.

-- Elizabeth

[adamjstewart]

@citibeth: solution 4: we're actually decommissioning this cluster soon and moving everything else to CentOS 7, so I've decided to ignore it for now. But the argument remains the same, since CentOS 7 will someday also be old.

Anyway, I struggle to see a fundamental difference between requiring a specific version of git/curl/wget and a specific version of cmake/autotools. Also, if a package install method calls cmake(), I think it's pretty obvious that it depends_on('cmake'), whether or not that's specified. I don't want to make package developers do extra work if they don't have to. Besides, currently I don't think Spack uses the CMake that it builds, it uses whatever it finds in the PATH before installation. This could be fixed with the build dependencies PR of course.

[luigi-calori]

I do not want to rise a flame, I did this as it seemed to me worthwhile that if there is a package that spack provide ( like curl or cmake ) that is needed, either at build or even at fetch time, other packages could use it by specifying a dependency.
In this sense, depending on curl means that you need the curl ...
As we do not have root pass, I think we can not use yum, at least for system installs, so I would download new curl manually and put in cache ( not merged yet ;-) or mirror.then with this PR is enough.
My assumption was that the need for a spack build curl was a rare exception that could justify the need to explicitly depends on curl.

[mathstuf]

Besides, currently I don't think Spack uses the CMake that it builds, it uses whatever it finds in the PATH before installation. This could be fixed with the build dependencies PR of course.

Fixed in bb968fc (had been part of build deps, but Todd fixed it separately).

[citibeth]

My Spack always uses the Cmake it builds. Maybe I hacked it to do so, I
don't know. In any case, if it doesn't already, it should.

-- Elizabeth

On Wed, Jun 15, 2016 at 11:12 AM, Ben Boeckel notifications@github.com
wrote:

Besides, currently I don't think Spack uses the CMake that it builds, it
uses whatever it finds in the PATH before installation. This could be fixed
with the build dependencies PR of course.

Fixed in bb968fc
bb968fc
(had been part of build deps, but Todd fixed it separately).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#902 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AB1cd-WX0Sp1SNEVGuYS3axjTcru8uwHks5qMBZMgaJpZM4IX18I
.

[luigi-calori]

As I' m a newbie, I just tried to make curl similar to cmake: if I need a spack built cmake in a package, I depends on cmake, if I need a spack built curl, I depends on curl.
Is it possible at all to use cmake in a package.py without depending on it?
Is it possible that if you not depends on cmake / m4 then you just use the system provided one?

If that is the case, here we make curl behaves like other tools, but maybe I' m missing some essential bit.

[davydden]

you can use system provided cmake and culr, but you should declare dependencies in package anyway and then configure you spack to use system provided one.

[mathstuf]

The weird thing about something like fetch deps is that it's really spack which is requiring that dependency. Spack could just as easily use wget or requests to do the same things (or even convert git checkouts from github into direct downloads rather than clones). However, for cmake, the package itself is requiring CMake; Spack doesn't care, so the package has to say it wants CMake.

[luigi-calori]

What I would like to obtain with this PR is to use spack build curl for fetching instead of system curl if you depend on it during an install that has curl as a dependency.
If this is a wanted behaviour, check if the implementation is correct, if is not wanted, I' ll remove the PR

[mathstuf]

While such a feature might make sense, this is probably not the way to signal it. For example, git requires curl for its library. That doesn't mean that the dependent curl should be used to download git's source. Since this is more of a spack-level thing, I think something like install --fetch-with curl@7.43.0 --fetch-with git@2.8.0 or something would make more sense. I don't know how much work that would be.

[citibeth]

Spack should use whatever curl you put in your path.

On Wed, Jun 15, 2016 at 12:04 PM, Luigi Calori notifications@github.com
wrote:

What I would like to obtain with this PR is to use spack build curl for
fetching instead of system curl if you depend on it during an install that
has curl as a dependency.
If this is a wanted behaviour, check if the implementation is correct, if
is not wanted, I' ll remove the PR

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#902 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AB1cd8vsb4JP2SCOAfrMQRJaq588VIGnks5qMCJ0gaJpZM4IX18I
.

[luigi-calori]

@mathstuf : Are there any reasons why a package that "truly" depends on curl like git should not use the spack built curl executable but should instead use the one from the original environment when the command spack install git was issued?
I understand that curl is a spack needed component so spack has to find it before analysing dependencies, but do not see much danger in fetching one package with a system curl and the next with a spack built curl. Does not happen the same for other system provided components?

@citibeth
If a package use which( 'mycommand' ) the expected behaviour is that mycommand is resolved in the env when spack has been launched or in the env spack is building, influenced by dependencies?

Anyway, let me know weather close it or keep it open.

[mathstuf]

Are there any reasons why a package that "truly" depends on curl like git should not use the spack built curl executable but should instead use the one from the original environment when the command spack install git was issued?

The system curl could know about special certificates required to fetch the data (e.g., the DoD has their own cert bundle that the system curl might know about, but Spack's would not). Spack's curl could also be too old to fetch (and the system has the newer version).

If a package use which( 'mycommand' ) the expected behaviour is that mycommand is resolved in the env when spack has been launched or in the env spack is building, influenced by dependencies?

If which is used in the package.py's install method, you're fine. Looks for the binary immediately. Instead, code in lib/spack should probably use Executable which defers lookup of the executable's path until it is called (e.g., with the build env from package.py or the host environment in things like spack find).

[adamjstewart]

I can't imagine a world in which Spack's cURL would be older than the system cURL.

[mathstuf]

Older isn't necessarily the problem; "knows about the DoD certs to fetch DoD package sources" is.

And people do run bleeding edge distros (personally, I run Fedora Rawhide at home which, e.g., already has 2.9.0).

[adamjstewart]

The DoD certs reasoning is pretty solid. I can't really argue with that. @luigi-calori I think you'll just have to rely on the system cURL for this one. If it isn't new enough like mine was, you'll have to update it manually, either with yum/apt-get or with a Spack installed cURL in your PATH.

[mathstuf]

@luigi-calori I think my proposal of spack install --fetch-with $spec (modulo bikeshedding the option name) to have spack activate (and possibly build) the specified spec to activate only during the fetch phase would be the best way to do this if you really can't use the system version (and would like spack to be the source of the version to use). One way which currently doesn't need patches is to just do spack activate curl before running your other builds.

[adamjstewart]

Closing this PR. Almost every package in Spack downloads with curl, and we certainly aren't going to add an optional curl dependency to every package. A more general solution would be to implement some kind of fetching dependency that is automated (i.e. doesn't involve modifying any package.py files). Until then, there is a perfectly valid workaround of installing a newer version of curl with yum/apt-get or Spack and making sure it is first in your PATH. If you have problems with finding CA certificates, you may need to configure curl properly. In truth, curl is a dependency of Spack, not of gmp.

[citibeth]

Getting a working curl is discussed in the updated docs.