publish it to maven central or some other repository

[dportabella]

publish it to maven central or some other repository?

[helgeho]

Hi, and thanks for your interest in Web2Warc. The reason why we haven't deployed it to some mvn repository is that it was built as a tool, not as a library. If you want to use it as a dependency in another project, the easiest way right now is to clone and publish it locally, simply run:

git clone https://github.com/helgeho/Web2Warc.git
cd Web2Warc
sbt publishLocal

Now you should be able to refer to it as de.l3s.web2warc_2.10 in your application.
That's in sbt: "de.l3s" %% "web2warc" % "1.0"
If you then build a JAR, for example using sbt-assembly, Web2Warc will be embedded.

Would you please let me know what you would like to use it for?

[dportabella]

I tried that, but it was not working for some reason. I need to look at it again.

I wanted to use Ammonite instead of calling scala -cp ../target/scala-2.10/web2warc-assembly-1.0.jar example.scala, as follows. A domain2Warc file with this content:

#!/usr/bin/env amm

import $ivy.`de.l3s::web2warc:1.0`, de.l3s.web2warc.Web2Warc
import ammonite.ops._

@main
def main(maxLevel: Int, outPath: String, domain: String, seedUrl: String) = {
  if (!domain.matches("(\\w+\\.)+\\w+")) throw new Exception("invalid domain name")

  Web2Warc.crawl.name = domain
  Web2Warc.writer.path = outPath
  Web2Warc.spec.maxLevel = maxLevel
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + java.util.regex.Pattern.quote(domain) + "(/.*|$)"
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + domain + "(/.*|$)"
  Web2Warc.spec.preserveUrlRegex = ".*"
  Web2Warc.spec.followRedirects = true
  Web2Warc.spec.increaseLevelOnRedirect = true
  Web2Warc.seeds += seedUrl

  Web2Warc.run()
}

and run it:

chmod +x ./domain2Warc
./domain2Warc 3 /tmp/out/ epfl.ch http://www.epfl.ch/

---

Note: I cannot crawl the website I need because of a bug in one of your dependency libraries. I issued a pull request: iipc/webarchive-commons#65

[helgeho]

Sorry, it took me a while to find time to look into this, however, I now just managed to run Web2Warc with Ammonite as follows:

The first problem for me, using your domain2Warc file, was that my Web2Warc was compiled against Scala 2.10, while Ammonite seems to use 2.11. Hence, the import statement import $ivy.de.l3s::web2warc:1.0, de.l3s.web2warc.Web2Warc extends to de.l3s::web2warc_2.11:1.0. Manually changing the import to import $ivy.de.l3s:web2warc_2.10:1.0, de.l3s.web2warc.Web2Warc solved this problem for me (please note, there is only one colon after de.l3s now, because of the explicit version in the artifact).

The second problem was that a dependency of Web2Warc (org.netpreserve.commons:webarchive-commons:1.1.5) depends on Hadoop and even though it is not used in Web2Warc and thus should be ignored, Ammonite tried to fetch it from Maven central, which did not succeed. To solve this, I explicitely excluded it in the build.sbt of Web2Warc: 5257d2c

This solved everything for me and I was able to run your example.
Please let me know if this is solving the issue for you, too.
Hope it helps!

[dportabella]

cool, thx!
I tried that, but it fails with:

object de.l3s:web2warc_2.10:1.0 is not a member of package $ivy
  import $ivy.`de.l3s:web2warc_2.10:1.0`, de.l3s.web2warc.Web2Warc

You run amm on scala 2.10? how?
Is there something preventing Web2Warc to use scala 2.11?

[helgeho]

After running sbt publishLocal it is actually showing you the name of the artifact that it has been published to in your local ivy. As web2warc_2.10 does not exist in your case, I suppose your sbt compiled it already against 2.11, but I am not sure about it. However, in that case it should work with the :: notation without an explicit version, except for your Ammonite is running on 2.10. So please check your versions of Scala as well as the Web2Warc artifact resulted from sbt publishLocal.

And yes, Web2Warc can be compiled against Scala 2.10 as well as 2.11, I just checked again and both seem to work. To explicitely compile against 2.11, please run the following commands in the sbt shell (sbt to open it):

++ 2.11.7
publishLocal

This should result in an artifact de.l3s:web2warc_2.11:1.0. To use it in Ammonite, please ensure you are running it on Scala 2.11 (I run it on 2.11, but used the 2.10 Web2Warc artifact, which worked without a problem).

[dportabella]

I don't manage to make it work.

I can compile Web2Warc (setting scalaVersion := "2.11.7" on build.sbt) and it works, but then running this on amm fails with:

$ sbt publishLocal
[info]  published web2warc_2.11 to /Users/david/.ivy2/local/de.l3s/web2warc_2.11/1.0/poms/web2warc_2.11.pom
[info]  published web2warc_2.11 to /Users/david/.ivy2/local/de.l3s/web2warc_2.11/1.0/jars/web2warc_2.11.jar
[info]  published web2warc_2.11 to /Users/david/.ivy2/local/de.l3s/web2warc_2.11/1.0/srcs/web2warc_2.11-sources.jar
[info]  published web2warc_2.11 to /Users/david/.ivy2/local/de.l3s/web2warc_2.11/1.0/docs/web2warc_2.11-javadoc.jar

$ amm
import $ivy.`de.l3s:web2warc_2.11:1.0`, de.l3s.web2warc.Web2Warc

ammonite.runtime.tools.IvyThing$IvyResolutionException: failed to resolve ivy dependencies unresolved dependency: commons-io#commons-io;2.4: configuration not found in commons-io#commons-io;2.4: 'master(compile)'. Missing configuration: 'compile'.

I cannot compile Web2Warc (setting scalaVersion := "2.10.5" on build.sbt), it fails with:

$ sbt publishLocal
sbt.ResolveException: unresolved dependency: commons-io#commons-io;2.4: configuration not found in commons-io#commons-io;2.4: 'master(compile)'. Missing configuration: 'compile'. It was required from org.netpreserve.commons#webarchive-commons;1.1.5 compile

[dportabella]

Note: my ivy repository was corrupted. After deleting it and starting from scratch,
I can compile Web2Warc both on scala 2.11.7 and 2.10.5, but it stills fails when running amm.

[helgeho]

I am sorry for having kept you waiting for so long. Meanwhile, I managed to create a Dockerfile that does the job. I hope this will help you troubleshooting it on your machine as well.

Just copy the domain2Warc file from your post above into an empty folder together with a file named Dockerfile with the following content:

FROM ubuntu

COPY domain2Warc domain2Warc.tmp
RUN tr "\r" " " < domain2Warc.tmp > domain2Warc
RUN chmod +x domain2Warc

RUN apt-get update
RUN apt-get install -y apt-transport-https
RUN apt-get install -y curl
RUN apt-get install -y git

# intall Ammonite
RUN curl -L -o /usr/local/bin/amm https://git.io/vP4Gw && chmod +x /usr/local/bin/amm

# install Scala and SBT
RUN apt-get install -y scala
RUN echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
RUN apt-get update
RUN apt-get install -y sbt

# get and build Web2Warc
RUN git clone https://github.com/helgeho/Web2Warc.git
WORKDIR Web2Warc
RUN sbt "++ 2.11.7" publishLocal

WORKDIR /
ENTRYPOINT ["./domain2Warc"]

To build it call docker build --rm -t domain2warc . from within that directory (of course, you need to have Docker installed). Once built you can run it with docker run --rm -v domain2warc 3 /tmp/out/ epfl.ch http://www.epfl.ch.

This is just running the commands as specified in the Dockerfile, so doing the same on your machine should work as well, or some misconfiguration on your system is preventing it from running, which I hope you will be able to figure out with this example.

Please keep me updated and thanks for using Web2Warc.
Hope this helps!

[dportabella]

hi, thanks for the docker image.
I published your library in my 3rd party mvn repo in order to make it easy for my work colleagues to call your library from a script (without the need to install and build Web2Warc).

The script domain2warc is as follows:

#!/usr/bin/env amm

import ammonite._, Resolvers._
val oss = Resolver.Http("dportabella-3rd-party-mvn-repo-releases", "https://github.com/dportabella/3rd-party-mvn-repo/raw/master/releases/", IvyPattern, false)
interp.resolvers() = interp.resolvers() :+ oss

@

import $ivy.`de.l3s::web2warc:1.0`, de.l3s.web2warc.Web2Warc
import ammonite.ops._

@main
def main(maxLevel: Int, outPath: String, domain: String, seedUrl: String) = {
  if (!domain.matches("(\\w+\\.)+\\w+")) throw new Exception("invalid domain name")

  Web2Warc.crawl.name = domain
  Web2Warc.writer.path = outPath
  Web2Warc.spec.maxLevel = maxLevel
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + java.util.regex.Pattern.quote(domain) + "(/.*|$)"
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + domain + "(/.*|$)"
  Web2Warc.spec.preserveUrlRegex = ".*" // preserve all crawled URLs (default)
  Web2Warc.spec.followRedirects = true
  Web2Warc.spec.increaseLevelOnRedirect = true
  Web2Warc.seeds += seedUrl

  println(Web2Warc.spec.urlRegex)
  println(Web2Warc.seeds)

  Web2Warc.run()
}

And I can call it
domain2warc 2 /tmp/out/ epfl.ch http://www.epfl.ch/

Unfortunately it does not work yet, because of a bug in amm, where a new resolver is not taken into account if run from within an amm script.

It would be great if you can publish Web2Warc in maven central.

[dportabella]

the latest version of amm fixes that bug, so now this updated domain2warcscript finally works:

#!/usr/bin/env amm

import coursier.MavenRepository
interp.repositories() ++= Seq(MavenRepository(
  "https://dl.dropboxusercontent.com/u/3796323/3rd-party-mvn-repo/releases/"
))

@

import $ivy.`de.l3s::web2warc:1.0`, de.l3s.web2warc.Web2Warc
import ammonite.ops._

@main
def main(maxLevel: Int, outPath: String, domain: String, seedUrl: String) = {
  if (!domain.matches("(\\w+\\.)+\\w+")) throw new Exception("invalid domain name")

  Web2Warc.crawl.name = domain
  Web2Warc.writer.path = outPath
  Web2Warc.spec.maxLevel = maxLevel
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + java.util.regex.Pattern.quote(domain) + "(/.*|$)"
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + domain + "(/.*|$)"
  Web2Warc.spec.preserveUrlRegex = ".*" // preserve all crawled URLs (default)
  Web2Warc.spec.followRedirects = true
  Web2Warc.spec.increaseLevelOnRedirect = true
  Web2Warc.seeds += seedUrl

  println(Web2Warc.spec.urlRegex)
  println(Web2Warc.seeds)

  Web2Warc.run()
}

And I can call it:
$ domain2warc 2 /tmp/out/ epfl.ch http://www.epfl.ch/

Note: I published the Web2Warc for Scala 2.11 in my 3rd-party-mvn-repo, so you need Ammonite Scala 0.9.3/2.11-0.9.3, installed by:

$ sudo curl -L -o /usr/local/bin/amm https://git.io/vHqn9 && sudo chmod +x /usr/local/bin/amm && amm

Still, it would be great if you can update the dependencies and publish Web2Warc in maven central. Do you want me to help in this?

[helgeho]

It is (finally) also available on Maven Central:

libraryDependencies += "com.github.helgeho" % "web2warc" % "1.1"

[dportabella]

great, thx! :)