Vagrant box URL for JSON metadata file

In my Vagrantfile, I can specify the URL of a box:

config.vm.box_url = "http://example.com/my-box.pkg"

According to the more recent documentation, I should be able to create a JSON file that contains the URLs for different versions of the box. The documentation also says that I can use the URL of this JSON file when running vagrant box add. I was hoping to be able to use the URL of that JSON file for config.vm.box_url. However, that doesn't seem to work. When I try it, it treats it like a box file:

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'my-box' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Adding box 'my-box' (v0) for provider: virtualbox
    default: Downloading: http://example.com/my-box.pkg.json
    default: Progress: 100% (Rate: 876k/s, Estimated time remaining: 0:00:01)
The box failed to unpackage properly. Please verify that the box
file you're trying to add is not corrupted and try again. The
output from attempting to unpackage (if any):

bsdtar.EXE: Unrecognized archive format: Illegal byte sequence
bsdtar.EXE: Error exit delayed from previous errors.

Is it possible to tell Vagrant to use a box metadata JSON file in my Vagrantfile? I'd rather not have to use Vagrant Cloud.

Solution 1:

As of today (2016-07-12, vagrant 1.8.4), if you want to run your own catalog in a manual way (that is, manually updating the boxes and editing the metadata.json file), but still have it behave like an actual catalog, keep in mind the following things:

There is no need for the file to be named "metadata.json". It can be named anything, as long as it contains the expected values. I'm using metadata.json here to clarify the steps further below.
each metadata.json file can only contain one single box. It can have multiple versions, and each version can have multiple providers (virtualbox, vmware, libvirt). If you need to have more than one box (say, fedora and ubuntu) you need two different metadata files.
Vagrant expects the metadata.json file to have a type of application/json (as Nicholas Hinds mentioned above. If your webserver does not return it (or, returns text/plain), vagrant will assume it's an actual box file, and try to parse it (and fail miserably).
Hashicorp's Atlas (what used to be Vagrant Cloud) is the exception to this, as the redirects lead you to content served as text/html. My best guess for this is it has something to do with the redirects (more on this below).
The box file does not need to be on the same place as the metadata file. You can have your metadata file in a local webserver, and the box in Amazon S3, no problem with that.

So, as far as I got, I found the easiest way to get this working on a webserver and still have pretty normal functionality is to do this:

On your webhost, create a file and directory structure similar to this:

d wwwroot/
d wwwroot/boxes
d wwwroot/boxes/yourname
f wwwroot/boxes/yourname/.htaccess
d wwwroot/boxes/yourname/box1
f wwwroot/boxes/yourname/box1/metadata.json
f wwwroot/boxes/yourname/box1/box1-$version1-$provider.box
f wwwroot/boxes/yourname/box1/box1-$version2-$provider.box
f wwwroot/boxes/yourname/box1/box1-$version2-$otherprovider.box
d wwwroot/boxes/yourname/box2
f wwwroot/boxes/yourname/box2/metadata.json
f wwwroot/boxes/yourname/box2/box2-$version1-$provider.box
(... etc)

(this layout means that your "metadata.json" for box1 will have to have it's URLs pointing to something like http://yourhost/boxes/yourname/box1/box1-$version1-$provider.box)

On your .htaccess, make sure that metadata.json is set for Directory index. The rest is optional, for negative cache and hiding the actual contents:

Header unset Pragma
FileETag None
Header unset ETag
DirectoryIndex metadata.json
IndexIgnore *
Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate, private"
Header set Pragma "no-cache"
Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT"

On your environment, export the VAGRANT_SERVER_URL pointing to your webhost. Note no trailing slash!

export VAGRANT_SERVER_URL="http://yourhost/boxes"

With this in place (and all the files with the correct contents), you can go and add your box directly:

vagrant box add yourname/box1

Since metadata.json is the index file for the box1 directory, it should redirect the contents right to it, vagrant will pick it up, interpret the metadata and download the appropriate box.

Solution 2:

After reading your question again, It seems you're trying to do something a little different than I am - but I think our end goal is the same.

I don't want to utilize the Vagrant Cloud service for hosting my base boxes, but I want to be able to distribute a development environment to my dev team, and utilize the features of the metadata.json file to maintain a versioning system for the development environment, which will then be available to my development team simply by using the facilities built into vagrant.

The vagrant documentation is really sparse in this area at the time of this writing (8/5/2014), presumably because it's a relatively new feature but I'm sure the fact that VagrantCloud has a paid tier has something to do with it also.

To figure out how to utilize the metadata.json file to version and distribute boxes, I took a look at some of the VMs available on the VagrantCloud. After looking through those, and reading some of the vagrant code - it became pretty easy to figure out how to accomplish my goal.

Package your box as you normally would. In my case, I'm packaging only for virtual box, because that's what our developers will be using to run the Vm. I also package a Vagrantfile with my basebox which does some provisioning for the development environment (setting up shares to appropriate folders, some basic apache configs, error logging, etc)

Create a metadata.json file to describe your base box, mine looks similar to this:

{
    "description": "long box description",
    "short_description": "short box description",
    "name": "company/developer-environment",
    "versions": [{
        "version": "1",
        "status": "active",
        "description_html": "<p>Dev Environment</p>",
        "description_markdown": "Dev Environment",
        "providers": [{
            "name": "virtualbox",
            "url": "http:\/\/vagrant.domain.local/dev/company-developer-environment-1.box"
        }]
    }]
}

Once I created my metadata.json file, I uploaded it to a local server running on our internal network (vagrant.domain.local/metadata.json). Once I did that, all that was left was to test it out with vagrant:

# add the box to vagrant using the definition from metadata.json
# (the box is actually downloaded here, so it can take a minute...or 10)
$ vagrant box add http://vagrant.domain.local/dev/metadata.json

# init the box (this creates a .vagrant folder and a Vagrantfile in the cwd with the appropriate box name)
$ vagrant init company/developer-environment

# boot the box
$ vagrant up

Voila, a remotely hosted, shared and versioned, private box that doesn't require usage of the Vagrant Cloud.

As you create new versions of your box, you'll package it up, and edit the metadata.json file. From what I can tell, you can use whatever versioning scheme you want be it semantic versioning (1.0.0, 1.0.1, etc) or just simple whole numbers for versions (1, 2, 3, etc). When your box users vagrant up vagrant automatically checks your metadata.json file for a new version, and will prompt them to do vagrant box update to update the box.

You can also skip the vagrant box add <metadata.json url> and vagrant init bits by defining a base Vagrantfile with the box name and box url, like so:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "company/developer-environment"
  config.vm.box_url = "https://vagrant.domain.local/dev/metadata.json"
end

You could distribute a Vagrantfile with those contents, and all users would just be able to vagrant up. Though, I'm unsure about how that works when the versions get updated.

Vagrant box URL for JSON metadata file

Solution 1:

Solution 2:

Related

Recent Posts