This one is based on Ubunu Bionic, instead of the default CentOS/7 image.

The rest is taken from the ceph/ceph repo.

1. start vms with only the ceph packages installed

   In src/pybind/mgr/ssh run vagrant up to create a cluster with a monitor, manager, and osd nodes. The osd node will have two small extra disks attached.

2. generate an ssh_config file for the vm hosts

   Execute vagrant ssh-config > /path/to/ssh_config to generate a ssh configuration file that contains hosts, usernames, and keys that will be used by the bootstrap cluster / ssh orchestrator to establish ssh connections to the vagrant vms.

3. install ssh orchestrator dependencies

   The primary dependency is the remoto package that contains a Python SSH client for connecting to remote nodes and executing commands.

   Install with dnf install python3-remoto. The version must be >= 0.0.35. At the time of writing this version is being packaged and is not available. To install from source:

   git clone https://github.com/ceph/remoto
   cd remoto
   python3 setup.py sdist
   pip3 install --prefix=/usr dist/remoto-0.0.35.tar.gz
   

4. start the bootstrap cluster (in this case a vstart.sh cluster)

   Start with a network binding to which the vms can route traffic:

   vstart.sh -n -i 192.168.121.1

   The following is a manual method for finding this address. TODO: documenting a automated/deterministic method would be very helpful.

   First, ensure that your firewall settings permit each VM to communicate with the host. On Fedora, the trusted profile is sufficient: firewall-cmd --set-default-zone trusted and also allows traffic on Ceph ports. Then ssh into one of the vm nodes and ping the default gateway, which happens to be setup as the host machine.

   [nwatkins@smash ssh]$ vagrant ssh mon0 -c "getent hosts gateway"
   192.168.121.1   gateway
   

5. setup the ssh orchestrator backend

   Enable and configure the ssh orchestrator as the active backend:

   ceph mgr module enable ssh
   ceph orchestrator set backend ssh
   
   # optional: this document assumes the orchestrator CLI is enabled
   ceph mgr module enable orchestrator_cli
   

   Configure the ssh orchestrator by setting the ssh_config option to point at the ssh configuration file generated above:

   ceph config set mgr mgr/ssh/ssh_config /path/to/config
   

   The setting can be confirmed by retrieving the configuration settings:

   [nwatkins@smash build]$ ceph config get mgr.
   WHO    MASK LEVEL    OPTION                            VALUE                                             RO
   mgr         advanced mgr/orchestrator_cli/orchestrator ssh                                               *
   mgr         advanced mgr/ssh/ssh_config                /home/nwatkins/src/ceph/src/pybind/mgr/ssh/config *
   

   The next set of instructions we should move to the docs folder

   ceph orchestrator host add osd0 ceph orchestrator host add mgr0 ceph orchestrator host add mon0 ceph orchestrator device ls ceph orchestrator mgr update 3 mgr0 mgr1