Salt - upgrade one or thousand servers at once

[viking60]

I was looking for a KISS (Keep It Stupidly Simple) approach to manage different boxes. There are programs like Puppet , Ansible,R(?)ex, Chef, Cdist, Cfengine, Cobbler and Fabric that can do the job too. But Salt seems to like Arch and I like Arch so I went for the CLI KISS approach with Salt. Salt will probably be the fastest of them if it has a Arch like KISS style.

And with it I should be able to install the same on all my boxes - at the same time. All 10000 of them (No joke Salt can handle all of them, it is the most scaleable solution).
But before I take off and land on the moon; lets just get started:

I installed salt from the community repo on my Arch boxes:

Code: Select all

sudo pacman -S salt

Then I went to /etc/salt and edited the minion file in two places:

Code: Select all

master:<IP of the Masterbox>

I had to uncoment it first.
Then I found

Code: Select all

#id:

and replaced it with

Code: Select all

id: 1st_server_to _be_controlled

Here you can provide any name you like (you are simply giving the box an ID, so do not give the same name to other boxes).

Then I started the service on the boxes I wanted to control:

Code: Select all

sudo systemctl start salt-minion

All "minion" boxes are the ones I want to control from my "Master"

On the Master box i started the master service:

Code: Select all

sudo systemctl start salt-master

(On non systemd boxes you may have to start the services differently. like this service salt-minion start)

I cannot hook the boxes together before they have exchanged secure keys and identification so to list them i did a:

Code: Select all

sudo salt-key -L

And voila there under Unaccepted keys the name I had given under "Id:" (see above) turned up.
to accept all keys I did a:

Code: Select all

salt-key -A

I could also have accepted them individually with

Code: Select all

salt-key -a 1st_server_to _be_controlled

etc.

Ok I am hooked up now and salt has this nice pacman module so I will be testing that next.....
The pacman module of Salt uses pkg.something (yes lots of commands but I found them easily).
So to find out what needed to be upgraded on all my hooked up boxes I did a:

Code: Select all

sudo salt '*' pkg.list_upgrades

As you can see here viking60-server was fully upgraded but heidi-lap had a few updates.
So time to upgrade all my Archboxes then:


Now I have also added a Centos server to my stack I will probably have to work with config files now - I'll tell you how it goes.....
....
Wow!
I used the pkg syntax to install gmrun on all boxes

Code: Select all

sudo salt '*' pkg.install gmrun

And it did nothing on viking60-server because it is already installed - same thing with heidi-lap. But it did install it on Centos!

I am really beginning to like this thing: It can update a load of computers with different distros- right "out of the box", with dependencies. That is really useful. ... And I have only scratched the surface...

And yes; you can update Windows with it too. The salt-minion can be installed on Windows and be controlled from a Linux master. Windows cannot work as a master.
So if you have 10000 Windows boxes where you want to role out the same config - Linux is your friend.

[viking60]

Here are some examples of more or less practical use. Since you, no doubt, are administrating servers all over the world it might be interesting to know how the weather is (temperature in particular). I can run any command on the "minions" with cmd.run: Code: Select all sudo salt '*' cmd.run 'inxi -w' This will give me the weather on all locations if Inxi is installed on the server. And to install Inxi on all servers: Code: Select all sudo salt'*' pkg.install inxi This will role out Inxi on all "minions" and update it, if it is present. If one box has the up to date Inxi it will be skipped. You can run Windows programs and commands too: Code: Select all sudo salt 'windows-pc' cmd.run 'dir  c:\' And the weather? Well mostly cloudy and 32F or 0 C .....

If you want a lot of interesting info try this: Code: Select all sudo salt '*' network.interfaces It will give you the network interface info on all minions. If you want to only check all servers that begin with "cento": Code: Select all sudo salt 'cento*' network.interfaces If you want to check all your Centos servers you can use the Grain function: Code: Select all salt -G 'os:Centos' test.ping It will give you a list of computers with the Centos OS like this Code: Select all centos_server:     True Naturally you can check your Windows boxes too: Code: Select all sudo salt -G 'os:Windows' test.ping Salt has won prizes as technology of the year 2014 - I'll go along with that. So yes; I have been salted

[viking60]

So updating Linux boxes is easy with salt if you have one or 10000. Most infrastructures have both Linux and Windows so it is important to control the Windows computers too.

So I thought I should install Firefox on one of the Windows boxes.
To find all windows computers we remember that we can do a:

Code: Select all

sudo salt -G 'os:Windows' test.ping

Somehow we need to get a repository for the Windows programs that we want to role out.
So I edited /etc/salt/master

Code: Select all

#####     Windows Software Repo settings #####
##############################################
# Location of the repo on the master
win_repo: '/srv/salt/win/repo'

All I did was un-commenting the last line.
Then I created the directories salt win and repo (they are not auto-created by un-commenting in the master file).
in /srv/salt/win/repo i created the Firefox directory

Code: Select all

sudo mkdir firefox

Here I created the file init.sls with the folowing content:
/srv/salt/win/repo/firefox/init.sls

Firefox:
26.0:
installer: 'http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/26.0/win32/en-US/Firefox%20Setup%2026.0.exe'
full_name: 'Mozilla Firefox 26.0 (x86 en-US)'
reboot: False
install_flags: ' /s '
uninstaller: 'C:\Programfiler (x86)\Mozilla Firefox\uninstall\helper.exe'
uninstall_flags: ' /S'

The uninstaler line must be changed according to your language in valhalian it is C:\Programfiler in German it would be C:\Programme and in english it would be C:\Program FIles

You can copy this and use it - the version will be automatically updated if you go through the motions right. It does not matter if the version you put in there is old already.

I also put the same content in /srv/salt/win/repo/firefox.sls since I was a bit confused by the Salt documentation on this point - just for good messure.

Then I did a:

Code: Select all

sudo salt '*' pkg.refresh_db

This is necessary for Windows machines, and will pick up the latest version of Firefox (or whatever)
Then I ran

Code: Select all

sudo salt 'Windows-pc' pkg.list_pkgs

And it lists the Windows software:

Code: Select all

Windows-pc:
    ----------
    7-Zip 4.65:
        Not Found
    Adobe Flash Player 12 ActiveX:
        12.0.0.77
    Adobe Flash Player 12 Plugin:
        12.0.0.77
    Adobe Reader 9.5.5 - Norsk:
        9.5.5
    Advanced SystemCare 7:
        7.2.1
    Avira Free Antivirus:
        14.0.3.350
    Bluefish 2.2.1:
        2.2.1
    Cain & Abel v4.9.32:
etc...etc...
......


Then I did a:

Code: Select all

sudo salt-run winrepo.genrepo

and a

Code: Select all

sudo salt '*' pkg.refresh_db

As described here
Then i checke what Firefox packages were available:

Code: Select all

sudo salt '*' pkg.available_version Firefox   

and it showed:

Code: Select all

Windows-pc:
    26.0

And that was about it so time to install Firefox on the remote (minion) Windows-pc from my Linux master then:

Code: Select all

sudo salt 'Windows-pc' pkg.install Firefox

And oh wonder:

Code: Select all

Windows-pc:
    ----------
    Firefox:
        ----------
        new:
            26.0
        old:
           
    Mozilla Maintenance Service:
        ----------
        new:
            26.0
        old:
           


It works

Easier to just install Firefox on the box directly you say? Well yes on one box, but not on 10 or 10000 and making sure that it is the same version.
Here you can set up all the software you want on your windows computers once - and role them out as often as you like.

Great stuff - and it is fast.
.....
I wanted to change the Firfox version to Norwegian
So I changed
sudo nano /srv/salt/win/repo/firefox/init.sls

Firefox:
26.0:
installer: 'http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/26.0/win32/nb-NO/Firefox%20Setup%2026.0.exe'
full_name: 'Mozilla Firefox 26.0 (x86 nb-NO)'
reboot: False
install_flags: ' /s '
uninstaller: 'C:\Programfiler (x86)\Mozilla Firefox\uninstall\helper.exe'
uninstall_flags: ' /S'

Basically I simply replaced en-US with nb-NO
Then I regenerated the repo

Code: Select all

sudo salt-run winrepo.genrepo

And updated the databases.

Code: Select all

sudo salt '*' pkg.refresh_db

And repeated the install:

Code: Select all

sudo salt 'viking' pkg.install Firefox

And it went through just fine: The foreign en-US Firefox was replaced with the more global nb-NO
So you can make country specific versions - no problem.
After all this work I decided to remove firefox from the Windows-pc again wit

Code: Select all

sudo salt 'viking' pkg.purge Firefox

But that did not work.

Code: Select all

sudo salt 'viking' pkg.remove Firefox

did not work either and it does not matter if I omit firefox. This should be possible - so maybe I have found my first bug

[viking60]

So you thought you had to write those cryptic file one million times before you can install software on your Windows boxes?
No! It is done for you in Git!
All you have to do is hoking it up in your master file and make the Windows section there look like this:

Code: Select all

#####     Windows Software Repo settings #####
##############################################
# Location of the repo on the master
win_repo: '/srv/salt/win/repo'

# Location of the master's repo cache file
win_repo_mastercachefile: '/srv/salt/win/repo/winrepo.p'

# List of git repositories to include with the local repo
win_gitrepos:
 - 'https://github.com/saltstack/salt-winrepo.git'

All you need to do is to un-comment code that is already there about line 593 to 603.
Then you have to run these three commands:

Code: Select all

sudo salt-run winrepo.update_git_repos

Code: Select all

sudo salt-run winrepo.genrepo

Code: Select all

sudo salt '*' pkg.refresh_db

Now you can install Blender or anything in the Git repo
For non english systems you may want to edit the uninstall information (c:Programme instead of ProgramFiles etc)
The files to edit will be in /srv/salt/win/repo/salt-winrepo.git/

[viking60]

I just noted another big advantage with Salt:
You know when you run

Code: Select all

rkhunter -c

you have to hit ENTER several times before it finishes. If you start it and go for that free beer; rkhunter will not continue by itself.

So I ran the command on a remote box to check out what would happen - like this:

Code: Select all

sudo salt 'remote-box' cmd.run 'rkhunter -c'

And it finished everything beautifully

[viking60]

To run the salt commands from a "minion" you can replace "salt" with salt-call.
Example:
To get an overview of the disk use on all the servers (minions) from the master:

Code: Select all

sudo salt '*' disk.percent

But if you work on the minion itself you can check it like this:

Code: Select all

sudo salt-call disk.percent

But I do not have two computers to test this you say?
No problem; just install the minion on a distro in Virtualbox and the master on you computer.

After you have done that you can upgrade and remote control the "computer" in Virtualbox. (It is great fun to see how the remotely installed Firefox pops up on the Windows desktop in VB)

[viking60]

Slightly negative update here.
Latlely the salt master is using 100% cpu and has stopped responding.

When I do a

Code: Select all

salt-master -l debug

I get:

Code: Select all

[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: thomas-pc
[DEBUG   ] Configuration file path: /etc/salt/master
[INFO    ] Setting up the Salt Master
[WARNING ] Unable to bind socket, error: [Errno 98] Address already in use
The ports are not available to bind


But when I check the ports they are used by salt-master

When I try to run something on my salt minions I get:

Code: Select all

Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.


Yeah right
This has been a problem before

https://github.com/saltstack/salt/issues/22270

So at the moment this thing is pretty useless.

[viking60]

Positive update
Some updates later salt runs smoothly again and I can update and remote control my computers as I like.
I just did a

Code: Select all

sudo salt 'Viking_slave-server' service.get_all

To get a full overview over all the running services on a remote server.
It is important when you install say the Apache server remotely - this is called httpd in some distros (like Centos) and apache in others.
Salt uses the systemd service name.
Based on this you can make files (with simple instructions) that send the correct files to the different computer groups.

To find a specific service on the remote server you can combine with grep:

Code: Select all

sudo salt 'Viking_slave-server' service.get_all | grep httpd

Here is a nice intro to Salt:

[viking60]

This is how a session looks like (with all the human errors )

http://showterm.io/486a66ef8341cac260945#fast

[viking60]

I have had some problems with the Manjaro minions and pkg.upgrade I tried to get some input in the forum but I guess this software is a bit special.
So I reported a bug upstream:

https://github.com/saltstack/salt/issues/27832
And those Salt guys recognized it and fixed it in 3 hours. So I'll wait for my next Manjaro update ..and this should be fixed

[viking60]

Salt is gaining momentum and is now also used by the US department of defense

So it obviously can handle large IT infrastructures.

I know exactly what they said:
If that thing is good enough to handle the bjoernvold.com infrastructure, then we must have it!