Graylog Sidecar

Note

Graylog 3.0 comes with a new Sidecar implementation.We still support the old Collector Sidecars, which can be found in the System / Collectors (legacy)menu entry. We encourage users to migrate to the new Sidecar, which is covered by this document.

Graylog Sidecar is a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) act as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host). The log collector configurations are centrally managed through the Graylog web interface. Periodically, the Sidecar daemon will fetch all relevant configurations for the target, using the REST API. On its first run, or when a configuration change has been detected, Sidecar will generate (render) relevant backend configuration files. Then it will start, or restart, those reconfigured log collectors.

Installation

You can get .deb and .rpm packages for Graylog Sidecar in our package repository. For Windows, you can download the installer from here. Please follow the version matrix to pick the right package:

Sidecar version	Graylog server version
1.1.x	3.2.5 or higher
1.0.x	3.0 or higher
0.1.x	2.2.x, 2.3.x, 2.4.x, 2.5.x, 3.0.x
0.0.9	2.1.x

All following commands should be executed on the remote machine where you want to collect log data from.

Install the Sidecar

Ubuntu

Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:

$ wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.deb
$ sudo dpkg -i graylog-sidecar-repository_1-2_all.deb
$ sudo apt-get update && sudo apt-get install graylog-sidecar

Edit the configuration (see Configuration ) andactivate the Sidecar as a system service:

$ vi /etc/graylog/sidecar/sidecar.yml
$ sudo graylog-sidecar -service install[Ubuntu 14.04 with Upstart]
$ sudo start graylog-sidecar[Ubuntu 16.04 and later with Systemd]
$ sudo systemctl start graylog-sidecar

CentOS

Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-2.noarch.rpm
$ sudo yum install graylog-sidecar

Edit the configuration (see Configuration ) andactivate the Sidecar as a system service:

$ vi /etc/graylog/sidecar/sidecar.yml

$ sudo graylog-sidecar -service install
$ sudo systemctl start graylog-sidecar

Windows

Use the Windows installer, it can be run interactively:

$ graylog_sidecar_installer_1.0.0-1.exe

Or in silent mode with:

$ graylog_sidecar_installer_1.0.0-1.exe /S -SERVERURL=http://10.0.2.2:9000/api -APITOKEN=yourapitoken

Optionally edit the configuration (see Configuration ) and register the system service:

notepad.exe C:\Program Files\Graylog\sidecar\sidecar.yml 
& "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install 
& "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start 

Install collectors

Next up, you can decide which collectors you want to use with your Sidecar and installthem as well. We only cover the installation of the most common ones here, but you are free to use other collectors as well. Graylog contains default collector configurations for Filebeat, Winlogbeat and NXLog.But since you’re able to define your own collector backends, there is nothing stopping you from running e.g. sysmon, auditd or packetbeat.

Beats on Linux

Install Filebeat or another Beats package by following the instructions on the official Filebeat download page.

Beats on Windows

The Windows Sidecar package already includes Filebeat and Winlogbeat.For other Beats packages follow the instructions on the official Beats download page.

NXLog on Ubuntu

Install the NXLog package from the official NXLog download page.Because the Sidecar takes control of stopping and starting NXlog it isnecessary to stop all running instances of NXlog and unconfigure the default system service:

$ sudo /etc/init.d/nxlog stop
$ sudo update-rc.d -f nxlog remove
$ sudo gpasswd -a nxlog adm
$ sudo chown -R nxlog.nxlog /var/spool/nxlog

NXLog on CentOS

The same on a RedHat based system:

$ sudo service nxlog stop
$ sudo chkconfig --del nxlog
$ sudo gpasswd -a nxlog root
$ sudo chown -R nxlog.nxlog /var/spool/nxlog

NXlog on Windows

Install the NXLog package from the official download page and deactivate thesystem service. We just need the binaries installed on the system:

`$ C:\Program Files (x86)\nxlog\nxlog -u`

Sidecar Configuration

On the command line you can provide a path to the configuration file with the -cswitch.The default configuration path on Linux systems is /etc/graylog/sidecar/sidecar.ymland C:\Program Files\Graylog\sidecar\sidecar.ymlon Windows. Most configuration parameters come with built-in defaults.The only parameters that need adjustment are server_urland server_api_token.You can get your API token by following the link on the Sidecars Overview page.

sidecar.yml Reference

Parameter	Description
server_url	URL to the Graylog API, e.g. http://192.168.1.1:9000/api/
server_api_token	The API token to use to authenticate against the Graylog server API.

e.g 1jq26cssvc6rj4qac4bt9oeeh0p4vt5u5kal9jocl1g9mdi4og3n
The token is mandatory and needs to be configured. | |node_id |The node ID of the sidecar. This can be a path to a file or an ID string.
Example file path: file:/etc/graylog/sidecar/node-id
Example ID string: 6033137e-d56b-47fc-9762-cd699c11a5a9
ATTENTION: Every sidecar instance needs a unique ID!
Default: file:/etc/graylog/sidecar/node-id | |node_name |Name of the Sidecar instance, will also show up in the web interface.
The hostname will be used if not set. | |update_interval |The interval in seconds the sidecar will fetch new configurations from the Graylog server
Default: 10 | |tls_skip_verify |This configures if the sidecar should skip the verification of TLS connections. Default: false | |send_status |This controls the transmission of detailed sidecar information like collector status,
metrics and log file lists. It can be disabled to reduce load on the Graylog server if needed.
Default: true | |list_log_files |Send a directory listing to Graylog and display it on the host status page,
e.g. /var/log. This can also be a list of directories. Default: [] | |cache_path |The directory where the sidecar stores internal data. Default: /var/cache/graylog-sidecar | |collector_configuration_directory |The directory where the sidecar generates configurations for collectors.
Default: /var/lib/graylog-sidecar/generated | |log_path |The directory where the sidecar stores its logs. Default: /var/log/graylog-sidecar | |log_rotate_max_file_size |The maximum size of the log file before it gets rotated. Default: 10MiB | |log_rotate_keep_files |The maximum number of old log files to retain. | |collector_binaries_accesslist |A list of binaries which are allowed to be executed by the Sidecar.
An empty list disables the white list feature.
Default:/usr/bin/filebeat, /usr/bin/packetbeat, /usr/bin/metricbeat, /usr/bin/heartbeat,
/usr/bin/auditbeat, /usr/bin/journalbeat, /usr/share/filebeat/bin/filebeat,
/usr/share/packetbeat/bin/packetbeat, /usr/share/metricbeat/bin/metricbeat,
/usr/share/heartbeat/bin/heartbeat, /usr/share/auditbeat/bin/auditbeat,
/usr/share/journalbeat/bin/journalbeat, /usr/bin/nxlog, /opt/nxlog/bin/nxlog |

First start

Once you installed the Sidecar package and started the service for the first time,you can verify that it shows up in the Sidecars Overview page. A new sidecar instance will not have any configurations assigned yet. Take the Step-by-step guide to create your first configuration.

Mode of Operation

When the Sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into thecollector_configuration_directory directory for each collector backend. E.g. if you assigned a Filebeat collector you will find afilebeat.ymlfile in that directory. All changes have to be made in the Graylog web interface.Every time the Sidecar detects an update to its configuration it willrewrite the corresponding collector configuration file. Manually editing these files is not recommended. Every time a collector configuration file is changed the collector process is restarted. The Sidecar takes care of the collector processes and reports the status back to the web interface

Sidecar Status

Each Sidecar instance is able to send status information back to Graylog. By enabling the option send_status metrics like load or the IP address of the host Sidecar is running onare sent. Also metrics that are relevant for a stable operation e.g. disk volumes over 75% utilization are included. Additionally with the list_log_files option a directory listing is displayed inthe Graylog web interface. In that way an administrator can see which files are available for collecting. The list is periodically updated and files with write access are highlighted for easy identification.After enabling send_status or send_status+ list_log_filesgo to the collector overview and click on one of them, a status page with the configured information will be displayed.

Step-by-step guide

We have prepared an example on how to configure the Sidecar using the Graylog web interface. The assumption is that we want to collect Apachelogfiles and ship them with a Filebeat collector to a Beats input that is listening on Port 5044 on your Graylog Server.

• The first step is to create a Beats input where collectors can send data to. Click on System / Inputs and start a global Beats input on the listening address 0.0.0.0 and port 5044.

• Navigate to the Sidecars overview. In your Graylog web interface click on System / Sidecars.

• Navigate to the Sidecar Configurationpage.

• Next we create a new configuration: We give the configuration a name and select filebeat on Linux as collector.(This collector definition is shipped with Graylog, and comes with a default configuration template).Most of the configuration defaults should work for you. However you need to change the hosts: setting and point itto your Beats input. You also might want to change the paths:to the location of your Apache logs.When done click Createto save your configuration.

• Next we need to assign our newly created configuration (and therefore the Filebeat collector) to our sidecar.Go to the Collector Administrationpage.

• You will see a list of sidecars and underneath them a list of collectors that could be assigned to them.Please note that collectors are assigned to sidecars by means of applying a collector configuration to the sidecar.Therefore, we first select the filebeatcollector and then click on the Configuremenu, where wecan select the filebeat-confconfiguration we created earlier.

• Confirming the assignment, will directly push this configuration to your sidecar which will go and startthe Filebeat collector with this configuration.

• If everything went fine, you should see the status running on the administration page.

• Congratulations your collector setup is working now!You can go back to the Sidecars overview and click on the Show messagesbutton tosearch for logs that have been collected via your sidecar.

Creating a new Log Collector

Graylog comes with a few predefined log collectors which can be easily extendedand changed to your needs.Let’s assume you want your sidecar to run rsyslog(8) for you.

• Navigate to the Sidecars overview. In your Graylog web interface click on System / Sidecars.

• Navigate to the Sidecar Configurationpage.

• After we click on Create Log Collector, we are presented with the following page,where we have to fill out some fields for our new collector.We give the collector a unique name and select Linuxand Foreground Execution.Given that you installed rsyslogd(8) under /usr/sbin/rsyslogdwe configure theexecutable path accordingly.If you are using Foreground Executionmake sure that the collector you are runningdoes not daemonize itself. Otherwise the sidecar has no way of controlling the collectoronce it has forked off into the background.For rsyslogd we therefore provide -nas .If your collector supports configuration validation, it is advised to use it.This acts as a pre-check, so that sidecar won’t restart a collector witha broken configuration. For rsyslogd the option to do a configuration check is -N 1.Optionally you can provide a which will be proposedonce you create a configuration for this collector.

• Next up you can use your newly created collector by creating a configurationfor it and assign it to a Sidecar. Please follow the Step-by-step guide accordingly.

• Note : Your Sidecar might refuse to start your collector, because it needsto be added to the collector_binaries_accesslist first. Please edit yourConfiguration and restart your Sidecar.

Using Configuration Variables

Configuration variables can contain arbitrary strings likethe IP address of your Graylog server or the port of an input.The variables can then be used in multiple collector configurations,which avoids duplication and simplifies management. To create a configuration variable go any Collector Configuration page: On the right you’ll find a box Collector Configuration Referencewhichcontains and .Click on Variablesand then Create Variable to receive the followingmodal: In this example we replace the hard coded IP and Port from ourBeats input with a new variable named ${user.BeatsInput}: We can now use this variable in all our configurations.If we ever need to change the IP/port of our input,we just change the variable.

Runtime Variables

Runtime variables contain runtime informations from each Sidecar thatis requesting this configuration.An important example is the ${sidecar.nodeId} variable.The collector configuration should contain an instruction to fillthat variable in an extra field .This allows Graylog to relate messages to the Sidecar that producedthem. (This is what makes the Show messages button on the Sidecars overview page work)

Secure Sidecar Communication

The Communication between Sidecar and Graylog will be secured if your API uses SSL . To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input.Now in the Sidecar Beats Output Configuration you just mark Enable TLS Supportand Insecure TLS connection. After this is saved, the communication between Beats and Graylog will use TLS.

Certificate based client authentication

If you want Graylog to only accept data from authenticated Collectors please follow the steps at Secured Graylog and Beats input

Run Sidecar as non-root user

The default is that the Sidecar is started with the root user to allow access to all log files. But this is not mandatory. If you like to start it with a daemon user, proceed like the following:

• Create a daemon user e.g. sidecar

The Sidecar itself is accessing the following files and directories:

• sidecar.yml- /etc/graylog/sidecar/sidecar.yml
• collector_configuration_directory- /var/lib/graylog-sidecar/generated/
• node_id - /etc/graylog/sidecar/node-id
• cache_path - /var/cache/graylog-sidecar/
• log_path- /var/log/graylog-sidecar/

So to make these directories readable for the sidecaruser, use:

• chown -R sidecar /etc/graylog/sidecar
• chown -R sidecar /var/cache/graylog-sidecar
• chown -R sidecar /var/lib/graylog-sidecar
• chown -R sidecar /var/log/graylog-sidecar

You can change all paths to different places in the file system. If you prefer to store all Sidecar data in the home directory of the sidecaruser, just change the paths accordingly.

Now systemdneeds to know that the Sidecar should be started with a non-root user. Open /etc/systemd/system/collector-sidecar.servicewith an editor and navigate to the [Service]section, add:

User = sidecar Group = sidecar 

To make use of these settings reload systemd:

$ sudo systemctl daemon-reload
$ sudo systemctl restart graylog-sidecar

Check the log files in /var/log/graylog-sidecarfor any errors. Understand that not only the Sidecar but also all backends, like filebeat, will be started as sidecaruser after these changes.So all log files that the backend should observe also need to be readable by the sidecaruser. Depending on the Linux distribution there is usually an administrator group which has access to most log files.By adding the sidecaruser to that group you can grant access fairly easy. For example on Debian/Ubuntu systems this group is called adm (see System Groups in Debian Wiki or Security/Privileges - Monitor system logs in Ubuntu wiki).

Upgrading from the Collector Sidecar

This guide describes how you can perform an upgrade from the deprecatedCollector Sidecars (0.1.x) to the new Sidecars (1.x). One major difference between the old and the new Sidecars, is thatwe replaced the UI based collector configuration approach withone where you can manage the plain text configuration of the collectors directly.This might seem like an inconvenience at first, butgives you the flexibility to configure any collector backend you want. Additionally, the new Sidecars don’t assign configurations based on tags anymore.Instead you have to assign configurations explicitly (see Step-by-Step guide).

1. Install New Sidecar

The new Sidecar has different paths and executable names, so it can coexist with the old one.Install the new Sidecar by following the Installation instructions and have your Sidecar running as described in First Start.

Note:

In case you were using filebeat on Linux, please make sure to also installthe official collector package, since the filebeat binary is not part of the Sidecar package anymore.

2. Migrate configuration

Next, we need to migrate the configuration that was previously renderedon each host by the Collector Sidecar, to a new Collector Configuration. We recommend to use the Sidecar Configuration Migrator .However, retrieving the old configuration can also be done manually by fetching it fromyour host at the /etc/graylog/collector-sidecar/generated/directory.

3. Adopt configuration to Graylog 3.0

There are a few things that might need attention after an upgrade:

• Use Runtime variables for static fields The imported configuration contains instructions that add static fieldswhich allows Graylog to relate messages to a Sidecar.You should replace the hardcoded values of gl2_source_collectorandcollector_node_idwith runtime variables.

In case of a Beats collector this would be:

fields.gl2_source_collector: ${sidecar.nodeId}
fields.collector_node_id: ${sidecar.nodeName}

• Migrate to the new Beats input Graylog 3.0 comes with a new Beats input. The former one was renamedto Beats(deprecated).The new input handles fields a little different. Therefore youshould define fields_under_root: truefor the new inputto get the Graylog fields work.

4. Switch over to the new Sidecar

Once you’re done creating a new configuration, you can assignit to your Sidecar (see Step-by-Step guide).If everything works as expected, make sure to uninstall the oldCollector Sidecar to avoid collecting your logs twice.

Sidecar Configuration Migrator

The task of the Sidecar configuration migrator is to extract the configurationfrom existing Collector Sidecars and convert it into new Sidecar configurations.

This feature needs a Collector Sidecar with version 0.1.8 or greater.Please upgrade the instance you want to import configurations from, if necessary.

• Navigate to the Collectors (legacy) overview. In your Graylog web interface click on System / Collectors (legacy).

• Click on the name of the Collector you want to import configurations from

• Click the Import Configurationbutton on a backend to import a configuration.If the import was successful, follow the link to create a new Sidecar configuration:

• After clicking on Create Configurationuse the Migratebuttonunderneath the configuration editor:

• A window opens up and lets you pick already imported configurations.Clicking Apply will paste the configuration into the editor.Afterwards you can edit and save the configuration as usual.

Sidecar Glossary

To understand the different parts of the Graylog Sidecar they are explained in the following section.

Configuration

A configuration is the representation of a log collector configuration file in the Graylog web interface.A configuration can be assigned to Sidecars, which also assigns the corresponding collector.You can have multiple configurations for a single log collector. However, you can notassign the same collector twice to a Sidecar.

Inputs

Inputs are the way how collectors ingest data. An input can be a log file that the collector should continuously read or a connection to the Windows event system that emits log events.An input is connected to an output, otherwise there would be no way of sending the data to the next hop. So first create an output and then associate one or many inputs with it.

Debug

The Sidecar is writing log files to the directory configured in log_path. One file for each backend, there you can check for general issues likefile permissions or log transmission problems. The Sidecar itself is writing to sidecar.log. Problems like failed connection to the Graylog API can be found there.

You can also start the Sidecar in foreground and monitor the output of the process:

$ graylog-sidecar -debug

Uninstall

On Linux just uninstall the package, to perform an uninstall on Windows run:

& "C:\Program Files\Graylog\graylog-sidecar.exe" - service stop 
& "C:\Program Files\Graylog\graylog-sidecar.exe" - service uninstall

Known Problems

Currently we know of two problems with NXLog:

• Since version 2.9.17 timestamps are transmitted without millisecond precision
• On Windows machines NXlog is not able to store its collector state so features like file tailing don’t work correctly in combination with Sidecar. Use Sidecar version 0.1.0-alpha.1 or newer.

Known issue if you use a loadbalancer or firewall in front of Graylog’s API:

• The Sidecar is using a persistent connection for API requests. Therefore it logs 408 Request Time-out if the loadbalancer session or http timeout is lower than the configured update_interval.