Roberto Carratalá

Principal AI Architect - AI BU @ Red Hat. MsC in AI/ML & MsC Telco Engineer

3 minute read

This blog post aims to provide a guide for collecting a Sosreport on a node of an OpenShift 4 Cluster.

Because OpenShift 4 uses RHCOS as the operating system for its nodes, the procedure for collecting a sosreport for further analysis has changed a bit compared to OpenShift 3/RHEL 7.x. This procedure is also valid for OpenShift worker nodes based on RHEL 8, instead of RHCOS.

Access to the OpenShift node

By default, the OpenShift 4 installer creates a single user named core with optional SSH keys specified at install time.

In our case, the SSH key generated and injected into the cluster at install time can be used, but additional SSH keys can be added to the OCP nodes following the procedure in Updating SSH Keys with the MCD.

  • SSH to the OpenShift cluster specific node
$ ssh -i ~/.ssh/bastion ec2-user@bastion.dev.ocp4.example.com

$ ssh -i ~/.ssh/ocp4.pem core@ip-10-148-204-82.eu-central-1.compute.internal
Red Hat Enterprise Linux CoreOS 410.8.20190718.1
WARNING: Direct SSH access to machines is not recommended.

$ sudo -i

Obtain the support-tools container in the node

  • Using podman, log in with your RH credentials on the node:
# podman login registry.redhat.io
Authenticating with existing credentials...
Existing credentials are invalid, please enter valid username and password
Username: rcarrata@redhat.com
Password:
Login Succeeded!

To collect the sosreport on the node, a specific container can be used: support-tools. The Red Hat Enterprise Linux Support Tools image contains tools to analyze the host system including sosreport, strace, and tcpdump.

  • Pull the support-tools container to the specific node (in our case a Master node):
# podman pull registry.redhat.io/rhel8/support-tools
Trying to pull registry.redhat.io/rhel8/support-tools...Getting image source signatures
Copying blob e61d8721e62e: 0 B / 67.75 MiB [-----------------------------------]
Copying blob e61d8721e62e: 9.35 MiB / 67.75 MiB [====>-------------------------]
Copying blob e61d8721e62e: 67.21 MiB / 67.75 MiB [=============================]
Copying blob e61d8721e62e: 67.75 MiB / 67.75 MiB [==========================] 3s
Copying blob c585fd5093c6: 1.47 KiB / 1.47 KiB [============================] 3s
Copying blob 77392c39ffcb: 8.67 MiB / 8.67 MiB [============================] 3s
Copying config 23a6cff4874d: 4.36 KiB / 4.36 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
23a6cff4874d03f84c7a787557b693afd58a1fb1f1123d5c9d254f785771c8fa
  • Execute the container with specific runlabel:
# podman container runlabel RUN registry.redhat.io/rhel8/support-tools
Command: /proc/self/exe run -it --name support-tools --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=support-tools -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest

Execute the sosreport to collect data

  • Once inside the support-tools container, run the sosreport command to collect the data:
bash-4.4# sosreport

sosreport (version 3.6)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/host/var/tmp/sos.x55wf_r3 and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: xxxxxxxx

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 6/78  cgroups         [Running: auditd block boot cgroups]                     caught exception in plugin method "cgroups.collect()"
writing traceback to sos_logs/cgroups-plugin-errors.txt
  Starting 18/78 grub2           [Running: cgroups chrony crio dracut grub2]              caught exception in plugin method "grub2.collect()"
writing traceback to sos_logs/grub2-plugin-errors.txt
  Starting 69/78 system          [Running: cgroups chrony grub2 logs selinux system]      [plugin:system] _copy_dir: Too many levels of symbolic links copying '/host/proc/sys/fs/binfmt_misc'
  Starting 70/78 systemd         [Running: cgroups chrony grub2 logs selinux systemd]     caught exception in plugin method "systemd.collect()"
writing traceback to sos_logs/systemd-plugin-errors.txt
  Finishing plugins              [Running: cgroups grub2 systemd]                         kaged]
Creating compressed archive...

Your sosreport has been generated and saved in:
  /host/var/tmp/sosreport-ip-10-148-204-82-xxxxxxxx-2019-08-08-jnsdcyp.tar.xz

The checksum is: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Please send this file to your support representative.
  • The sosreport file is located in the /host/var/tmp/ folder inside the container:

bash-4.4# ls -lhrt /host/var/tmp/sosreport-2019-08-08-jnsdcyp.tar.xz
-rw-------. 1 root root 11M Aug  8 10:00 /host/var/tmp/sosreport-ip-10-148-204-82-02444698-2019-08-08-jnsdcyp.tar.xz
  • Outside the container (due to the SCC under which the container was executed), the sosreport is located in the /var/tmp/ folder of our RHCOS node:
# ll /var/tmp/sosreport-2019-08-08-jnsdcyp.tar.xz
-rw-------. 1 root root 11233928 Aug  8 10:00 /var/tmp/sosreport-2019-08-08-jnsdcyp.tar.xz
  • Just scp/rsync it to any location, and after that it can be uploaded or analysed to obtain more information.

NOTE: Opinions expressed in this blog are my own and do not necessarily reflect that of the company I work for.

Happy OpenShifting!

You may also enjoy