Debugging systemd Using systemctl

To debug failed services using systemctl:

List failed services in the system:
Terminal window
```
sudo systemctl --failed
```
You will get a list of failed services. To see a list of all the services in the system, use the command:
Terminal window
```
sudo systemctl list-unit-files --type=service
```
Check status of the failed service for more detailed information:
Terminal window
```
sudo systemctl status <service_name>.service
```
See the service logs for more insight:
Terminal window
```
sudo journalctl -b -u <service_name>.service
```
You can further increase the log level to get debug level information:
Terminal window
```
sudo systemctl log-level debug
```
Reload the systemd daemon and restart a service:
Terminal window
```
sudo systemctl daemon-reload
sudo systemctl restart  <service_name>.service
```
Now you can see debug level information in the service log.
You can also attach strace with the service daemon to see system call and signal status:
- Get the PID of the main process from service status. It is listed as Main PID:.
- Attach strace with the PID:
  Terminal window
```
sudo strace -f -s 100 -p <Main_PID>
```
Retune the service configuration in runtime:
Terminal window
```
systemctl edit --runtime <service_name>.service
```
- Uncomment the [Service] section and the configuration you want to enable or disable. You can add any new configuration. This basically overrides your base configuration.
- Save the configuration as /run/systemd/system/<service_name>.d/override.conf.
- Reload the systemd daemon and restart the service as mentioned in step 4.
- You can check if your service is using the new configuration with the command:
  Terminal window
```
sudo systemctl show <service_name>.service
```
- For checking the base configuration:
  Terminal window
```
sudo systemctl cat <service_name>.service
```
If the new configuration works for you, you can check the exposure level of the service using the command:
Terminal window
```
systemd-analyze security
systemd-analyze security <service_name>.service #For detailed information
```
Update the configuration in the Ghaf repository and build it. Hardened service configs are available in the directory ghaf/modules/common/systemd/hardened-configs.