Category Archives: Linux

MySQL/MariaDB, logrotate, and SELinux

Assumption: You have SELinux Enforcing on your database server. If you’re still solving problems by permanently setting SELinux to Permissive, I don’t think you can really call yourself an IT professional.

Here are the commands to set the SELinux context of all text MySQL log files to var_log_t so that they can be rotated by logrotate:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/.*\.log(-[0-9]+(\.gz)*)*)+"
restorecon -R -v -F /var/lib/mysql

NOTE: This expression will also match the file /var/lib/mysql/tc.log and change its context, which will prevent MySQL from starting. If your version of MySQL uses tc.log, here is one possibility:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/db-server-01\.log(-[0-9]+(\.gz)*)*)+"

This expression is server-specific, because MySQL uses the hostname to name the log file.

Continue reading

Synology DiskStation DS1515+ Review

Summary: The Synology DS 1515+ is a capable little NAS with a large feature set, but it has some software reliability issues and limited technical support.

Hardware

The DS1515+ is a compact unit that feels sturdy. It holds five 2.5″ or 3.5″ drives, and 3.5″ drives can be installed without any tools. You will need a Phillips-head screwdriver to install an additional RAM module, but that’s also a very simple process. The fans are also easily replaceable.

Configuration

You will need to connect the DiskStation to the Internet to install the operating system, which is called DiskStation Manager (DSM). The installation process went smoothly. From a security standpoint, this process is slightly scary, and I hope that the automated installation process used some method to verify the integrity of the operating system.

Once the operating system is installed, there is a nice graphical user interface to manage all aspects of the system. It is possible to enable SSH to allow access to the unit by command line, but the CLI is officially undocumented, and Synology support won’t answer questions about it. The documentation on the Synology support site is pretty thorough, so I won’t go through the process to create a RAID or volumes.

Continue reading

anacron run-parts generates invalid or malformed syslog messages

On RedHat and CentOS 6 and 7, anacron generates syslog messages that are mangled when they are forwarded by rsyslog. I found the cause and a solution in a comment by Tomas Heinrich on this old Fedora bug. Sadly, that bug was closed instead of getting fixed, probably because Fedora is oriented toward desktop users who are not generally forwarding syslog messages! Syslog uses the following default template for messages:

"< %PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"

The syslogtag field is truncated to 32 characters, but cronie-anacron includes a lot of data in its tag. The tag gets truncated to 32 characters, resulting in invalid syslog. If you try to send them to Logstash, you’ll get errors like this:

{"message":"<77>Nov 4 19:01:01 my-web-03 run-parts(/etc/cron.hourly)[1858 starting 0anacron","@version":"1","@timestamp":"2016-11-05T00:01:01.146Z","port":55456,"type":"syslog","tags":["_grokparsefailure"],"host_ip":"192.168.17.23"}

Continue reading

Making Windows work for Linux and OS X admins

If you are a Linux or OS X power user, then you’re used to having all the necessary tools built into your OS. When you log into a Windows system (What! No command line?) you may feel lost. These tools and shortcuts will help you be productive on Windows systems.

Windows Shortcuts

  • Alt-x is a magic shortcut key on Windows 8 and Server 2012. It pops up a little menu in the lower right corner of the screen which contains just the items that an admin needs. Try it!

Continue reading

Troubleshooting SSSD, realm, kerberos, and SSH

SSSD (System Security Services Daemon) allows Linux systems (specifically, Red Hat, CentOS, and Fedora) to verify identity and authenticate against remote resources. If you have a CentOS or Red Hat enterprise system, and you need to authenticate against a domain controller such as FreeIPA or Active Directory, SSSD is the way to go. I use SSSD on CentOS 7 systems, but it is now available on CentOS 6 as well. A few years ago, adclient (an open-source project from Centrify) was your only option to make a CentOS 6 server authenticate against Active Directory. adclient seems to have reached end of life, so SSSD is definitely the path forward.

I won’t repeat the procedure for using Active Directory as an identity provider on a Red Hat 7 system. Instead, I want to provide a few troubleshooting tips, since limited information is available on SSSD and related tools.
Continue reading

LVM device-mapper: reload ioctl failed: Invalid argument

LVM2 (Logical Volume Management) is pretty amazing, but when something goes wrong, it’s not easy to troubleshoot. This is not the fault of the tools, but a reflection that LVM is relatively new in Linux, and not widely understood.

What I Tried to Do

I tried to increase the size of a logical volume with the lvextend command:

lvextend --extents 100%FREE /dev/VolumeGroup1/var

This form of the command is supposed to use all of the free space in the volume group.

What Failed

The command responded with an error message that contained this text:

device-mapper: reload ioctl failed: Invalid argument

Running lvdisplay showed the following status for the volume:

LV Status    suspended

Continue reading

Monitoring with SNMP, Part 3: Automate active monitoring with Nagios

My last post showed how to monitor networked devices with SNMP. You could try to remember to manually check the status of things periodically, but that would be missing the point of computers. Instead, automate your monitoring with Nagios, a web-based monitoring tool for Linux that automates the process of actively querying devices and doing something with the information. Nagios is available as free open source software (Nagios Core), and the company offers additional non-free products with premium features. The open-source version is fine for getting started and setting up basic monitoring. Nagios does a lot more than just SNMP monitoring. I’ll refer you to the Nagios Core documentation to get Nagios up and running, and I’ll focus on how to set up Nagios to actively monitor devices with SNMP.
Continue reading

Monitoring with SNMP, Part 2: Command-line tools for active SNMP

In Part 1, I summarized the basic concepts of SNMP and defined the terms and acronyms used in this post. Now, I will show how to use SNMP to monitor actual devices. As an example, I will monitor an enterprise-grade uninterruptible power supply (UPS) and power distribution unit (PDUs) from Tripp-Lite. These devices have an SNMPWEBCARD installed to support communication over Ethernet.

Command-line tools for SNMP communication should be available for any Linux distribution (or any other UNIX-derived OS). Documentation for the basic SNMP tools is available online. The challenge with SNMP is figuring out what parameters are supported by a particular device. Most devices support a set of standard OIDs that return basic information such as device name, uptime, etc.
Continue reading

Monitoring with SNMP, Part 1: Fundamentals of SNMP

SNMP is a protocol for conveying information and controlling devices over a network. SNMP can be used in two ways:

  • Active: a device sends a command to set a parameter or request information for another device
  • Passive: a device sends an alert (called a trap) to another device, which is configured to receive traps and do something with the information.

The “payload” of an SNMP message is called an Object Identifier, or OID. An OID is an ordered list of non-negative numbers, such as:

1.3.6.1.2.1.1.3.0

The sequence is hierarchical, starting with the highest-level object and progressing to lower-level objects. The above sequence corresponds to:

iso(1) org(3) dod(6) internet(1) mgmt(2) mib-2(1) system(1) sysUpTime(3) 0

When this command is sent to a device, it will return the uptime of the device.

The translation between the numerical sequence and the human-readable form is stored in a text file called a Management Information Base, or MIB. The format of the MIB is defined in RFC 2578. Some MIB files are standard and contain object IDs that are recognized by almost all devices. Device manufacturers also provide custom MIB files in which they define specialized object IDs for a particular device. Unfortunately, some devices don’t have MIB files, and you will have to query the device to see what objects it supports and decipher what they mean.

In Part 2 of this series, I will use active SNMP to monitor infrastructure.

Running network experiments on the GENI project

The GENI Project is a networking testbed that is used by researchers studying novel networking technologies. While the technology is fascinating, the web site is, unfortunately, a confusing mess. Here are some pointers to get you started (or refresh your memory). This post will be updated as I learn more.

Key GENI Links

GENI Portal

This is where you log into the GENI Project. Your institution must have Shibboleth enabled and be part of the InCommon Federation. Click on the “Use GENI” button, enter the name of your institution into the search box, and you will be redirected to your institution’s login page.

GENI Documention

Tutorials, How-Tos, and other documentation can be found at the GENI Experimenter Page.
Continue reading