F5 BigIP Troubleshooting

From RZWiki

This article suggests a number of troubleshooting 'targets' that may help you diagnose issues with a particular device and isolate faults. The technical sections of this article are roughly sorted in OSI Model ascending order, starting at layer 1, physical. Depending on the type of issue you're facing, you may also find our Network Troubleshooting and Web Applications Troubleshooting articles useful.

Timeline/Action Log

Whenever troubleshooting, always keep a rough log of your actions, any device changes you make and the time of these and any other events. These will be invaluable when later explaining the resolution, justifying your actions and detailing the sequence of events. If you are making changes, take a look at our Device Change Masterclass for information on the best practise to follow.

Known Issues

Visit http://ask.f5.com and find out if you are suffering from a known issue for the version of software you are running. There are always quite a few.

Memory

noinclude/noinclude

v43.x

Use the bigpipe ms command to display detailed and summary statistics for memory usage.

v9.x

Use the bigpipe memory command to check memory usage for TMM and the Linux Host environment;

• The Linux Host environment rarely has much free memory available, this is normal. Use the free command to ascertain more about the memory usage of the Linux environment.

• If the bigpipe memory command does not display the memory split, (some old versions don't,) use the bigpipe global command instead. You can also use the switchboot command with the -d option to display how much memory is assigned to the linux environment, look for the mem= parameter.

See also the Memory Usage section of Persistence, F5 BigIP

CPU

v4.x

Use the cpu bigip command to display CPU usage statistics. Note that a number of services, including the SSH proxy service run under the Unix user process, not under the BIGIP process.

v9.x

Unfortunately real CPU usage is not easy to ascertain. Ignore output from the linux top or ps commands as the TMM process will utilise up to 100% of the CPU even when idle.

• With little or no traffic flow, TMM will release up to 99% of CPU time if other processes need it.
• Under heavy load TMM will only release up to 20% of CPU time if other processes need it.

Use the bigpipe global command command to ascertain how much CPU time is actually being used by TMM. Dependent on version, the CPU or TMM Cycles rows should be analysed. Idle cycles are used by TMM, sleep cycles are used by the linux environment.

Disk

Use the df and du commands to check overall file system usage and which directories are using what within a particular file system

Network Interfaces

font color=redCommon Issue:/font The speed/duplex settings of load balancer interfaces are frequently 'lost' after a reboot or service restart, this is always worth checking

font color=redCommon Issue:/font The enabled or disabled state of load balancer interfaces are frequently 'lost' after a reboot or service restart, this is always worth checking

Check interface operational status from a Linux perspective using the ifconfig command

Check interface statistics and erorrs from a Linux perspective using the netstat command as follows; netstat -i

Check all of the above from an LTM perspective using the bigpipe interface show all command

Note: The MTU of the tmm0 interface is normally: 1566

Spanning Tree Protocol

Connections

Use the bigtop command to check inbound connection levels.

For platforms running v4.x software, ensure concurrent connections do not or have not exceeded the concurrent connection limits, using the bigpipe summary command and analysing the max conn value.

NAT

Services Daemons

v9.x

Use the bigstart status command to confirm the status of the device's services. Use the following information on the function of each service to determine if an unavailable service is the cause of your issue, essential services are highlighted bold;

• alertd - Monitors error messages and triggers proper action including SNMP traps and front panel messages
• bcm56xxd - Controls the BIG-IP switch hardware
• bigd - Controls health and performance monitoring
• bigdbd - Provides initial bigdb database values to the MCPD service and persists any database changes to the BigDB.dat file
• chmand - Provides chassis monitoring and configuration, as well as other related functions
• crond - Runs scheduled commands
• cssd - Performs configuration synchronization for redundant systems
• fpdd - Handles front-panel display functions
• httpd - Provides HTTP web server functions
• lacpd - Creates trunks based on the industry-standard Link Aggregation Control Protocol (LACP) and controls the Switchboard Fail-safe feature for redundant systems
• MCPD - The Master Control Program Daemon (aka Master Configuration Process Daemon,) controls messaging and configuration
• snmpd - Provides System Network Management Protocol (SNMP) functions. Also includes the two subagents rmondsnmpd and tmsnmpd
• SOD - Controls failover for redundant systems
• sshd - Provides remote access to the BIG-IP system command line interface (CLI)
• stpd - Implements the IEEE spanning tree protocols for preventing bridging loops
• syslogd - Performs system logging based on the syslog-ng utility
• tamd - Provides remote authentication and authorization.
• TMM - Known as the Traffic Management Microkernel, manages switch traffic

If you have an advanced routing module installed;

• NSM - Dymanic routing daemon
• tmrouted - Propagates dynamic routes in the Linux kernel routing table (installed by the NSM daemon) to the TMM

Use the bigpipe daemon command to confirm the status of the device daemons.

To restart a specific service, use this command syntax at the CLI: bigstart restart service_name

To restart all services, use this command syntax at the CLI: bigstart restart

Network Protocols

Check LTM ICMP statistics and errors with the bigpipe icmp command

Check LTM UDP statistics and errors with the bigpipe udp command

Check LTM TCP statistics and errors with the bigpipe tcp command

Check LTM SSL statistics and errors with the bigpipe ssl command

SSL

An SSL cipher string containing the @strength parameter will result in older browsers, including Internet Explorer 6, failing to display content, even though it is sent by the load balancer.

When modifying parent profile settings, changes are not always cascaded to any child profiles. If you are relying on a change to a parent profile to be cascaded to a child, check the child profile carefully to ensure this has happened.

Configsync

See the Config Sync Troubleshooting, F5 BigIP article for further information

MAC Masquerade

font color=redCommon Issue:/font If you've just configured the MAC Masquerade feature, it normally won't work until each load balancer has it's services restarted with the bigstart restart all command

If switch port security features such as the Cisco port security feature are used in your environment, ensure the switch ports connected to both the primary and secondary load balancer permit/allow the use of the MAC masquerade address(es).

Failover

noinclude/noinclude

If a failover has occurred but no traffic appears to be reaching the newly ACTIVE device, this is likely due to nearby layer 3 devices not updating their ARP table entries with the new MAC address of the floating IP(s) configured on the load balancers. The MAC address changes as by default the MAC address of the floating IP is the MAC address of the first interface placed in the VLAN each floating IP resides in. These MAC addresses are different on each load balancer and thus floating IP MAC addresses change when a failover occurs.

ARP Table entry timeouts on most devices are quite long. To avoid this issue, the newly ACTIVE load balancer sends a gratuitous ARP to update the ARP table entry's on nearby layer 3 devices but firewalls generally don't accept these (as a fake gratuitous ARP could 'blackhole' traffic.) Some higher-end Cisco switches can also be configured to drop gratuitous ARPs (using the ip gratuitous-arps command; the AutoSecure feature also configures this command.)

If you wish to avoid this issue, use the MAC masquerade feature on the load balancers. This creates a 'shared' MAC address to be used with the floating IP address and negates the need for gratuitous ARPs as the MAC address associated with the floating IP address does not change.

If switch port security features such as the Cisco port security feature are used in your environment, ensure the switch ports connected to both the primary and secondary load balancer permit/allow the use of the MAC masquerade address(es).

See the Failover, F5 BigIP article for further information

Switch Security Features

If you are using the MAC masquerade feature on your load balancers in conjunction with switch port security features such as the Cisco port security feature, ensure the switch ports connected to both the primary and secondary load balancer permit/allow the use of the MAC masquerade address(es).

If you are not using the MAC masquerade feature on your load balancers but you are using switch security features that restrict ARP changes or ignore gratuitous ARPs, you may suffer from traffic loss when load balancer failover occurs (see above for reasons why.) Consider implementing the MAC masquerade feature to avoid this.

Passwords

v9.x

User account passwords other than that for the root user should be changed with the f5passwd command rather than the passwd command.

See the f5passwd Troubleshooting article for issues related to using this command, however, f5passwd relies on passwd so refer to the passwd Troubleshooting article as well.

If all user access to the device has been lost, see the F5 BigIP Password Recovery article.

Information on initial/default passwords can be found in this article: F5 Networks Default Passwords.

Configuration

The bigpipe config command can be used to create a backup of the current LTM configuration.

The bigpipe base save command can be used to create a backup of the current base configuration (management interface etc.)

The bigpipe export command can be used to create a combined flat text file of the base and LTM configuration.

The snapshot command could be used to create a backup of the entire system image.

If necessary, use the Sys-reset Script to return the device to it's factory default configuration. (v9.x only.)

Web Management Interface

If you're having issues of any kind with the web-based management interface, you may wish to restart the httpd and tomcat4 daemons that serve the management content. Enter these commands at the CLI: bigstart httpd restart and bigstart tomcat4 restart.

If the web-based management interface produces errors that appear to be memory related, particularly when viewing performance graphs, you may want to consider assigning extra memory to the Tomcat daemon/service. To do so, use this command,specifying a suitable amount of memory (20Mb is used in this example): bigpipe db Provision.Tomcat.extraMB 20. Note that specifying too much memory may cause management interface issues, even at the CLI.

Logs

Default maximum available log disk space is 7Gb, use the resize-logFS command to increase this size, up to a maximum of 10Gb.

See our Logging, F5 BigIP article for further log file information

SNMP

SNMP trap/inform traffic does not use the management interface (eth0) by default. To ensure this interface is used, configure static route(s) for your trap destination hosts.

Use the bigstart status snmpd command to check the status of the snmp daemon.

The SNMP Daemon uses this log file: /var/log/daemon.log

Virtual Servers

...

You'll also find the TCP/IP behaviour information in our Virtual Servers In Detail, F5 BigIP article useful.

We really do appreciate all feedback so please do send your comments, suggestions or corrections to sjiveson#routerzone.eu (replacing the # with an @)