Cisco WLC or AP device certificate expired.

In network administration, mainly when dealing with older Cisco Wireless LAN Controllers (WLCs) and Access Points (APs), an issue that may arise is the sudden disappearance of APs from the WLC’s management interface. This problem can often be traced to matters related to the digital certificates used for secure authentication between the WLC and the APs.

Understanding Device Certificates in Cisco WLCs and APs:

1. Manufacturing Process: Cisco embeds a unique device certificate into each WLC and AP during manufacturing. This certificate is a crucial component for secure communication between the devices.
2. Role in Authentication: The embedded certificate is utilized to authenticate the WLC and any AP attempting to join the network. This authentication process is essential for establishing a secure Datagram Transport Layer Security (DTLS) tunnel, which is used to encrypt Control And Provisioning of Wireless Access Points (CAPWAP) control traffic.
3. Impact on Connectivity: Without successful mutual authentication, the WLC and AP cannot establish a secure DTLS tunnel. As a result, the AP will be unable to join the WLC, leading to the disappearance of the AP from the WLC management interface.
4. Certificate Validity Period: The device certificates installed in WLCs and APs typically have a validity period of 10 years from the date of manufacture.
5. Time Synchronization: The certificates’ validity is closely tied to the time and date settings of the WLC and the connecting APs. The WLC and APs must synchronize time settings, as the APs derive their time from the WLC during the initial connection attempt. Even if the CAPWAP/DTLS connection is not successfully established, the AP will still receive the time from the WLC.
6. When managing older Cisco WLCs and APs, knowing the potential certificate-related issues that can affect connectivity is crucial. Ensuring the device certificates are valid and the time settings are synchronized between the WLC and APs can help prevent unexpected disconnections and maintain a stable wireless network.

The Problem

I myself have run into two problems related to device certificates:

• Older access point does not want to join any WLC
• Newer access point does not want to join an older WLC

The first issue shows up when you have an old access point that has hit that 10-year mark and its device certificate has finally hit the expiration date. At the time of writing this article, these access points are usually of the models AP-1131 and AP-1142. Your AP will not be disconnected from the WLC immediately on the certificate’s expiration date but in case of a restart of either the AP or the WLC where a new CAPWAP-tunnel must to established between the two, the connection will not be completed.

Connecting to the console port of an AP is the easiest method to see this problem in action and it looks something like this. 10.10.66.250 is the IP address of the WLC.

*Sep 13 18:26:24.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.10.66.250 peer_port: 5246

*Sep 13 18:26:24.000: %CAPWAP-5-CHANGED: CAPWAP changed state to

*Sep 13 18:26:24.099: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.  

The certificate (SN: 3C1E27950000000CAAAA) has expired.    Validity period ended on 19:56:24 UTC Sep 12 2019

*Sep 13 18:26:24.099: %LWAPP-3-CLIENTERRORLOG: Peer certificate verification failed

*Sep 13 18:26:24.099: %CAPWAP-3-ERRORLOG: Certificate verification failed!

The error message is pretty clear, the AP itself is alerting us that its certificate has expired.

The second issue presents itself when you have a newer access point that is trying to connect to an older WLC, whose own device certificate has reached 10 years of age and therefore expired. At the time of writing this article, the earliest manufactured models of the WLC 5508 controller have been starting to run into this issue as of May 2018.

Connecting to the console port of an AP also shows this problem, the error message could look a bit different depending on if you using an AP running the older IOS software (pre-1800/2800/3800 series APs) or newer AP-COS software (1800/2800/3800 and forward).

For old IOS-based access points the error could look like this:

[*09/09/2019 12:34:50.0099] Cert Verification FAILED with error 10 (certificate has expired) at 0 depth...

[*09/09/2019 12:34:50.0099] [*09/09/2019 12:34:50.0099] /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AIR-CT5508-K9-00211bfeAAAA/[email protected]

For new AP-COS-based access points could look like this:

[*09/09/2019 04:55:26.3299] CAPWAP State: DTLS Teardown

[*09/09/2019 04:55:30.9385] CAPWAP State: Discovery

[*09/09/2019 04:55:30.9385] Did not get log server settings from DHCP.

[*09/09/2019 04:55:41.0000] CAPWAP State: DTLS Setup

[*09/09/2019 04:55:41.3399] Bad certificate alert received from peer.

[*09/09/2019 04:55:41.3399] DTLS: Received packet caused DTLS to close connection

The Solution

If the certificate of one or more of your APs has expired you got two options. You can either:

1) Turn off NTP and manually set the clock of your WLC to a time and date where the certificates are still valid. The access point will also get their clock updated when they try to join the WLC so there is no need to set the time directly on the access point.

If you are currently using NTP to set the time of the WLC, use the command show time to see which NTP-servers you are using at the moment. All of your NTP servers are assigned an index number (like 1, 2, 3, and so on) and you need to refer to that index number to delete the NTP-server from being used. In my case, I only have one NTP-server configured so all I need to delete is the one with index 1. After that, we will set the time manually.

   (Cisco Controller)> config time ntp delete 1

   (Cisco Controller)> config time manual 09/30/18 11:30:00

NTP-servers can also be deleted in the web-GUI under Controller > NTP > Server and then hover your mouse over the little blue triangle next to each NTP-server to see an option to remove them. Time can also be configured manually in the web-GUI under Commands > Set Time

2) Disable the device certificate authentication completely and let the AP join the WLC anyway using:

   (Cisco Controller)> config ap cert-expiry-ignore mic enable

If the certificate of your WLC has expired you may need to use both workarounds to get newer access points to join the WLC at all. Depending on your WLC version, only using one of the workarounds might not work as there were some changes to these workarounds in version 8.5 of the AireOS software, which is the operating system of the WLCs.

How do I check when my WLC’s device certificate expires?

Maybe you haven’t been hit by the issues above yet but you want to know how much time you have left until you either need to apply the workarounds or invest in new hardware. SSH into your WLC and run the following command to list all certificates installed in your WLC. Only one of the certificates installed in the WLC is used for device authentication towards the access points, so make sure to look for this one (“Cisco device cert”):

(Cisco Controller)> show certificate all

Scroll down until you find this particular certificate and check the Validity End Date:

----------------------------
Certificate Name: Cisco SHA1 device cert

     Subject Name :
         C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT5508-K9-ccd8e14058a0, [email protected]
     Issuer Name :
         O=Cisco Systems, CN=Cisco Manufacturing CA
     Serial Number (Hex):
         792EA1F60000002140AA
     Validity :
         Start : Sep 10 14:29:13 2009 GMT  
         End   : Sep 10 14:39:13 2019 GMT <<<<<<<<<<<<<<<<<<<<<
     Signature Algorithm :
         sha1WithRSAEncryption
     Hash key :
         SHA1 Fingerprint  : 3d:54:c0:b3:8b:ab:8f:51:dd:28:04:ed:54:77:16:1c:ae:c6:aa:bb
         SHA256 Fingerprint  : b8:94:86:6b:9f:04:bd:8c:0c:43:c6:46:c4:30:f7:9f:59:0c:f9:97:da:c9:07:e6:2f:18:c8:0e:aa:ae:bb:2d