[Technical] - ACS 'Server Secret Mismatch' Error.

I saw an issue where the following error was being displayed even when the shared key matched. (Or seemed to..)

"Authentication Server not responding: AAA decode failure.. server secret mismatch"

This was seen on an Cisco ASA firewall running 8.0.4 code. The ACS log would either show "Key Mismatch" or "CS password invalid" (using Radius)

The shared key was the same as that in the Network Configuration of ACS, however "Network Device Groups" were being used and the default key for the network device group was different to the specific device key and took priority.


Use the network device group (NDG) key or don't set a NDG key.


Bob

[Technical] - Error Message 'External DB is not operational' seen on Cisco ACS appliance. Wireless users fail to authenticate.

The following is a bit of advice on how to troubleshoot the error message 'External DB is not operational' or the error "Connection to Windows authentication agent established; Failed to connect to any Windows authentication agent." on a CiscoSecure ACS appliance.

This will most likely occur when authentication is being backed off to a Windows Active Directory from a CiscoSecure ACS appliance. The authentication agent software is a small application that's ideally loaded onto a Domain Controller that accepts authentication requests from the ACS and responds after querying the AD.

This issue above will most likely occur where the ACS appliance can't communicate with the agent because;

- The domain controller hosting the agent software is down.
- The agent service "csagent" has stopped responding.
- A firewall is blocking traffic between the ACS and Domain Controller.
- The IP address of the domain controller hosting the agent has changed.
- The software version of ACS and the agent need to match.
- The certificate has expired on the ACS appliance or is now invalid for the domain.



The above error message will likely be seen in the ACS "failed attempts" log file.

To troubleshoot this issue do the following;

1 - Check connectivity from the ACS appliance to the domain controller.
2 - Check that the csagent service is running on the domain controller.
3 - Stop the csagent service and restart it from DOS in debug mode.

To use debug mode open a command prompt and go to the csagent directory. Then start the agent with the -z and -p options.

C:\Program Files\Cisco\CiscoSecure ACS Agent\bin>csagent -z -p

Two windows will open, a successfully communication will look as follows:

C:\Program Files\Cisco\CiscoSecure ACS Agent\bin>csagent -z -p
Running ACSRemoteAgent server from command line..
Debug printing on..
ACSRemoteAgent server starting ==============================
Running as console application.
Will listen on port 2004
Configuration will be fetched from 192.x.x.103:2003
Agents: CSWinAgent
CSWinAgent File: ..\bin\CSWinAgent.exe
CSWinAgent Port: 2005
1 agents configured
Permitted CSAgent Clients: *.*.*.*
Hit Return/Enter to stop...

Listener activated
Watchdog activated
CSWinAgent launched
Client connecting from 192.x.x.103:1159
RPC: Info request received
RPC: Info reply sent
Client disconnected, thread 2628 terminating

--------- In the second window ----------

CSWinAgent server starting ==============================
Running as console application.
Will listen on port 2005
Permitted CSWinAgent Clients: *.*.*.*
NTLIB: Library behaviour mode 2
NTLIB: Initialising locally
NTLIB: The local computer name is MYDC12
NTLIB: The 'insist on domain' feature is enabled
NTLIB: We ARE a domain controller
NTLIB: We are a backup domain controller for domain MY.COM
NTLIB: Found 0 trusted domains
Listener activated
Client connecting from 192.x.x.103:1160
RPC: NT_MSCHAPAuthenticateUser received
NTLIB: Attempting Windows authentication for user JIM.SMITH$
NTLIB: Windows authentication SUCCESSFUL (by MYDC12)
RPC: NT_MSCHAPAuthenticateUser reply sent

Please feedback any other experiences.

Some useful URLs are;

• Remote Agent Installation Guide
• Another Similar Example

Bob

CCIE Security Anyone?

[Technical] – Unable to define a source-interface with TACACS+ on the Cisco ASA/PIX

We had a requirement to have an ASA authenticate with 2 geographically dispersed ACS servers for resilience.

• ACS1 is connected via the dmz1 interface.
• ACS2 is connected via the dmz2 interface.
• ACS1 replicates to ACS2.

The Cisco ASA/PIX doesn’t support using a source-interface for TACACS+ like a Cisco IOS based router does. So the ACS sees the request as coming from the IP address of the outgoing interface on the ASA.

The only solution is to configure two IPs within ACS per ASA one that relates to the dmz1 and one that relates to the dmz2 interfaces.

[Technical] - “Flow is denied by configured rule” message with ASA packet-tracer and traffic being dropped at the VPN phase.

I came across an interesting issue today where traffic wasn’t passing across a new IPSec Lan-to-Lan VPN correctly. On both sides were Cisco ASA firewalls 8.0(2), with the local one managed by me and the remote by a 3rd party.

Some traffic was flowing correctly and some wasn’t. A packet trace of a failing stream showed the following:


ASA#packet-tracer input inside tcp 10.1.1.65 16664 20.1.1.1 http

Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Additional Information:
Found no matching flow, creating a new flow

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Additional Information:
in 0.0.0.0 0.0.0.0 outside

Phase: 5
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
nat (inside) 0 access-list inside_nat0_outbound
match ip inside 10.1.1.64 255.255.255.192 outside 20.1.1.0 255.255.255.0
NAT exempt
translate_hits = 3, untranslate_hits = 0

Phase: 6
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Clearly the VPN phase shouldn’t have been dropping the traffic.

This was confusing there were no ACLs blocking traffic, the route, nat and crypto acl were all ok. However upon investigation the remote crypto ACL didn’t have an entry for this stream.

Upon modifying them so they were symmetrical the issue was resolved.

[Technical] - How to Static NAT two Public IPs to 1 Private IP?

A common question I get is can I NAT more than one public IP to a private IP using the Cisco ASA (or PIX) firewall.

The simple answer is yes, but you can't using the "static" command as you would expect or else you'll get the error "ERROR: duplicate of existing static".

So the following configuration will fail;
static (Inside,Outside) 201.10.10.2 10.10.10.1 netmask 255.255.255.255
static (Inside,Outside) 100.20.30.3 10.10.10.1 netmask 255.255.255.255

However using policy NAT on the PIX/ASA using code 7.x and beyond (Tested on 8.x) the following will work.
access-list policy_1 extended permit ip host 10.10.10.1 any
access-list policy_2 extended permit ip host 10.10.10.1 any
static (Inside,Outside) 201.10.10.2 access-list policy_1
static (Inside,Outside) 100.20.30.3 access-list policy_2

Interestingly if you ping 100.20.30.3 from the outside you see the echo-reply come from 201.10.10.2. ICMP isn't stateful through the firewall so the Policy NAT will use the first IP in the list as the source address of any outgoing initiated flow.

Bob

[Technical] - Using a Cisco ASA to authenticate an SSL VPN user to a Microsoft AD using LDAP. (IE: Without Radius)

For this example I used a Cisco ASA5520 running 8.0(3) authenticating against a Microsoft Windows 2003 domain. (company1.co.uk)


The logic here will allow SSL VPN users to connect so long as they are a member of either the SupportStaff or Managers group within the Microsoft active directory. Members of the Managers group within the AD will have more restricted access that members of SupportStaff. If an AD user isn’t a member of one of these groups, they will be denied access.

The following key aspects of configuration need to be completed;

• aaa-server
• ldap attribute-map
• access-lists
• ip address pools
• webvpn parameters
• group-policy
• tunnel-group

Configure Your AAA server details. The user “ldap_user” is a standard user within the Microsoft AD, ideally this users password (ldap_users_password) should be set to never expire.

aaa-server Company1-LDAP protocol ldap
aaa-server Company1-LDAP host 10.1.1.2
ldap-base-dn dc=company1,dc=co,dc=uk
ldap-scope subtree
ldap-naming-attribute sAMAccountName
ldap-login-dn “ldap_user”
ldap-login-password “ldap_users_password”
server-type microsoft
ldap-attribute-map Company1-Map

Define your ldap attribute map. This will tell the Cisco ASA which locally configured group policy to apply depending on the group membership status, within the Microsoft AD of the user connecting via the SSL VPN.

ldap attribute-map Company1-Map
map-name memberOf IETF-Radius-Class
map-value memberOf "CN=SupportStaff,OU=User Accounts,DC=1stquote,DC=co,DC=uk" ALLOWSupportAccess
map-value memberOf "CN=Managers,OU=User Accounts,DC=1stquote,DC=co,DC=uk" ALLOWManagerAccess

Then Create access lists for the split-tunnel policy (if appropriate) and for any traffic filters you wish to apply to the SSL VPN.

access-list Managers-Split-Tunnel standard permit host 10.1.1.15
access-list Managers-Split-Tunnel standard permit host 10.1.1.25
access-list Support-Split-Tunnel standard permit 10.0.0.0 255.0.0.0
access-list Restrict-Manager-Access extended permit tcp any host 10.1.1.15 eq smtp
access-list Restrict-Manager-Access extended permit tcp any host 10.1.1.25 eq www
access-list Restrict-Manager-Access extended deny ip any any

Define an IP address pool for remote users;

ip local pool ssl_vpn_pool 10.9.9.1-10.9.9.100 mask 255.255.255.0

Enable webvpn and specify the location of your svc on your ASA.

webvpn
enable outside
svc image disk0:/anyconnect-win-2.1.0148-k9.pkg 1
svc enable


Define your group-policies. These determine if a user can login and once logged in what access they have by tying back to the access-lists. Also a policy needs to be created that denies access. The LDAP map links policies to users and the NOACCESS policy is defined in the tunnel group as the default policy.

group-policy ALLOWSupportAccess internal
group-policy ALLOWSupportAccess attributes
banner value Welcome you are logged in with Support rights and full access.
dns-server value 10.1.2.3
vpn-simultaneous-logins 1
vpn-tunnel-protocol svc
split-tunnel-policy tunnelspecified
split-tunnel-network-list value Support-Split-Tunnel
default-domain value company1.co.uk
webvpn
svc ask none default svc

group-policy ALLOWManagerAccess internal
group-policy ALLOWManagerAccess attributes
banner value You are logged in as a Manager with limited access.
dns-server value 10.1.2.3
vpn-simultaneous-logins 1
vpn-filter value Restrict-Manager-Access
vpn-tunnel-protocol svc
split-tunnel-policy tunnelspecified
split-tunnel-network-list value Manager-Split-Tunnel
default-domain value company1.co.uk
webvpn
svc ask none default svc
hidden-shares none
file-entry disable
file-browsing disable

group-policy NOACCESS internal
group-policy NOACCESS attributes
vpn-simultaneous-logins 0
vpn-tunnel-protocol IPSec svc webvpn
webvpn
svc ask none default svc

Finally the tunnel-group sets the various settings for SSL VPN access to the Cisco ASA and ties the other parts of the config together.

tunnel-group SSLVPN type remote-access
tunnel-group SSLVPN general-attributes
address-pool ssl_vpn_pool
authentication-server-group Company1-LDAP
authorization-server-group Company1-LDAP
authorization-server-group (inside) Company1-LDAP
default-group-policy NOACCESS
password-management password-expire-in-days 7
authorization-required

To troubleshoot any issues enable the following debugs.

debug aaa authentication enabled at level 1
debug aaa authorization enabled at level 1
debug aaa common enabled at level 15
debug ldap enabled at level 15


Look for the following to ensure the user is authenticated.

[3891] Performing Simple authentication for Phil to 10.1.1.2
[3891] LDAP Search:
Base DN = [dc=company1,dc=co,dc=uk]
Filter = [sAMAccountName=Phil]
Scope = [SUBTREE]
[3891] User DN = [CN=Phil,OU=User Accounts,DC=company1,DC=co,DC=uk]
[3891] Talking to Active Directory server 10.1.1.2
[3891] Reading password policy for Phil, dn:CN=Phil,OU=User Accounts,DC=company1,DC=co,DC=uk
[3891] Read bad password count 0
[3891] Binding as user
[3891] Performing Simple authentication for Phil to 10.1.1.2
[3891] Processing LDAP response for user Phil
[3891] Checking password policy
[3891] Authentication successful for Phil to 10.1.1.2

Look for the following to ensure the user is authorized and the LDAP is mapping as expected.


[3891] memberOf: value = CN=SupportStaff,OU=User Accounts,DC=company1,DC=co,DC=uk
[3891] mapped to IETF-Radius-Class: value = ALLOWSupportACCESS

...

user attributes:
1 Class(25) 12 "ALLOWSupportACCESS[00]"


Thanks for looking and as always feedback on this example is always welcome.


Bob.