.NET技术+25台服务器怎样支撑世界第54大网站

发表于2014-07-25 04:39| 次阅读| 来源HighScalability| 196 条评论| 作者Tod Hoff

大数据StackOverflowRedis架构.NET

摘要：同时使用Linux和Windows平台产品，大量使用静态的方法和类，Stack Overflow是个重度性能控。同时，取代横向扩展，他们坚持着纵向扩展思路，因为“硬件永远比程序员便宜”。

【编者按】StackOverflow是一个IT技术问答网站，用户可以在网站上提交和回答问题。当下的StackOverflow已拥有400万个用户，4000万个回答，月PV5.6亿，世界排行第54。然而值得关注的是，支撑他们网站的全部服务器只有25台，并且都保持着非常低的资源使用率，这是一场高有效性、负载均衡、缓存、数据库、搜索及高效代码上的较量。近日，High Scalability创始人Todd Hoff根据Marco Cecconi的演讲视频“ The architecture of StackOverflow”以及Nick Craver的博文“ What it takes to run Stack Overflow”总结了StackOverflow的成功原因。

---

免费订阅“CSDN大数据”微信公众号，实时了解最新的大数据进展！

CSDN大数据，专注大数据资讯、技术和经验的分享和讨论，提供Hadoop、Spark、Imapala、Storm、HBase、MongoDB、Solr、机器学习、智能算法等相关大数据观点，大数据技术，大数据平台，大数据实践，大数据产业资讯等服务。

---

以下为译文

意料之中，也是意料之外，Stack Overflow仍然重度使用着微软的产品。他们认为既然微软的基础设施可以满足需求，又足够便宜，那么没有什么理由去做根本上的改变。而在需要的地方，他们同样使用了Linux。究其根本，一切都是为了性能。

另一个值得关注的地方是，Stack Overflow仍然使用着纵向扩展策略，没有使用云。他们使用了384GB的内存和2TB的SSD来支撑SQL Servers，如果使用AWS的话，花费可想而知。没有使用云的另一个原因是Stack Overflow认为云会一定程度上的降低性能，同时也会给优化和排查系统问题增加难度。此外，他们的架构也并不需要横向扩展。峰值期间是横向扩展的杀手级应用场景，然而他们有着丰富的系统调整经验去应对。该公司仍然坚持着Jeff Atwood的名言——硬件永远比程序员便宜。

Marco Ceccon曾提到，在谈及系统时，有一件事情必须首先弄明白——需要解决问题的类型。首先，从简单方面着手，StackExchange究竟是用来做什么的——首先是一些主题，然后围绕这些主题建立社区，最后就形成了这个令人敬佩的问答网站。

其次则是规模相关。StackExchange在飞速增长，需要处理大量的数据传输，那么这些都是如何完成的，特别是只使用了25台服务器，下面一起追根揭底：

状态

• StackExchange拥有110个站点，以每个月3到4个的速度增长。
• 400万用户
• 800万问题
• 4000万答案
• 世界排名54位
• 每年增长100%
• 月PV 5.6亿万
• 大多数工作日期间峰值为2600到3000请求每秒，作为一个编程相关网站，一般情况下工作日的请求都会高于周末
• 25台服务器
• SSD中储存了2TB的SQL数据
• 每个web server都配置了2个320G的SSD，使用RAID 1
• 每个ElasticSearch主机都配备了300GB的机械硬盘，同时也使用了SSD
• Stack Overflow的读写比是40:60
• DB Server的平均CPU利用率是10%
• 11个web server，使用IIS
• 2个负载均衡器，1个活跃，使用HAProxy
• 4个活跃的数据库节点，使用MS SQL
• 3台实现了tag engine的应用程序服务器，所有搜索都通过tag
• 3台服务器通过ElasticSearch做搜索
• 2台使用了Redis的服务器支撑分布式缓存和消息
• 2台Networks（Nexus 5596 + Fabric Extenders）
• 2 Cisco 5525-X ASAs 
• 2 Cisco 3945 Routers
• 主要服务Stack Exchange API的2个只读SQL Servers
• VM用于部署、域控制器、监控、运维数据库等场合

平台

• ElasticSearch
• Redis
• HAProxy
• MS SQL
• Opserver
• TeamCity
• Jil——Fast .NET JSON Serializer，建立在Sigil之上
• Dapper——微型的ORM

UI

• UI拥有一个信息收件箱，用于新徽章获得、用户发送信息、重大事件发生时的信息收取，使用WebSockets实现，并通过Redis支撑。
• 搜索箱通过 ElasticSearch 实现，使用了一个REST接口。
• 因为用户提出问题的频率很高，因此很难显示最新问题，每秒都会有新的问题产生，从而这里需要开发一个关注用户行为模式的算法，只给用户显示感兴趣的问题。它使用了基于Tag的复杂查询，这也是开发独立Tag Engine的原因。
• 服务器端模板用于生成页面。

服务器

• 25台服务器并没有满载，CPU使用率并不高，单计算SO（Stack Overflow）只需要5台服务器。
• 数据库服务器资源利用率在10%左右，除下执行备份时。
• 为什么会这么低？因为数据库服务器足足拥有384GB内存，同时web server的CPU利用率也只有10%-15%。
• 纵向扩展还没有遇到瓶颈。通常情况下，如此流量使用横向扩展大约需要100到300台服务器。
• 简单的系统。基于.Net，只用了9个项目，其他系统可能需要100个。之所以使用这么少系统是为了追求极限的编译速度，这点需要从系统开始时就进行规划，每台服务器的编译时间大约是10秒。
• 11万行代码，对比流量来说非常少。
• 使用这种极简的方式主要基于几个原因。首先，不需要太多测试，因为Meta.stackoverflow本来就是一个问题和bug讨论社区。其次，Meta.stackoverflow还是一个软件的测试网站，如果用户发现问题的话，往往会提出并给予解决方案。
• 纽约数据中心使用的是Windows 2012，已经向2012 R2升级（Oregon已经完成了升级），Linux系统使用的是Centos 6.4。

SSD

• 默认使用的是Intel 330（Web层等）
• Intel 520用于中间层写入，比如Elastic Search
• 数据层使用Intel 710和S3700
• 系统同时使用了RAID 1和RAID 10（任何4+以上的磁盘都使用RAID 10）。不畏惧故障发生，即使生产环境中使用了上千块2.5英寸SSD，还没碰到过一块失败的情景。每个模型都使用了1个以上的备件，多个磁盘发生故障的情景不在考虑之中。
• ElasticSearch在SSD上表现的异常出色，因为SO writes/re-indexes的操作非常频繁。
• SSD改变了搜索的使用方式。因为锁的问题，Luncene.net并不能支撑SO的并发负载，因此他们转向了ElasticSearch。在全SSD环境下，并不需要围绕Binary Reader建立锁。

高可用性

• 异地备份——主数据中心位于纽约，备份数据中心在Oregon。
• Redis有两个从节点，SQL有2个备份，Tag Engine有3个节点，elastic有3个节点，冗余一切，并在两个数据中心同时存在。
• Nginx是用于SSL，终止SSL时转换使用HAProxy。
• 并不是主从所有，一些临时的数据只会放到缓存中
• 所有HTTP流量发送只占总流量的77%，还存在Oregon数据中心的备份及一些其他的VPN流量。这些流量主要由SQL和Redis备份产生。

数据库

• MS SQL Server
• Stack Exchange为每个网站都设置了数据库，因此Stack Overflow有一个、Server Fault有一个，以此类推。
• 在纽约的主数据中心，每个集群通常都使用1主和1只读备份的配置，同时还会在Oregon数据中心也设置一个备份。如果是运行的是Oregon集群，那么两个在纽约数据中心的备份都会是只读和同步的。
• 为其他内容准备的数据库。这里还存在一个“网络范围”的数据库，用于储存登陆凭证和聚合数据（大部分是stackexchange.com用户文件或者API）。
• Careers Stack Overflow、stackexchange.com和Area 51等都拥有自己独立的数据库模式。
• 模式的变化需要同时提供给所有站点的数据库，它们需要向下兼容，举个例子，如果需要重命名一个列，那么将非常麻烦，这里需要进行多个操作：增加一个新列，添加作用在两个列上的代码，给新列写数据，改变代码让新列有效，移除旧列。
• 并不需要分片，所有事情通过索引来解决，而且数据体积也没那么大。如果有filtered indexes需求，那么为什么不更高效的进行？常见模式只在DeletionDate = Null上做索引，其他则通过为枚举指定类型。每项votes都设置了1个表，比如一张表给post votes，1张表给comment votes。大部分的页面都可以实时渲染，只为匿名用户缓存，因此，不存在缓存更新，只有重查询。
• Scores是非规范化的，因此需要经常查询。它只包含IDs和dates，post votes表格当下大约有56454478行，使用索引，大部分的查询都可以在数毫秒内完成。
• Tag Engine是完全独立的，这就意味着核心功能并不依赖任何外部应用程序。它是一个巨大的内存结构数组结构，专为SO用例优化，并为重负载组合进行预计算。Tag Engine是个简单的windows服务，冗余的运行在多个主机上。CPU使用率基本上保持在2-5%，3个主机专门用于冗余，不负责任何负载。如果所有主机同时发生故障，网络服务器将把Tag Engine加载到内存中持续运行。
• 关于Dapper无编译器校验查询与传统ORM的对比。使用编译器有很多好处，但在运行时仍然会存在fundamental disconnect问题。同时更重要的是，由于生成nasty SQL，通常情况还需要去寻找原始代码，而Query Hint和parameterization控制等能力的缺乏更让查询优化变得复杂。

编码

• 流程

• 大部分程序员都是远程工作，自己选择编码地点
• 编译非常快
• 然后运行少量的测试
• 一旦编译成功，代码即转移至开发交付准备服务器
• 通过功能开关隐藏新功能
• 在相同硬件上作为其他站点测试运行
• 然后转移至Meta.stackoverflow测试，每天有上千个程序员在使用，一个很好的测试环境
• 如果通过则上线，在更广大的社区进行测试

• 大量使用静态类和方法，为了更简单及更好的性能
• 编码过程非常简单，因为复杂的部分被打包到库里，这些库被开源和维护。.Net 项目数量很低，因为使用了社区共享的部分代码。
• 开发者同时使用2到3个显示器，多个屏幕可以显著提高生产效率。

缓存

• 缓存一切
• 5个等级的缓存
• 1级是网络级缓存，缓存在浏览器、CDN以及代理服务器中。
• 2级由.Net框架 HttpRuntime.Cache完成，在每台服务器的内存中。
• 3级Redis，分布式内存键值存储，在多个支撑同一个站点的服务器上共享缓存项。
• 4级SQL Server Cache，整个数据库，所有数据都被放到内存中。
• 5级SSD。通常只在SQL Server预热后才生效。
• 举个例子，每个帮助页面都进行了缓存，访问一个页面的代码非常简单：

• 使用了静态的方法和类。从OOP角度来看确实很糟，但是非常快并有利于简洁编码。
• 缓存由Redis和Dapper支撑，一个微型ORM

• 为了解决垃圾收集问题，模板中1个类只使用1个副本，被建立和保存在缓存中。监测一切，包括GC操。据统计显示，间接层增加GC压力达到了某个程度时会显著的降低性能。
• CDN Hit 。鉴于查询字符串基于文件内容进行哈希，只在有新建立时才会被再次取出。每天3000万到5000万Hit，带宽大约为300GB到600GB。
• CDN不是用来应对CPU或I/O负载，而是帮助用户更快的获得答案

部署

• 每天5次部署，不去建立过大的应用。主要因为

• 可以直接的监视性能
• 尽可能最小化建立，可以工作才是重点

• 产品建立后再通过强大的脚本拷贝到各个网页层，每个服务器的步骤是：

• 通过POST通知HAProxy下架某台服务器
• 延迟IIS结束现有请求（大约5秒）
• 停止网站（通过同一个PSSession结束所有下游）
• Robocopy文件
• 开启网站
• 通过另一个POST做HAProxy Re-enable

• 几乎所有部署都是通过puppet或DSC，升级通常只是大幅度调整RAID阵列并通过PXE boot安装，这样做非常快速。

协作

• 团队

• SRE （System Reliability Engineering）：5人
• Core Dev（Q&A site）6-7人
• Core Dev Mobile：6人
• Careers团队专门负责SO Careers产品开发：7人

• Devops和开发者结合的非常紧密
• 团队间变化很大
• 大部分员工远程工作
• 办公室主要用于销售，Denver和London除外
• 一切平等，些许偏向纽约工作者，因为面对面有助于工作交流，但是在线工作影响也并不大
• 对比可以在同一个办公室办公，他们更偏向热爱产品及有才华的工程师，他们可以很好的衡量利弊
• 许多人因为家庭而选择远程工作，纽约是不错，但是生活并不宽松
• 办公室设立在曼哈顿，那是个人才的诞生地。数据中心不能太偏，因为经常会涉及升级
• 打造一个强大团队，偏爱极客。早期的微软就聚集了大量极客，因此他们征服了整个世界
• Stack Overflow社区也是个招聘的地点，他们在那寻找热爱编码、乐于助人及热爱交流的人才。

编制预算

• 预算是项目的基础。钱只花在为新项目建立基础设施上，如此低利用率的 web server还是3年前数据中心建立时购入。

测试

• 快速迭代和遗弃
• 许多测试都是发布队伍完成的。开发拥有一个同样的SQL服务器，并且运行在相同的Web层，因此性能测试并不会糟糕。
• 非常少的测试。Stack Overflow并没有进行太多的单元测试，因为他们使用了大量的静态代码，还有一个非常活跃的社区。
• 基础设施改变。鉴于所有东西都有双份，所以每个旧配置都有备份，并使用了一个快速故障恢复机制。比如，keepalived可以在负载均衡器中快速回退。
• 对比定期维护，他们更愿意依赖冗余系统。SQL备份用一个专门的服务器进行测试，只为了可以重存储。计划做每两个月一次的全数据中心故障恢复，或者使用完全只读的第二数据中心。
• 每次新功能发布都做单元测试、集成测试盒UI测试，这就意味着可以预知输入的产品功能测试后就会推送到孵化网站，即meta.stackexchange（原meta.stackoverflow）。

监视/日志

• 当下正在考虑使用http://logstash.net/做日志管理，目前使用了一个专门的服务将syslog UDP传输到SQL数据库中。网页中为计时添加header，这样就可以通过HAProxy来捕获并且融合到syslog传输中。
• Opserver和Realog用于显示测量结果。Realog是一个日志展示系统，由Kyle Brandt和Matt Jibson使用Go建立。
• 日志通过HAProxy负载均衡器借助syslog完成，而不是IIS，因为其功能比IIS更丰富。

关于云

• 还是老生常谈，硬件永远比开发者和有效率的代码便宜。基于木桶效应，速度肯定受限于某个短板，现有的云服务基本上都存在容量和性能限制。
• 如果从开始就使用云来建设SO说不定也会达到现在的水准。但毫无疑问的是，如果达到同样的性能，使用云的成本将远远高于自建数据中心。

性能至上

• StackOverflow是个重度的性能控，主页加载的时间永远控制在50毫秒内，当下的响应时间是28毫秒。
• 程序员热衷于降低页面加载时间以及提高用户体验。
• 每个独立的网络提交都予以计时和记录，这种计量可以弄清楚提升性能需要修改的地方。
• 如此低资源利用率的主要原因就是高效的代码。web server的CPU平均利用率在5%到15%之间，内存使用为15.5 GB，网络传输在20 Mb/s到40 Mb/s。SQL服务器的CPU使用率在5%到10%之间，内存使用是365GB，网络传输为100 Mb/s到200 Mb/s。这可以带来3个好处：给升级留下很大的空间；在严重错误发生时可以保持服务可用；在需要时可以快速回档。

学到的知识

1. 为什么使用MS产品的同时还使用Redis？什么好用用什么，不要做无必要的系统之争，比如C#在Windows机器上运行最好，我们使用IIS；Redis在*nix机器上可以得到充分发挥，我们使用*nix。

2. Overkill即策略。平常的利用率并不能代表什么，当某些特定的事情发生时，比如备份、重建等完全可以将资源使用拉满。

3. 坚固的SSD。所有数据库都建立在SSD之上，这样可以获得0延时。

4. 了解你的读写负载。

5. 高效的代码意味着更少的主机。只有新项目上线时才会因为特殊需求增加硬件，通常情况下是添加内存，但在此之外，高效的代码就意味着0硬件添加。所以经常只讨论两个问题：为存储增加新的SSD；为新项目增加硬件。

6. 不要害怕定制化。SO在Tag上使用复杂查询，因此专门开发了所需的Tag Engine。

7. 只做必须做的事情。之所以不需要测试是因为有一个活跃的社区支撑，比如，开发者不用担心出现“Square Wheel”效应，如果开发者可以制作一个更更轻量级的组件，那就替代吧。

8. 注重硬件知识，比如IL。一些代码使用IL而不是C#。聚焦SQL查询计划。使用web server的内存转储究竟做了些什么。探索，比如为什么一个split会产生2GB的垃圾。

9. 切勿官僚作风。总有一些新的工具是你需要的，比如，一个编辑器，新版本的Visual Studio，降低提升过程中的一切阻力。

10. 垃圾回收驱动编程。SO在减少垃圾回收成本上做了很多努力，跳过类似TDD的实践，避免抽象层，使用静态方法。虽然极端，但是确实打造出非常高效的代码。

11. 高效代码的价值远远超出你想象，它可以让硬件跑的更快，降低资源使用，切记让代码更容易被程序员理解。

原文链接： StackOverflow Update: 560M Pageviews A Month, 25 Servers, And It's All About Performance（编译/仲浩 审校/魏伟）

Cisco ASA troubleshooting commands

With my requirements for any networking layer 3 security device I collected the basic commands that you have to know or you will not be able to manage your device.

1.0 Check the basic settings and firewall states

Check the system status
Check the hardware performance
Check the High Availability state
Check the session table of the firewall

2.0 Check the interface settings

Check the state, speed and duplexity an IP of the interfaces
Check the ARP Table

3.0 Check the Routing Table

Check the matching route

4.0 VPN Troubleshooting

Change the tunnel state
Check the tunnel state
Check packet counters for the tunnel
Check the uptime of the VPN Tunnels

5.0 sniffertrace

6.0 View logging on cli

Configure logging
Viewing the logs

7.0 Inspection and asp-drop

8.0 Threat Detection (check the top talkers)

9.0 Backup and Restore

1.0 Check the basic settings and firewall states

Check the system status

To see the actual software version, operational mode, HA, etc and the system time:

myfirewall/pri/act# show firewall Firewall mode: Router myfirewall/pri/act# show version Cisco Adaptive Security Appliance Software Version 9.1(1) Device Manager Version 7.1(1)52 Compiled on Wed 28-Nov-12 10:38 by builders System image file is "disk0:/asa911-k8.bin" Config file at boot was "startup-config" myfirewall up 218 days 1 hour failover cluster up 5 years 10 days Hardware: ASA5520, 2048 MB RAM, CPU Pentium 4 Celeron 2000 MHz, Internal ATA Compact Flash, 256MB BIOS Flash M50FW080 @ 0xfff00000, 1024KB Encryption hardware device : Cisco ASA-55xx on-board accelerator (revision 0x0) Boot microcode : CN1000-MC-BOOT-2.00 SSL/IKE microcode : CNLite-MC-SSLm-PLUS-2.03 IPSec microcode : CNlite-MC-IPSECm-MAIN-2.08 Number of accelerators: 1 0: Ext: GigabitEthernet0/0 : address is 001f.abcc.a8c6, irq 9 1: Ext: GigabitEthernet0/1 : address is 001f.abcc.a5e7, irq 9 2: Ext: GigabitEthernet0/2 : address is 001f.abcc.a5e8, irq 9 3: Ext: GigabitEthernet0/3 : address is 001f.abcc.a5e9, irq 9 4: Ext: Management0/0 : address is 001f.abcc.a5ea, irq 11 5: Int: Not used : irq 11 6: Int: Not used : irq 5 Licensed features for this platform: Maximum Physical Interfaces : Unlimited perpetual Maximum VLANs : 150 perpetual Inside Hosts : Unlimited perpetual Failover : Active/Active perpetual Encryption-DES : Enabled perpetual Encryption-3DES-AES : Enabled perpetual Security Contexts : 2 perpetual GTP/GPRS : Disabled perpetual AnyConnect Premium Peers : 2 perpetual AnyConnect Essentials : Disabled perpetual Other VPN Peers : 750 perpetual Total VPN Peers : 750 perpetual Shared License : Disabled perpetual AnyConnect for Mobile : Disabled perpetual AnyConnect for Cisco VPN Phone : Disabled perpetual Advanced Endpoint Assessment : Disabled perpetual UC Phone Proxy Sessions : 2 perpetual Total UC Proxy Sessions : 2 perpetual Botnet Traffic Filter : Disabled perpetual Intercompany Media Engine : Disabled perpetual Cluster : Disabled perpetual This platform has an ASA 5520 VPN Plus license. Failover cluster licensed features for this platform: Maximum Physical Interfaces : Unlimited perpetual Maximum VLANs : 150 perpetual Inside Hosts : Unlimited perpetual Failover : Active/Active perpetual Encryption-DES : Enabled perpetual Encryption-3DES-AES : Enabled perpetual Security Contexts : 4 perpetual GTP/GPRS : Disabled perpetual AnyConnect Premium Peers : 4 perpetual AnyConnect Essentials : Disabled perpetual Other VPN Peers : 750 perpetual Total VPN Peers : 750 perpetual Shared License : Disabled perpetual AnyConnect for Mobile : Disabled perpetual AnyConnect for Cisco VPN Phone : Disabled perpetual Advanced Endpoint Assessment : Disabled perpetual UC Phone Proxy Sessions : 4 perpetual Total UC Proxy Sessions : 4 perpetual Botnet Traffic Filter : Disabled perpetual Intercompany Media Engine : Disabled perpetual Cluster : Disabled perpetual This platform has an ASA 5520 VPN Plus license. Serial Number: JMX4567L1DA Running Permanent Activation Key: 0x650e6758 0x345sb616 0x1233615a 0xc234fca3 0x111e9982 Configuration register is 0x1 Configuration last modified by admin at 10:41:22.791 CEDT Fri Sep 13 2013

The failover state.

myfirewall/pri/act(config)# sh failover state State Last Failure Reason Date/Time This host - Primary Active None Other host - Secondary Standby Ready Ifc Failure 17:38:56 CEDT Jun 10 2013 dmz5: Failed inside: Failed ====Configuration State=== Sync Done Sync Done - STANDBY ====Communication State=== Mac set

To see what the firewall has seen so far, the traffic mix conserning the enabled inspections:

myfirewall/pri/act(config)# sh service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: dns preset_dns_map, packet 6206448, drop 1493, reset-drop 0, v6-fail-close 0 Inspect: ftp, packet 0, drop 0, reset-drop 0, v6-fail-close 0 Inspect: netbios, packet 285884, drop 0, reset-drop 0, v6-fail-close 0 Inspect: tftp, packet 0, drop 0, reset-drop 0, v6-fail-close 0 Inspect: icmp, packet 14657730, drop 1226951, reset-drop 0, v6-fail-close 0 Inspect: icmp error, packet 10377, drop 0, reset-drop 0, v6-fail-close 0 Inspect: dcerpc, packet 199070, drop 0, reset-drop 0, v6-fail-close 0 tcp-proxy: bytes in buffer 0, bytes dropped 0

Check the hardware performance

To see what is the state of the cpu and the memory:

myfirewall/pri/act(config)# sh cpu usage CPU utilization for 5 seconds = 8%; 1 minute: 9%; 5 minutes: 9% myfirewall/pri/act(config)# myfirewall/pri/act(config)# myfirewall/pri/act(config)# sh memory Free memory: 1722679208 bytes (80%) Used memory: 424804440 bytes (20%) ------------- ------------------ Total memory: 2147483648 bytes (100%) myfirewall/pri/act# show processes cpu-usage sorted PC Thread 5Sec 1Min 5Min Process 0x0827e731 0x6e5d2d8c 8.4% 8.7% 8.5% Dispatch Unit 0x0878d2de 0x6e5bf254 0.2% 0.9% 0.4% ARP Thread 0x090b0155 0x6e5b7fb4 0.2% 0.2% 0.1% ssh 0x08785b0e 0x6e5bf460 0.0% 0.0% 0.0% IP Thread 0x081735b4 0x6e5c56a0 0.0% 0.0% 0.0% CTM message handler 0x08cdd5cc 0x6e5c2580 0.0% 0.0% 0.0% update_cpu_usage 0x084e2936 0x6e5c04c0 0.0% 0.0% 0.0% fover_health_monitoring_thread 0x0935c832 0x6e5bc964 0.0% 0.0% 0.0% vpnfol_thread_timer 0x080596a4 0x6e5d31a4 0.0% 0.0% 0.0% block_diag 0x08854a74 0x6e5d2974 0.0% 0.0% 0.0% WebVPN KCD Process 0x084c6b6d 0x6e5d2768 0.0% 0.0% 0.0% CF OIR 0x08eafaec 0x6e5d255c 0.0% 0.0% 0.0% lina_int 0x0807209d 0x6e5d1f38 0.0% 0.0% 0.0% Reload Control Thread 0x08086369 0x6e5d1d2c 0.0% 0.0% 0.0% aaa 0x0916ad6d 0x6e5d1b20 0.0% 0.0% 0.0% UserFromCert Thread 0x0916ad6d 0x6e5d1914 0.0% 0.0% 0.0% aaa_shim_thread 0x080bae3c 0x6e5d14fc 0.0% 0.0% 0.0% CMGR Server Process 0x080bd4ad 0x6e5d12f0 0.0% 0.0% 0.0% CMGR Timer Process 0x0816d455 0x6e5d049c 0.0% 0.0% 0.0% CTM Daemon 0x081df2c5 0x6e5d0290 0.0% 0.0% 0.0% SXP CORE 0x081d7041 0x6e5d0084 0.0% 0.0% 0.0% RBM CORE 0x081cde3c 0x6e5cfe78 0.0% 0.0% 0.0% cts_task 0x081cf2ed 0x6e5cfc6c 0.0% 0.0% 0.0% cts_timer_task 0x0827c804 0x6e5cf43c 0.0% 0.0% 0.0% dbgtrace 0x0856b194 0x6e5cec0c 0.0% 0.0% 0.0% 557mcfix 0x0856b126 0x6e5cea00 0.0% 0.0% 0.0% 557statspoll ... myfirewall/pri/act# show processes internals Invoked Giveups Max_Runtime Process 1 0 0.025 block_diag 1926681692 1926681692 32.679 Dispatch Unit 3768836 0 0.189 WebVPN KCD Process 1 0 0.012 CF OIR 1 0 0.001 lina_int 1 0 0.003 Reload Control Thread 374305 233705 0.135 aaa 10 4 1.427 UserFromCert Thread 64 63 0.104 aaa_shim_thread 2 0 0.009 CMGR Server Process 2 0 0.008 CMGR Timer Process 1 0 0.001 CTM Daemon 62 0 0.044 SXP CORE ... myfirewall/pri/act(config)# sh perfmon PERFMON STATS: Current Average Xlates 0/s 0/s Connections 0/s 0/s TCP Conns 0/s 0/s UDP Conns 0/s 0/s URL Access 0/s 0/s URL Server Req 0/s 0/s TCP Fixup 0/s 0/s TCP Intercept Established Conns 0/s 0/s TCP Intercept Attempts 0/s 0/s TCP Embryonic Conns Timeout 0/s 0/s HTTP Fixup 0/s 0/s FTP Fixup 0/s 0/s AAA Authen 0/s 0/s AAA Author 0/s 0/s AAA Account 0/s 0/s VALID CONNS RATE in TCP INTERCEPT: Current Average N/A 100.00%

Check the High Availability state

to get the High Availability state info with show failover command:

myfirewall/pri/act(config)# show failover ? exec mode commands/options: descriptor Show failover interface descriptors. Two numbers are shown for each interface. When exchanging information regarding a particular interface, this unit uses the first number in messages it sends to its peer. And it expects the second number in messages it receives from its peer. For trouble shooting, collect the show output from both units and verify that the numbers match. exec Show failover command execution information history Show failover switching history interface Show failover command interface information state Show failover internal state information statistics Show failover command interface statistics information | Output modifiers

Check the failover state:

myfirewall/pri/act(config)# show failover Failover On Failover unit Primary Failover LAN Interface: failover GigabitEthernet0/2 (up) Unit Poll frequency 1 seconds, holdtime 15 seconds Interface Poll frequency 5 seconds, holdtime 25 seconds Interface Policy 1 Monitored Interfaces 3 of 160 maximum Version: Ours 9.1(1), Mate 9.1(1) Last Failover at: 07:31:49 CEST Feb 12 2013 This host: Primary - Active Active time: 18841674 (sec) slot 0: ASA5520 hw/sw rev (2.0/9.1(1)) status (Up Sys) Interface dmz5 (192.168.36.1): Normal (Monitored) Interface dmz6 (192.168.47.1): Normal (Not-Monitored) Interface inside (172.24.3.5): Normal (Monitored) Interface oob (192.168.99.1): Normal (Monitored) Interface management (0.0.0.0): No Link (Not-Monitored) slot 1: empty Other host: Secondary - Standby Ready Active time: 0 (sec) slot 0: ASA5520 hw/sw rev (2.0/9.1(1)) status (Up Sys) Interface dmz5 (192.168.36.2): Normal (Monitored) Interface dmz6 (192.168.47.2): Normal (Not-Monitored) Interface inside (172.24.3.6): Normal (Monitored) Interface oob (192.168.99.2): Normal (Monitored) Interface management (0.0.0.0): Normal (Not-Monitored) slot 1: empty Stateful Failover Logical Update Statistics Link : failover GigabitEthernet0/2 (up) Stateful Obj xmit xerr rcv rerr General 372747905 0 2453073 0 sys cmd 2452421 0 2452415 0 up time 0 0 0 0 RPC services 0 0 0 0 TCP conn 1275302 0 0 0 UDP conn 17706401 0 36 0 ARP tbl 351007284 0 621 0 Xlate_Timeout 0 0 0 0 IPv6 ND tbl 0 0 0 0 VPN IKEv1 SA 0 0 0 0 VPN IKEv1 P2 0 0 0 0 VPN IKEv2 SA 0 0 0 0 VPN IKEv2 P2 0 0 0 0 VPN CTCP upd 0 0 0 0 VPN SDI upd 0 0 0 0 VPN DHCP upd 0 0 0 0 SIP Session 0 0 0 0 Route Session 306520 0 0 0 User-Identity 5 0 1 0 CTS SGTNAME 0 0 0 0 CTS PAC 0 0 0 0 TrustSec-SXP 0 0 0 0 IPv6 Route 0 0 0 0 Logical Update Queue Information Cur Max Total Recv Q: 0 88 2453116 Xmit Q: 0 29 381560801 myfirewall/pri/act(config)# show failover interface interface failover GigabitEthernet0/2 System IP Address: 192.168.92.109 255.255.255.252 My IP Address : 192.168.92.109 Other IP Address : 192.168.92.110 myfirewall/pri/act(config)# show failover descriptor dmz5 send: 000200000e000000 receive: 000200000e000000 dmz6 send: 0002000041000000 receive: 0002000041000000 inside send: 0002010064000000 receive: 0002010064000000 oob send: 00020300ffff0000 receive: 00020300ffff0000 management send: 01010000ffff0000 receive: 01010000ffff0000 myfirewall/pri/act(config)# show failover history ========================================================================== From State To State Reason ========================================================================== 07:30:59 CEST Feb 12 2013 Not Detected Negotiation No Error 07:31:03 CEST Feb 12 2013 Negotiation Cold Standby Detected an Active mate 07:31:05 CEST Feb 12 2013 Cold Standby Sync Config Detected an Active mate 07:31:15 CEST Feb 12 2013 Sync Config Sync File System Detected an Active mate 07:31:15 CEST Feb 12 2013 Sync File System Bulk Sync Detected an Active mate 07:31:29 CEST Feb 12 2013 Bulk Sync Standby Ready Detected an Active mate 07:31:49 CEST Feb 12 2013 Standby Ready Just Active HELLO not heard from mate 07:31:49 CEST Feb 12 2013 Just Active Active Drain HELLO not heard from mate 07:31:49 CEST Feb 12 2013 Active Drain Active Applying Config HELLO not heard from mate 07:31:49 CEST Feb 12 2013 Active Applying Config Active Config Applied HELLO not heard from mate 07:31:49 CEST Feb 12 2013 Active Config Applied Active HELLO not heard from mate ========================================================================== myfirewall/pri/act(config)# show failover state State Last Failure Reason Date/Time This host - Primary Active None Other host - Secondary Standby Ready Ifc Failure 17:38:56 CEDT Jun 10 2013 dmz5: Failed inside: Failed ====Configuration State=== Sync Done Sync Done - STANDBY ====Communication State=== Mac set myfirewall/pri/act(config)# show failover statistics tx:384585696 rx:29127977

Check the failover configuration:

myfirewall/pri/act(config)# sh run all failover failover failover lan unit primary failover lan interface failover GigabitEthernet0/2 failover polltime unit 1 holdtime 15 failover polltime interface 5 holdtime 25 failover interface-policy 1 failover link failover GigabitEthernet0/2 failover interface ip failover 192.168.92.109 255.255.255.252 standby 192.168.92.110

Check the session table of the firewall

With class-map you can set the maximum session for a specific traffic or generally with any:

myfirewall(config)# class-map CONNS myfirewall(config-cmap)# match any myfirewall(config-cmap)# policy-map CONNS myfirewall(config-pmap)# class CONNS myfirewall(config-pmap-c)# set connection conn-max 1000 embryonic-conn-max 3000

The values from the session table of the firewall (the max against the used if configured):

myfirewall/pri/act(config)# show conn ? exec mode commands/options: address Enter this keyword to specify IP address all Enter this keyword to show conns including to-the-box and from-the-box count Enter this keyword to show conn count only detail Enter this keyword to show conn in detail long Enter this keyword to show conn in long format port Enter this keyword to specify port protocol Enter this keyword to specify conn protocol scansafe Enter this keyword to show conns being forwarded to scansafe server security-group Enter this keyword to show security-group attributes in conns state Enter this keyword to specify conn state user Enter this keyword to specify conn user user-group Enter this keyword to specify conn user group user-identity Enter this keyword to show user names | Output modifiers myfirewall/pri/act(config)# show conn count 77 in use, 1013 most used myfirewall/pri/act(config)# show conn state ? exec mode commands/options: WORD Enter any number of the following conn states using ',' as separator: up finin finout http_get smtp_data nojava data_in data_out sunrpc h225 h323 sqlnet_fixup_data conn_inbound sip mgcp ctiqbe skinny service_module stub tcp_embryonic vpn_orphan myfirewall/pri/act(config)# show conn state up 80 in use, 1013 most used TCP dmz5 192.168.38.250:4634 inside 172.24.1.2:54320, idle 0:02:29, bytes 12905, flags UIOB TCP dmz5 192.168.38.250:4633 inside 172.24.1.2:135, idle 0:02:29, bytes 684, flags UIOB TCP dmz6 192.168.47.8:80 dmz5 192.168.37.227:55335, idle 0:00:00, bytes 1618307080, flags UIOB TCP dmz6 192.168.47.10:80 dmz5 192.168.37.227:65521, idle 0:00:00, bytes 61797243, flags UIOB TCP dmz6 192.168.47.11:80 dmz5 192.168.37.227:55339, idle 0:00:00, bytes 3811666664, flags UIOB TCP dmz5 192.168.36.251:80 inside 172.31.229.68:62940, idle 0:00:00, bytes 335503, flags UIO TCP dmz5 192.168.36.251:80 inside 172.24.162.217:57429, idle 0:00:00, bytes 474510, flags UIO TCP dmz5 192.168.38.250:23757 inside 172.24.3.38:1165, idle 0:00:00, bytes 59747307, flags UIO TCP dmz5 192.168.38.250:3389 inside 192.168.252.66:4042, idle 0:00:48, bytes 337870, flags UIO TCP dmz5 192.168.38.250:23757 inside 172.24.3.40:63433, idle 0:00:00, bytes 93168991, flags UIO

You can filter to the session that you looking for (example):

myfirewall/pri/act(config)# show conn long address 192.168.47.10 74 in use, 1013 most used Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, B - initial SYN from outside, b - TCP state-bypass or nailed, C - CTIQBE media, c - cluster centralized, D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN, G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, i - incomplete, J - GTP, j - GTP data, K - GTP t3-response k - Skinny media, M - SMTP data, m - SIP media, n - GUP O - outbound data, P - inside back connection, p - Phone-proxy TFTP connection, q - SQL*Net data, R - outside acknowledged FIN, R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, V - VPN orphan, W - WAAS, X - inspected by service module, x - per session, Y - director stub flow, y - backup stub flow, Z - Scansafe redirection, z - forwarding stub flow TCP dmz6: 192.168.47.10/80 (192.168.47.10/80) dmz5: 192.168.37.227/65521 (192.168.37.227/65521), flags UIOB , idle 0s, uptime 20D23h, timeout 1h0m, bytes 478172338

Check the traffic on interfaces, the packet and byte counters.

myfirewall/pri/act(config)# show traffic dmz5: received (in 1661754.406 secs): 14637140684 packets 673671106797 bytes 8001 pkts/sec 405002 bytes/sec transmitted (in 1661754.406 secs): 38728179279 packets 53732439765301 bytes 23000 pkts/sec 32334000 bytes/sec 1 minute input rate 1382 pkts/sec, 67193 bytes/sec 1 minute output rate 3546 pkts/sec, 4923809 bytes/sec 1 minute drop rate, 0 pkts/sec 5 minute input rate 1375 pkts/sec, 67887 bytes/sec 5 minute output rate 3589 pkts/sec, 4994000 bytes/sec 5 minute drop rate, 0 pkts/sec dmz6: received (in 1661754.416 secs): 38627911784 packets 53724170049557 bytes 23002 pkts/sec 32329000 bytes/sec transmitted (in 1661754.416 secs): 14299138045 packets 572124451016 bytes 8000 pkts/sec 344002 bytes/sec 1 minute input rate 3535 pkts/sec, 4923119 bytes/sec 1 minute output rate 1354 pkts/sec, 54206 bytes/sec 1 minute drop rate, 0 pkts/sec 5 minute input rate 3577 pkts/sec, 4993200 bytes/sec 5 minute output rate 1345 pkts/sec, 53821 bytes/sec 5 minute drop rate, 0 pkts/sec inside: received (in 1661754.416 secs): 826826503 packets 60669330026 bytes 1 pkts/sec 36000 bytes/sec transmitted (in 1661754.416 secs): 245271895 packets 109518736779 bytes 0 pkts/sec 65000 bytes/sec 1 minute input rate 44 pkts/sec, 2772 bytes/sec 1 minute output rate 25 pkts/sec, 13180 bytes/sec 1 minute drop rate, 21 pkts/sec 5 minute input rate 45 pkts/sec, 2829 bytes/sec 5 minute output rate 28 pkts/sec, 14443 bytes/sec 5 minute drop rate, 21 pkts/sec

Check the timeout values in the firewall:

myfirewall2/pri/act# sh run timeout timeout xlate 3:00:00 timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02 timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00 timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00 timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute timeout tcp-proxy-reassembly 0:01:00 timeout floating-conn 0:00:00

2.0 Check the interface settings

Check the state, speed and duplexity an IP of the interfaces

Show the running config only for the interfaces with ip address:

myfirewall/pri/act(config)# sh run ip address ! interface GigabitEthernet0/0.14 vlan 14 nameif dmz5 security-level 0 ip address 192.168.36.1 255.255.252.0 standby 192.168.36.2 ! interface GigabitEthernet0/0.65 vlan 65 nameif dmz6 security-level 0 ip address 192.168.47.1 255.255.255.0 standby 192.168.47.2 ! interface GigabitEthernet0/1.100 vlan 100 nameif inside security-level 100 ip address 192.168.3.5 255.255.248.0 standby 172.24.3.6

Show ip address and security level only:

myfirewall2/pri/act# sh ip System IP Addresses: Interface Name IP address Subnet mask Method Port-channel1.1001 dmz1 5.5.5.5 255.255.255.192 CONFIG Port-channel2 Failover 192.168.92.13 255.255.255.252 unset Port-channel4.721 inside 172.17.131.151 255.255.255.0 CONFIG Current IP Addresses: Interface Name IP address Subnet mask Method Port-channel1.1001 dmz1 5.5.5.5 255.255.255.192 CONFIG Port-channel2 Failover 192.168.92.13 255.255.255.252 unset Port-channel4.721 inside 172.17.131.151 255.255.255.0 CONFIG myfirewall2/pri/act# sh nameif Interface Name Security Management0/0 management 100 Port-channel1.1001 dmz1 0 Port-channel4.721 inside 100

Check the MAC and the state of the interfaces. The name of the interface in the example below is internal.

Here you can see following in the output

- Interface name
– MAC
– Link state
– Speed
– Duplex
– MTU
– Packet and Byte counters
– Errors

myfirewall/pri/act# show interface Interface GigabitEthernet0/0 "", is up, line protocol is up Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps) Input flow control is unsupported, output flow control is off Available but not configured via nameif MAC address 001f.abcc.a5e6, MTU not set IP address unassigned 53280934440 packets input, 55671972432495 bytes, 0 no buffer Received 167625118 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 pause input, 0 resume input 0 L2 decode drops 53043155385 packets output, 55516746848674 bytes, 0 underruns 0 pause output, 0 resume output 0 output errors, 0 collisions, 2 interface resets 0 late collisions, 0 deferred 0 input reset drops, 0 output reset drops, 0 tx hangs input queue (blocks free curr/low): hardware (255/230) output queue (blocks free curr/low): hardware (255/122) Interface GigabitEthernet0/0.14 "dmz5", is up, line protocol is up Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec VLAN identifier 14 Description: dmz5 MAC address 001f.abcc.a5e6, MTU 1500 IP address 192.168.36.1, subnet mask 255.255.252.0 Traffic Statistics for "dmz5": 14641601950 packets input, 673897945554 bytes 38739676247 packets output, 53748403391129 bytes 51923927 packets dropped Interface GigabitEthernet0/0.65 "dmz6", is up, line protocol is up Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec VLAN identifier 65 Description: dmz6 MAC address 001f.abcc.a5e6, MTU 1500 IP address 192.168.47.1, subnet mask 255.255.255.0 Traffic Statistics for "dmz6": 38639332463 packets input, 53740092462779 bytes 14303479193 packets output, 572298134370 bytes 83451 packets dropped

Check the ARP Table

This contains the permanent and the dynamic ARP entries

myfirewall/pri/act# show arp dmz5 192.168.38.43 0020.4ab0.a59f 0 dmz5 192.168.37.226 2c27.d733.a9e2 0 dmz5 192.168.37.236 2c27.d733.a89e 0 dmz5 192.168.37.235 78ac.c0b2.4066 0 dmz5 192.168.37.240 0019.99ae.847c 0 dmz5 192.168.39.240 0019.9987.5676 0 ...

3.0 Check the Routing Table

With the show route you can see the actual routing table from the firewall with the statis and the dynamic routes and the directly connected networks.

myfirewall/pri/act# show route Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route Gateway of last resort is 172.24.2.2 to network 0.0.0.0 C 172.24.0.0 255.255.248.0 is directly connected, inside C 192.168.99.0 255.255.255.0 is directly connected, oob C 192.168.47.0 255.255.255.0 is directly connected, dmz6 C 192.168.92.108 255.255.255.252 is directly connected, failover S* 0.0.0.0 0.0.0.0 [1/0] via 172.24.2.2, inside C 192.168.36.0 255.255.252.0 is directly connected, dmz5

Check the matching route

Are you looking for a specific route in a big database? No problem use the show route with more details:

myfirewall/pri/act# sh route inside 172.31.231.246 Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route Gateway of last resort is 172.24.2.2 to network 0.0.0.0

4.0 VPN Troubleshooting

The most significant part for vpn is the time on the devices. The check the time use the following command:

myfirewall/pri/act# show clock 11:19:45.485 CEDT Wed Sep 18 2013 myfirewall/pri/act# show ntp status Clock is synchronized, stratum 3, reference is 172.24.10.100 nominal freq is 99.9984 Hz, actual freq is 99.9968 Hz, precision is 2**6 reference time is d5e3ed1d.b0b7a760 (11:13:01.690 CEDT Wed Sep 18 2013) clock offset is 0.1998 msec, root delay is 18.55 msec root dispersion is 36.01 msec, peer dispersion is 15.64 msec

Change the tunnel state

Bring up a vpn tunnel manually. No traffic required.

Shut down a vpn tunnel manually.

All tunnels: myfirewall3/pri/act# clear crypto isakmp sa

Only specific tunnel:

myfirewall3/pri/act# clear ipsec sa peer 2.2.2.2 myfirewall2/pri/act# clear cry ikev1 sa 2.2.2.2 shutdown for longer time: myfirewall2/pri/act(config)# no crypto map l2lvpns 10 set peer 211.66.176.18

Check the tunnel state

If there is no SA that means the tunnel is down and does not work. To see if the tunnel is up we need to check if any SA exist.
To see if the tunnel is up you can use the “show crypto isakmp sa” or “show crypto ipsec sa” command.
Tunnel state is down

Tunnel does not exist if there is no output of the commands below:

myfirewall3/pri/act# sh cry isakmp sa There are no IKEv1 SAs There are no IKEv2 SAs myfirewall3/pri/act# show crypto ipsec sa There are no ipsec sas Tunnel state is up

Informations from the output of the command below:
– vpn peers
– encrypted traffic (source and destination)
– traffic counters for encrypted traffic
– SPI for encrypt and decrypt
– Encryption method

myfirewall2/pri/act# show cry ips sa peer 3.3.3.3 peer address: 3.3.3.3 Crypto map tag: firmen, seq num: 22, local addr: 5.5.5.5 access-list tun-voss extended permit ip host 172.19.212.10 192.168.15.72 255.255.255.248 time-range End-Dec-2035 local ident (addr/mask/prot/port): (172.19.212.10/255.255.255.255/0/0) remote ident (addr/mask/prot/port): (192.168.15.72/255.255.255.248/0/0) current_peer: 3.3.3.3 #pkts encaps: 26, #pkts encrypt: 26, #pkts digest: 26 #pkts decaps: 9, #pkts decrypt: 9, #pkts verify: 9 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 26, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #send errors: 0, #recv errors: 0 local crypto endpt.: 5.5.5.5/0, remote crypto endpt.: 3.3.3.3/0 path mtu 1500, ipsec overhead 74, media mtu 1500 current outbound spi: AB092E6E current inbound spi : 910F4308 inbound esp sas: spi: 0x910F4308 (2433696520) transform: esp-aes-256 esp-sha-hmac no compression in use settings ={L2L, Tunnel, PFS Group 2, } slot: 0, conn_id: 25923584, crypto-map: firmen sa timing: remaining key lifetime (kB/sec): (4373999/3360) IV size: 16 bytes replay detection support: Y Anti replay bitmap: 0x00000000 0x000003FF outbound esp sas: spi: 0xAB092E6E (2869505646) transform: esp-aes-256 esp-sha-hmac no compression in use settings ={L2L, Tunnel, PFS Group 2, } slot: 0, conn_id: 25923584, crypto-map: firmen sa timing: remaining key lifetime (kB/sec): (4373997/3360) IV size: 16 bytes replay detection support: Y Anti replay bitmap: 0x00000000 0x00000001

Check packet counters for the tunnel

To see if the encryption and decryption of the packages works use 2 or more times the show cry ipsec sa command and compare the values. On the second and third outputs the counter should show larger number.

On the following output the firewall has 1 active vpn peer.

myfirewall2/pri/act# show vpn-sessiondb l2l Session Type: LAN-to-LAN Connection : 9.9.9.9 Index : 5671 IP Addr : 9.9.9.9 Protocol : IKEv1 IPsec Encryption : 3DES Hashing : MD5 Bytes Tx : 83496278 Bytes Rx : 420469160 Login Time : 02:17:25 CEDT Wed Sep 18 2013 Duration : 12h:15m:49s Connection : 3.3.3.3 Index : 6329 IP Addr : 3.3.3.3 Protocol : IKEv1 IPsec Encryption : AES256 Hashing : SHA1 Bytes Tx : 6100 Bytes Rx : 5992 Login Time : 14:26:13 CEDT Wed Sep 18 2013 Duration : 0h:07m:01s

Check the uptime of the VPN tunnels

Uptime for site to site VPN

asa-firewall/pri/act# show vpn-sessiondb l2l Session Type: LAN-to-LAN Connection : 25.25.25.25 Index : 34872 IP Addr : 25.25.25.25 Protocol : IKEv1 IPsec Encryption : IKEv1: (1)AES256 IPsec: (3)AES256 Hashing : IKEv1: (1)SHA1 IPsec: (3)SHA1 Bytes Tx : 73653504 Bytes Rx : 31342653 Login Time : 01:15:18 CEST Thu Nov 28 2013 Duration : 12h:36m:51s Connection : dyn-vpn-tunnel Index : 34902 IP Addr : 35.35.35.35 Protocol : IKEv1 IPsec Encryption : IKEv1: (1)AES256 IPsec: (1)AES256 Hashing : IKEv1: (1)SHA1 IPsec: (1)SHA1 Bytes Tx : 17679966 Bytes Rx : 2626429 Login Time : 12:38:17 CEST Thu Nov 28 2013 Duration : 1h:13m:52s

SA Lifetime for IKE /phase1/ for site to site (lifetime in seconds)

asa-firewall/pri/act# show crypto isa sa detail IKEv1 SAs: Active SA: 4 Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey) Total IKE SA: 4 1 IKE Peer: 45.45.45.45 Type : L2L Role : responder Rekey : no State : AM_ACTIVE Encrypt : aes-256 Hash : SHA Auth : preshared Lifetime: 14400 Lifetime Remaining: 12039 2 IKE Peer: 55.55.55.55 Type : L2L Role : responder Rekey : no State : MM_ACTIVE Encrypt : 3des Hash : MD5 Auth : preshared Lifetime: 14400 Lifetime Remaining: 12462

SA Lifetimes for inbound and outbound esp sa-s /phase2/ for site to site (lifetime in seconds)

asa-firewall/pri/act# show crypto ipsec sa interface: outside Crypto map tag: tunnel, seq num: 20, local addr: 46.46.46.46 access-list tun-acl1 extended permit ip host 10.10.10.11 192.168.1.48 255.255.255.240 time-range End-Dec-2035 local ident (addr/mask/prot/port): (10.10.10.11/255.255.255.255/0/0) remote ident (addr/mask/prot/port): (192.168.1.48/255.255.255.240/0/0) current_peer: 13.13.13.13 #pkts encaps: 38097, #pkts encrypt: 38097, #pkts digest: 38097 #pkts decaps: 34559, #pkts decrypt: 34559, #pkts verify: 34559 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 38097, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #TFC rcvd: 0, #TFC sent: 0 #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0 #send errors: 0, #recv errors: 0 local crypto endpt.: 46.46.46.46/0, remote crypto endpt.: 13.13.13.13/0 path mtu 1500, ipsec overhead 74(44), media mtu 1500 PMTU time remaining (sec): 0, DF policy: copy-df ICMP error validation: disabled, TFC packets: disabled current outbound spi: 22512A19 current inbound spi : 8F46C331 inbound esp sas: spi: 0x8F46C331 (2403779377) transform: esp-aes-256 esp-sha-hmac no compression in use settings ={L2L, Tunnel, IKEv1, } slot: 0, conn_id: 143024128, crypto-map: tunnel sa timing: remaining key lifetime (kB/sec): (4371840/26381) IV size: 16 bytes replay detection support: Y Anti replay bitmap: 0xFFFFFFFF 0xFFFFFFFF outbound esp sas: spi: 0x22512A19 (575744537) transform: esp-aes-256 esp-sha-hmac no compression in use settings ={L2L, Tunnel, IKEv1, } slot: 0, conn_id: 143024128, crypto-map: tunnel sa timing: remaining key lifetime (kB/sec): (4350795/26381) IV size: 16 bytes replay detection support: Y Anti replay bitmap: 0x00000000 0x00000001

Uptime for old vpn client

asa-firewall/pri/act# show vpn-sessiondb ra-ikev1-ipsec Session Type: IKEv1 IPsec Username : einsteina@vpn-tungrp1 Index : 3856 Assigned IP : 192.168.236.249 Public IP : 37.209.44.113 Protocol : IKEv1 IPsecOverTCP License : Other VPN Encryption : AES128 Hashing : SHA1 Bytes Tx : 667580222 Bytes Rx : 195368751 Group Policy : vpn-grp-p1 Tunnel Group : vpn-de-ol Login Time : 10:15:51 CEST Tue Nov 19 2013 Duration : 9d 3h:37m:37s Inactivity : 0h:00m:00s NAC Result : Unknown VLAN Mapping : N/A VLAN : none Username : leonardo@vpn-tungrp2 Index : 12473 Assigned IP : 192.168.244.151 Public IP : 145.253.227.158 Protocol : IKEv1 IPsecOverTCP License : Other VPN Encryption : AES128 Hashing : SHA1 Bytes Tx : 64670782 Bytes Rx : 49769295 Group Policy : vpn-grp-p2 Tunnel Group : vpn-ext-rsa Login Time : 09:07:46 CEST Wed Nov 27 2013 Duration : 1d 4h:45m:42s

Uptime for new vpn client (Anyconnect)

asa-firewall/pri/act# sh vpn-sessiondb anyconnect Session Type: AnyConnect Username : beck@vpn-tun-grp3 Index : 12579 Assigned IP : 192.168.236.194 Public IP : 84.163.80.247 Protocol : AnyConnect-Parent SSL-Tunnel License : AnyConnect Essentials Encryption : 3DES Hashing : none SHA1 Bytes Tx : 552426724 Bytes Rx : 264841827 Group Policy : vpn-grp-p3 Tunnel Group : DefaultWEBVPNGroup Login Time : 10:21:29 CEST Wed Nov 27 2013 Duration : 1d 3h:44m:57s Inactivity : 0h:00m:00s NAC Result : Unknown VLAN Mapping : N/A VLAN : none Username : baromarcu@vpn-tun-grp3 Index : 13405 Assigned IP : 192.168.238.212 Public IP : 91.14.67.250 Protocol : AnyConnect-Parent SSL-Tunnel License : AnyConnect Essentials Encryption : 3DES Hashing : none SHA1 Bytes Tx : 376838398 Bytes Rx : 153802768 Group Policy : vpn-grp-p3 Tunnel Group : DefaultWEBVPNGroup Login Time : 07:22:24 CEST Thu Nov 28 2013 Duration : 6h:44m:02s Inactivity : 0h:00m:00s NAC Result : Unknown VLAN Mapping : N/A VLAN : none

5.0 sniffertrace

The basic command is “capture”, after that you have to define the interface* (or the keyword any):
raise the packet-lenght to a higher value, if you need the payload from the packets!

myfirewall2/pri/act# capture capturename packet-length 1600 match tcp host 2.2.2.2 any eq 443 myfirewall2/pri/act# myfirewall2/pri/act# sh cap capture capturename type raw-data [Capturing - 0 bytes] match tcp host 2.2.2.2 any eq https

you can you access-list for more detailed traffic…

To export the sniffertrace to a pcap file use the command:

myfirewall2/pri/act# copy /pcap capture: tftp Source capture name []? capturename Address or name of remote host []? 3.3.3.3 Destination filename [capturename]? capturename.pcap !!!! myfirewall2/pri/act#

6.0 View logging on cli

The buffer size is limited and if the buffer is full the old logs will be overwritten.
To check your log settings issue the following:

myfirewall3/pri/act# sh run logging logging enable logging timestamp logging buffered alerts logging trap errors logging asdm debugging logging mail alerts logging from-address firewall@mycompany.com logging recipient-address network@mycompany.com level alerts logging host fw-trans 172.24.2.218 logging host fw-trans 172.24.2.219 logging permit-hostdown

Configure logging

Important commands are the:

logging enable
logging timestamp
logging host fw-trans 172.24.2.218
logging trap errors

Save the logs from buffer to file and after you can copy it to your tftp server.

myfirewall3/pri/act# logging savelog mylogs myfirewall3/pri/act# cd syslog myfirewall3/pri/act# dir Directory of disk0:/syslog/ 113 -rwx 2880 14:41:18 Sep 18 2013 mylogs 255426560 bytes total (181706752 bytes free)

Viewing the logs

Too see the buffer logs issue:

myfirewall3/pri/act# show logging

7.0 Inspection and asp-drop

These commands should be issued multiple times to see which counter actually increases, that can lead to a problem.
Issuing the command just once has not too much sence, since we do not know since when the counters show the actual values.

myfirewall/pri/act# sh service-policy set connection detail Interface germany: Service-policy: voice-http-map Class-map: voice-http-map Set connection policy: drop 0 Set connection advanced-options: max-mss-size Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Invalid ACK drops : 0 SYN-ACK with data drops: 0 Out-of-order (OoO) packets : 0 OoO no buffer drops: 0 OoO buffer timeout drops : 0 SEQ past window drops: 208 Reserved bit cleared: 0 Reserved bit drops : 0 IP TTL modified : 0 Urgent flag cleared: 0 Window varied resets: 0 TCP-options: Selective ACK cleared: 0 Timestamp cleared : 0 Window scale cleared : 0 Other options cleared: 0 Other options drops: 0

———————————————————————————————

myfirewall/pri/act# sh asp drop flow Inspection failure (inspect-fail) 14616790 SSL handshake failed (ssl-handshake-failed) 85 SSL received close alert (ssl-received-close-alert) 40 Last clearing: Never

———————————————————————————————

myfirewall/pri/act# sh asp drop frame Flow is being freed (flow-being-freed) 121 Invalid TCP Length (invalid-tcp-hdr-length) 1 No valid adjacency (no-adjacency) 36 Reverse-path verify failed (rpf-violated) 6990253 Flow is denied by configured rule (acl-drop) 864778803 Flow denied due to resource limitation (unable-to-create-flow) 1374 First TCP packet not SYN (tcp-not-syn) 471046343 Bad TCP flags (bad-tcp-flags) 46770 TCP data send after FIN (tcp-data-past-fin) 128 TCP failed 3 way handshake (tcp-3whs-failed) 1560684 TCP RST/FIN out of order (tcp-rstfin-ooo) 30625519 TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 9582 TCP SYNACK on established conn (tcp-synack-ooo) 8770 TCP packet SEQ past window (tcp-seq-past-win) 77478 TCP invalid ACK (tcp-invalid-ack) 53427 TCP ACK in 3 way handshake invalid (tcp-discarded-ooo) 5710 TCP Out-of-Order packet buffer full (tcp-buffer-full) 1 TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 5541 TCP RST/SYN in window (tcp-rst-syn-in-win) 326943 TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 769 TCP packet failed PAWS test (tcp-paws-fail) 1530 Expired flow (flow-expired) 284 ICMP Inspect bad icmp code (inspect-icmp-bad-code) 300 ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 633646 ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 1869 DNS Inspect invalid packet (inspect-dns-invalid-pak) 35 DNS Inspect invalid domain label (inspect-dns-invalid-domain-label) 628 DNS Inspect packet too long (inspect-dns-pak-too-long) 5044504 DNS Inspect id not matched (inspect-dns-id-not-matched) 1589860 Unable to obtain connection lock (connection-lock) 13 Interface is down (interface-down) 35 RM connection limit reached (rm-conn-limit) 136021 Dropped pending packets in a closed socket (np-socket-closed) 27886 Last clearing: Never

———————————————————————————————

8.0 Threat Detection (check the top talkers)

threat-detection configuration example:

myfirewall/pri/act(config)# sh run threat-detection threat-detection basic-threat threat-detection statistics host threat-detection statistics port threat-detection statistics protocol threat-detection statistics access-list no threat-detection statistics tcp-intercept

show commands threat-detection:

This command -IF activated- can give us really useful basic information about network flows, passing through the firewall.
Or if we have a performance problem with our internet connection, we can see who owns currently the line (whos head must be under the guillotine.)

myfirewall/pri/act# sh threat-detection statistics top ? access-list Enter this keyword to display top N access-list statistics host Enter this keyword to display top N host statistics port-protocol Enter this keyword to display top N port statistics rate-1 Enter this keyword to display top N's first rate statistics rate-2 Enter this keyword to display top N's second rate statistics rate-3 Enter this keyword to display top N's third rate statistics tcp-intercept Show statistics information for tcp intercept | Output modifiers

an example with port and protocol

myfirewall/pri/act# sh threat-detection statistics top port-protocol Top Name Id Average(eps) Current(eps) Trigger Total events 0-min Sent attack: 0-min Recv attack: 01 DNS 53 2972 3552 27100 1783308 02 LDAP 389 639 474 2549 383645 03 HTTP 80 162 152 14066 97668 04 NetBIOS-Name 137 160 193 8031 96239 05 HTTPS 443 131 85 11242 79013 06 Port-8191-65535 108 97 3513 64974 07 XMPP-SSL-Uno 5223 48 10 224 28884 08 SNMPTRAP 162 46 46 50537 27859 09 SYSLOG 514 36 32 9773 21995 10 MS-DS/SMB 445 30 40 45220 18030 1-hour Sent byte: 01 HTTP 80 25194299 24939838 0 90699477563 02 MS-DS/SMB 445 8260884 8225102 0 29739184085 03 Port-8191-65535 7038543 10227395 0 25338757949 04 LDAP 389 2334189 2347930 0 8403081060 05 Microsoft SQL 1433 1373774 1196909 0 4945586558 06 HTTPS 443 1318144 1258745 0 4745319756 07 HTTP-Alternat 8080 520889 566088 0 1875202977 08 DNS 53 430705 452066 0 1550540194 09 Port-7780 7780 264564 258684 0 952431991 10 Port-3380 3380 230415 12096 0 829497591 1-hour Sent pkts: 01 MS-DS/SMB 445 40571 41786 0 146057206 02 HTTP 80 22612 22957 0 81406406 03 Port-8191-65535 8834 11379 0 31804979 04 HTTPS 443 2528 2777 0 9101589 05 LDAP 389 1956 1954 0 7041854 06 Microsoft SQL 1433 1723 1527 0 6204903 07 Port-135 135 679 572 0 2445229 08 HTTP-Alternat 8080 414 447 0 1493298 09 DNS 53 393 387 0 1418233 10 ICMP * 1 281 365 0 1012609 1-hour Recv byte: 01 MS-DS/SMB 445 8241588 8308370 0 29669717400 02 HTTP 80 3148829 4675871 0 11335784733 03 Port-8191-65535 2908739 2644375 0 10471460696 04 Port-2055 2055 292614 281589 0 1053413852 05 SYSLOG 514 269208 323164 0 969151225 06 HTTPS 443 266550 283114 0 959582362 07 Microsoft SQL 1433 200255 173645 0 720919352 08 LDAP 389 149348 149286 0 537653925 09 SMTP 25 88919 104011 0 320111885 10 Port-135 135 76251 63814 0 274507044 1-hour Recv pkts: 01 MS-DS/SMB 445 40120 41355 0 144433605 02 HTTP 80 16028 17115 0 57703486 03 Port-8191-65535 7853 8933 0 28273380 04 Microsoft SQL 1433 1441 1281 0 5188677 05 LDAP 389 1329 1339 0 4785811 06 HTTPS 443 988 921 0 3559831 07 Port-135 135 694 588 0 2498510 08 SYSLOG 514 292 355 0 1051921 09 HTTP-Alternat 8080 272 289 0 981307 10 DNS 53 252 251 0 909608

and the top talkers list for hosts:

myfirewall/pri/act(config)# sh threat-detection statistics top host Top Name Id Average(eps) Current(eps) Trigger Total events 20-min Sent attack: 01 145.45.45.226 11 0 60162 13697 02 145.45.45.242 9 9 5657 11297 03 145.45.45.232 7 0 40045 9173 04 145.45.45.234 6 45 33096 7890 05 192.168.135.146 6 7 8214 7536 06 145.45.45.211 5 7 6109 6024 07 145.45.45.210 4 4 19756 5209 08 172.31.4.41 2 1 8 2620 09 172.16.2.224 1 1 202 2247 10 10.10.123.2 1 1 5 2048 20-min Recv attack: 01 192.168.135.136 3 3 1977 4278 02 172.16.28.6 1 2 0 2398 03 172.31.241.99 1 1 0 2160 04 145.45.45.211 1 0 830 1575 05 192.168.133.191 1 1 319 1293 06 10.16.200.27 1 0 17 1256 07 172.26.30.20 0 0 0 1004 08 172.16.1.10 0 0 216 903 09 172.16.22.11 0 0 1382 713 10 10.10.123.2 0 0 7983 653 ...

7.0 Backup and Restore

Backup command with tftp server:

myfirewall3/pri/act# copy running-config tftp Source filename [running-config]? Address or name of remote host []? 3.3.3.3 Destination filename [running-config]? Cryptochecksum: ee921f66 a8586880 f2d4fc17 c76933b2

For more info read my post: Migrate Cisco ASA configuration, certificates and private keys
Thats all folks!