This document discusses the development of test tools and RealFlow testing at a third-party testing lab. It describes various self-designed test tools developed to automate testing of devices like routers, switches, and wireless products. It also outlines how RealFlow testing uses real network traffic replay techniques to test products under realistic conditions captured from a beta testing site. Finally, it mentions some experiments conducted with Software Defined Networking and OpenFlow technologies.

1. ~Since 2002
1

2. Agenda
• Introduction
• Tool Development
• RealFlow
• Software Defined Network and OpenFlow
2

3. Introduction
• 3rd-Party Testing Lab
• Self-designed Tools
• RealFlow Test
• Research Team
• Look forward for the future
3

4. 4
Test Coverage
Area DUT/FUT Test Coverage
Bridging and
Routing
Bridging and
Routing
WLAN and VoIP
Ethernet L2/L3 Switch Functionality, Conformance, Stability,
RFC 2544/2889
NCSec UTM, Anti-Virus, IPS, SSL VPN,
IPSec VPN, P2P/IM Management
Functionality, Interoperability, Stability,
Session Capacity and Rate, Accuracy
WLAN and
VoIP
SOHO Router, DSL Router, IAD
Gateway, SIP Phone, SIP
Gateway, SIP Proxy, Access
Point
Voice Quality, Mobility, Functionality,
Interoperability, Stability, Session Capacity
and Rate

5. 5
Public Benchmark Event
2001
網路安全閘道器
網路頻寬管理器
網頁交換器
網路服務品質
Wireless LAN SOHO路由器
2002
VoIP產品
電子商務網站
2004
無線區域網路產品
網路安全閘道器
Internet內容遞送
2010
Network
Anti-botnet
Solution
Benchmarking
2005
VoIP互通性插拔大會
網路/內容安全產品
2006
入侵防禦系統
10GbE Ethernet Switch
VoWLAN 產品的語音品質
2007
P2P Friendly Properties of NAT
Wireless SIP Residential Gateways
2009
SOHO NAT於真實網路流
量下的穩定性表現
 Benchmarking, Workshop, Publish
2013
International SDN
Symposium Taiwan

6. 6
Industrial Customers

7. Self-Designed Test Tool
更新日期：
2010/2/10
NBL開發工具/系統目前已授權提供多家國內廠商
適合的待測物類型
(DUT)
工具/系統名稱(Tool
Name)
與預估開發人力
規格/特色
(SPEC & Features)
應用說明
(Description)
工具類型測試類型
SOHO Router
DSL Router
IAD Gateway
Security Appliance
In-Lab Live
Test(ILLT)
(6人月)
1. HW: PC
2. SW: Java and C programs
3. Continuously increasing real-world traffic
4. Support Bridge/Router/NAT mode
5. Automatically check the status of DUT
6. Stateless and Stateful Replay
[ 穩定性測試與除錯]
利用錄製、播放流量的技術，完整呈現真
實網路環境。在實驗室內部再造真實網路
流量環境，與Field Test相較之下，在測試
上更具彈性、網路流量涵蓋範圍更大、並
且更容易重製Bug。
流量產生、
結果解析、
自動化
Stability
Security Appliance
UTM
Anti-Virus
IPSP2P/IM
Network Forensics
PCAP Library
(3人月)
1. HW: PC
2. SW: Java and C progrmas
3. A database of PCAP files
4. Continuously increasing real-world traffic
5. Classified netowrk traffic such as P2P,
Attack
[ 網路流量資料庫]
流量資料庫，儲存各種未分類與已分類的
PCAPs。目前分類的類別包括P2P、Attack、
HTTP、SMTP、POP3、FTP等。
[ 協助漏判(FN)與誤判(FP)情況的改善]
可應用在"找出可能造成產品漏判與誤判
的網路流量"，進一步提供這些可疑的網路
流量(i.e. PCAP files)給研發人員參考，方
便找出其特徵值或行為模式。
流量產生、
結果解析、
自動化
Accuracy
QoS
SOHO Router
Security Appliance
UTM
Anti-Virus
IPSP2P/IM
Network Forensics
APP-Test
(5人月)
1. HW: PC
2. SW: Java and C programs
3. 支援多種Applications:
HTTP、FTP、Mail、SIP/RTP、IM、P2P
、Skype、Streaming (ex. Youtube、MLB
、NBA)
4. 自動分析DUT所產生的Log資訊來判斷
測試結果
[自動產生各種類型的應用]
透過使用者端應用軟體，自動產生各類應
用層網路流量，例如測試QoS時，可產生
各類streaming應用，檢查各streaming應用
是否有達到頻寬的保證。
[自動解析測試結果]
從各類待測物的output中(ex. Log)收集相關
資訊，來判斷本次的測試結果為Pass or
Fail
流量產生、
結果解析、
自動化
Functionali
ty

8. Self-Designed Test Tool (cont.)
適合的待測物類型
(DUT)
工具/系統名稱(Tool
Name)
與預估開發人力
規格/特色
(SPEC & Features)
應用說明
(Description)
工具類型測試類型
SSL VPN
SSL VPN Tunnel
Capacity
(2人月)
1. HW: PC
2. SW: Java and C programs
3. Generate more than 500 SSL VPN Tunnels
in a single PC
4. Support background traffic in the tunnel
such as HTTP, CIFS
測試SSL VPN Tunnel數量，使用一台PC即
可建立超過500條以上的Tunnel，並且在每
條建立起來的Tunnel中可傳送應用層流量，
如網頁的瀏覽、網路芳鄰抓檔。
流量產生、
結果解析、
自動化
Performanc
e
PPTP/L2TP VPN
PPTP/L2TP VPN
Tunnel Capacity
(2人月)
1. HW: PC
2. SW: Java and C programs
3. Generate more than 500 PPTP/L2TP VPN
Tunnels in a single PC
4. Support background traffic in the tunnel
such as HTTP, CIFS
測試PPTP/L2TP VPN Tunnel數量，使用一
台PC即可建立超過500條以上的Tunnel，並
且在每條建立起來的Tunnel中可傳送應用
層流量，如網頁的瀏覽、網路芳鄰抓檔。
流量產生、
結果解析、
自動化
Performanc
e
Switch
IGMP Snooping
(2人月)
1. HW: SmartBits
2. SW: Tcl/Tk programs
3. Support IGMP v1/v2/v3
測試switch在IGMP snooping的功能是否正
確。流量產生、
結果解析、
自動化
Functionali
ty
General
MIB-AutoChecker
(3人月)
1. HW: PC
2. SW: Net-SNMP and Perl programs
3. MIB Object Accessibility
4. MIB Definition Customization
5. SNMP v1, v2c and v3 Support
即自動化MIB 物件存取技術。
MIB AutoChecker 可匯入自訂ASN.1的物件
檔案，它以Net-SNMP 動作為基礎，自動
地展開指定的數個群組的所有物件並加以
存取，記錄下所有訊息使得測試者很快就
找出有問題的部份。
以Net-SNMP為基礎，可以在MIB中指定的
結點與其下的sub-tree自動化存取物件。
自動化Functionali
ty

9. Self-Designed Test Tool (cont.)
適合的待測物類型
(DUT)
工具/系統名稱
(Tool Name)
與預估開發人力
規格/特色
(SPEC & Features)
應用說明
(Description)
工具類型測試類型
Switch/Router
ANVL-MultiParser
(3人月)
1. HW: PC
2. SW: ANVL and Perl programs
清楚解析出ANVL的測試結果，讓研發人員
可以快速找出產品的問題點。結果解析Conforman
ce
Switch/Router
Performance Test
Suites(PTS)
(3人月)
1. HW: SmartBits
2. SW: Tcl/Tk and PHP programs
3. 圖型化使用者介面：DUT Configurator、
SmartBits Configurator、Test Editor、Test/
Result Viewer
4. 自動化組態待測物：
5. 自動化收集測試結果及產生測試報告
Layer 2/3 Switch的效能測試自動化。
自動化Performanc
e
Switch/Router
Auto-ANVL
(3人月)
1. HW: PC
2. SW: ANVL and Perl programs
3. 圖型化使用者介面：DUT Configurator、
IxANVL Configurator、Test Editor、Test
Viewer、Result Viewer、Preference
4. 自動化組態待測物
5. 自動化執行Test Suite及測試案例
6. 自動化剖析測試記錄檔及分析結果
7. 自動化產生測試報告
Layer 3 Switch的符合性測試自動化。
自動化Conforman
ce
SOHO Router
DSL Router
WebUIAutoChecker
(2人月)
1. HW: PC
2. SW: Java programs
3. Web GUI invalid value auto check
4. Suppoer Firefox and IE
Web GUI 自動檢測工具，將Web GUI輸入
各種錯誤值的測試自動化，提高測試效率
與準確性。自動化Functionali
ty

10. Self-Designed Test Tool (cont.)
適合的待測物類型
(DUT)
工具/系統名稱(Tool
Name)
與預估開發人力
規格/特色
(SPEC & Features)
應用說明
(Description)
工具類型測試類型
WiFi Phone
WLAN Card
Access Point
WLAN Mobility Test
System(WMTS)
(6人月)
Mobility Test Scenarios:
1. HW: Azimuth
2. SW: Tcl/Tk
3. Distance Test for Computer Adapter
4. Distance Test for Mobile Phone
5. Motion Adaptation Test for Computer
Adapter
6. Motion Adaptation Test for Mobile Phone
7. Motion Roaming Test for Computer
Adapter
8. Motion Roaming Test for Adaptation for
Mobile Phone
9. Switch Roaming Test for Computer
Adapter
10. Switch Roaming Test for Adaptation for
Mobile Phone
Programmable Channel Model:
11. Free Space and Typical
Home/Office/Street Channels
12. More channels can be added by the
programmable parameters
Traffic Generation/Analysis:
1. HW: Abacus and PC
2. SW: IxChariot and Perl programs
3. Ixia IxChariot® (with the scripts and QoS
selective)
4. Abacus® for PESQ/PSQM analog testing
5. NBL traffic analyzers for link adaptation
and re- association
自動化WLAN移動測試系統（WMTS）可
進行Roaming的延遲分析。提供具有可參數
化channel模擬與WMM支援的Distance
Adaptation和Roaming測試Scenarios
For Distance, Adaptation, and Roaming
scenarios with a parameterized channel
approximation and WMM support
The WLAN Mobility Test System (MTS) for
Azimuth W-Series Platform® version 1.3 can
support 8 novel mobility test scenarios and
integrate standard traffic utilities like
IxChariot® and VQT®, including QoS
enabled configuration. A programmable
channel model is introduced for the mobility
control to offer the approximation of signal
fading for the in-lab wireless testing, which is
more stable and repeatable than the outdoor
testing.
結量產生、
結果解析、
自動化

11. from Lab Test to RealFlow Test
11

12. Solutions
12
真實網路
流量測試
Real
Network
•Users
•Beta Site
Replay
Tech.
•Capture
•Replay
Test
System
•PCAP Library
•In-Lab Live
Test (IL2T)
RealFlow

13. Beta Site
 Zone 1
 End-user software
 Zone 2
 Ethernet L2/L3 Switch
 Wireless AP
 Zone 3
 Core Router
 Zone 4 (Inline, one-in-one-out)
 UTM, IPS, Anti-Virus, QoS
Firewall
 Zone 5 (Sniff)
 Network Forensic
 Anti-Malware/Botnet
 Zone 6 (ILLT)
 SOHO Router, Home Gateway
 Broadband Gateway
 DSL Router, IAD Gateway
13

14. RealFlow Certification
•Function
•Performance
Phase
1
Phase Stability
2
14
USG 2000
ZyXEL Security
Appliance
DIR-300
D-Link SOHO Router
DIR-615C1
D-Link SOHO Router
Mini guard
Lionic Security
Appliance
DIR-655
D-Link SOHO Router

15. Experiment in OpenFlow

16. Service
Solution
• Solution Prototype
• SDN Enabled Wi-Fi
Tool
• Test Tool develop and Automation
• Cloud based conformance test tool based on oftest
• SDN Enabled Testbed
Service
• Test/Benchmark of SDN
• Device and Apps
• OpenFlow Certification
16

19. Approved Labs

20. Why SDN - Requirement
• Too many different kinds of devices with different operation
method for 1 or 2 persons
• Devices with high cost
• Impossible to fully match the campus network administration
policy

21. What solution we need?
• An unify control message and method to integrated different
devices from different vendors
• An easy way to develop the related service/software and deploy
the suitable devices with suitable spec. and cost
• An flexible method to integrate the other services and devices

22. Advantage of OpenFlow (Manage)
• Open source with low cost
• A generic protocol to control different kinds of devices
• The network administrator could choose the suitable HW devices
and policy/SLA for each site and account(person)
• Don’t need to add more human resource to manage more
devices (cost again)

23. Advantage of OpenFlow (Technical)
• Control the forwarding table of switch/router and wireless devices
from different vendors with the same message
• Control the forwarding policy by the software development of
each site with different rule.
• Easy to control the traffic to integrated with other services

24. Campus Wi-Fi Solution

25. Campus Wi-Fi Solution (cont.)

26. Example: AuthWorkflow

27. Example: System control
Web UI
RYU
AP
REST APP
OpenFlow
OpenFlow
CURL
OpenFlow 1.3
Experimenter message
RESTAPI POST

28. Feature List
• Auto-Provisioning and Configuration through experimenter
message
• Multi-path routing/switching with fail-over and switch-over
• Switch and AP control and management
• Multicast Routing
• Streaming Control
• Wi-Fi AP-STA Association Load Balance
28

29. Test

30. SDN Test (concept)
APP/Controller
Topology Generator
•Automatically
•Physical devices
•Simulator
Monitor
•Flow Checker
(App for Testing)
Traffic Generator Test bed
•Trackable mechanism
Test Script
•Test Methodology
•Test Plan
Test Controller
Test Portal
Test bed
Report Generator
Monitor
Test Script
Topology Generator
Traffic Generator

31. SDN Test (Lab and Auto)
SDN Network Device
1. Execute Automation Test
Program
Monitor
System information collector
CPU Loading,
Link utilization,
counters,
client simulator
Packet Generator
Scriptable
Error packets
Attack traffic
Trackable packets and mechanism*
L4+ level packet generator
Topology generator
GUI based traffic tracker
Lab test automation
SDN-enabled Layer 1 switch
Flow entry add/remove
Controllable VM migration
Test plan/case/script/reporter*
SDN-enabled
Layer 1 switch
2. Topology Generator generate a
topology
3. Monitor Start to gather
information DUT & SUT
4. Start to generate the required
packets
5. If any Link Needs to be
Removed During Test Period,
the L1 Switch can Do it.

32. SDN Test (BetaSite)
Traffic Reproducer
Real-time time machine
Reproduce defect of Apps
Reproduce traffic
Reproduce topology by simulator and emulator
Lab Test Automation
Flow entry add/remove
Controllable VM migration
Test plan/case/script/reporter*
1. Execute Automation Test
Program
Topology
Generator
2. AuGtUoIm baatsiceadl ltyr aGfafitch terra cthkee r
Topology Information
3. Monitor Start to gather
information DUT & SUT
Monitor
System information
collector
CPU Loading,
Link utilization,
counters,
client simulator
4. Traffic Start
SDN Network Device
5. 1 Capture traffic
5. 2 Replay traffic to reproduce
bugs
PKT
Packet
Genera
tor

33. oftest GUI

34. oftest GUI (cont.)

35. oftest GUI (cont.)

36. 基於SDN和Cloud架構之
無線/寬網技術與服務
SDN-enabled Cloud-based Wireless and
Broadband Network Technologies &
Services

37. Project Goal
• Establish the first end-to-end test platform for SDN in Asia
• Contribute to the open networking (SDN related) standards and
set up international reputation
• Cultivate human talents with SDN related technology and service
knowledge and experience to build an SDN industry in Taiwan
• Strengthen SDN IP portfolio for Taiwan industry
• Enhance the competitive advantage of Taiwan ICT industry

38. Technology Layered Architecture
38
①CHT (Wireless/BB/Security) ②CHT (Cloud/Billing/IoT) ③D-Link ④
IoT/M2M ②
Cloud
Technologi
es ② ④ ⑧
Northbound API ①③④⑥⑧
Controller
Multi-controller ④
Southbound API
Switch
Testbed and Methods ④
Campus Field Trial ①②③④
Northern TW Field Trial ①②
③④⑥⑧
Service APP OA&M
Network APP
Security ④⑧
Performance ④
Accounting ①
Configuration ①③⑥
Fault
EstiNet
⑤MediaTek ⑥ Arcadyan ⑦ Xinguard ⑧ Inventec
Enterprise/
Campus
③ ⑥ ⑧
Broadband
Technologi
es ① ⑦
Network Virtualization
V&T
SDN Chip
⑤
SDN Switch OS
④
Wireless
Technologi
es ①
Smart Data Pricing
②
Common controller ①

39. Subproject E – Integration and Field Test
39
小型SDN環境
(小型OpenFlow Switch)
Internet
OpenFlow
WiFi
有線網路
無線網路
OpenFlow CPE
終端SDN環境
雲端資料中心
雲端SDN環境
x4 x2
行動網路
UE
TL Core
Network
小型SDN環境
(小型OpenFlow
Switch)
中型SDN環境Internet
(中型OpenFlow
Switch )
OpenFlow
WiFi
有線網
路
無線網
路
SDN
Controllers
OpenFlow CPE
終端SDN環境
雲端資料中
心
x4
x2
雲端SDN環境
行動網
路
UE
NCTU Core
Net.
emulator
小型SDN環境
(小型OpenFlow
Switch)
Internet
SDN Core Switch
(中型OpenFlow Switch )
OpenFlow
WiFi
有線網
路
無線網
路
SDN
Controllers
OpenFlow CPE
雲端資料中
心
x4
x2
雲端SDN環境
CHT
中型SDN環境
(中型OpenFlow
Switch )
NCTU NTHU
SDN Master
Controllers
SDN Controllers
中型SDN環境
(中型OpenFlow
Switch )
Internet
SDN APP
SDN APP
SDN APP

40. SDN Enabled Wi-Fi Solution
40

41. Outlines
• System Architecture
• Topology Example
• Environment Description
• Features vs. Components
• Vendor-Defined Feature Specification
• Feature: Auto Provision Function
• Feature: Authentication Portal Function
• Demo: OAM Web System
41

42. System Architecture
42

43. Topology Example
43

44. Environment Description
(1/3)
• SDN Controller
Ryu SDN Framework 3.8
Ubuntu 12.04
• SDN Access Point
Device: TPLink TL-WR1043ND V2, AP222, …
OS: OpenWRT trunk
Patch: ofsoftswitch13
• SDN OAM Server / SDN Portal Server
PHP 5.5
Apache 2.4.10
• SDN Database Server
MySQL 5.5

45. Environment Description
(2/3)
• Ryu SDN Framework
Python-based Project
Written in Python language
Environment
OS: Ubuntu 10.04 or higher
Network simulation tool: Mininet
Features provided
Component-based SDN framework
Support OpenFlow v1.0, v1.2, v1.3, v1.4
Support various protocols for managing network devices, such as
OpenFlow, Netconf, OF-config, etc.
Provide REpresentational State Transfer (REST) service
45

46. Environment Description
(3/3)
REpresentational State Transfer (REST) Service
46
Controller
Ryu App
data in
JSON
format
REST API
Operations via HTTP
request with URL.
• GET
• POST
• DELETE
• PUT
PHP
HTTP request
OAM Web UI
curl HTTP request
Mininet
(Simulated network)
OpenFlow
Switch
(e.g., Access Point)

47. Features vs. Components
SDN Controller
Component
SDN Access Point OAM App
System Management
Generic Wireless Control
Auto Provision Function
Authentication Portal
Function
Device Information
Management
Channel Non-overlapping
Quality of Service
Roaming
47

48. Vendor-Defined Feature
Specification (1/2)
• Vendor-defined features
Implemented with “Experimenter Messages”
Experimenter Message
 An optional field (padding) in OpenFlow protocol
 Provide SDN vendors to develop self-defined functions and services
• Experimenter Message Format
48
Header:
EXPERIMENTER_ID
(8 bytes)
EXP_TYPE
(8 bytes)
Data:
EXP_SUBTYPE
(1 byte)
PAYLOAD
(255 bytes)
• The format should be defined in both controller and access points.
• Corresponding handler function will be triggered according to the EXP_TYPE and EXP_SUBTYPE.

49. Vendor-Defined Feature
Specification (2/2)
49
Experimenter Message Type and Subtype List Example

50. Feature: Auto Provision Function (1/3)
50
System Scenario

51. Feature: Auto Provision Function (2/3)
51
Trigger Proxy Mode of Configured AP Automatically
Trigger/Close Proxy Mode of Configured AP Manually by Administrator

52. Feature: Auto Provision Function (3/3)
52
Relay the Configuration Request from Unconfigured AP and Response

53. Feature: Authentication Portal Function (1/3)
53
Topology Example

54. Feature: Authentication Portal Function (2/3)
54
Message Flow

55. Feature: Authentication Portal Function (3/3)
55
Demo Portal Webpage

56. Demo: OAM Web System
[Demo URL] http://140.113.243.175/login.php
56
Login Webpage &
Dashboard Webpage