CCTDE - Covert Channels and Tunnels Detection Engine v0.2 - README Copyright (C) 2003, 2004 Simon Castro - scastro@entreelibre.com =============================================================================== This file is part of CCTDE - Covert Channels and Tunnels Detection Engine v0.2 (C) Simon Castro CCTDE is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. CCTDE is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with CCTDE; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA =============================================================================== =============================================================================== GRAY-WORLD.NET / CCTDE ====================== The CCTDE program is part of the Gray-World.net projects. Our Gray-World Team uses the http://gray-world.net website to present the projects and publications we are working at. Our activity dwells in the NACS (Network Access Control System) bypassing research field and is related to computer and network security topics. =============================================================================== CCTDE ====== CCTDE is the first attempt at implementing the ideas presented in the Gray- World.net 'Covert Channel and Tunneling over the HTTP protocol Detection : GW implementation theoretical design' paper [1]. The main goal of this project is to provide a way to register and disclose informations leading to the detection of unauthorized tunnels and covert channels embedded into the HTTP protocol but the concept can be extended to other data flows and / or other high level protocols. Positioned between a mandatory http proxy server and the http clients (or before the NACS if no proxy exists), CCTDE is trying to detect if someone on the internal located network is using a CC|T (Covert Channel OR Tunneling) tool to bypass the NACS. Positioned in front of the corporate servers within the DMZ, CCTDE is trying to detect if someone located on the Internet is using server side tools such as WebShell [2] or Firepass [3] to run across the NACS boundaries. CCTDE can also be used as a convenient way to fingerprint CC&T (Covert Channel AND Tunneling) tools. NOTES ===== As a concept tool, there is at this time no click-and-play installation. CCTDE is actually designed for research purpose so I suggest you to read the source code and to play with it until it feeds your needs (if it ever manages to). At this time, CCTDE watches the OSI 3th and 4th layers but as my first target is to detect abnormal data streams embedded in the HTTP protocol, recording informations about HTTP requests/responses is at the top of my TODO list. If you have any question, don't hesitate to send me a mail or to post a message on the CCTDE board. A) DESIGN --------- CCTDE is currently designed as an analysis back-end to the Snort NIDS [4]. Snort acts as a network sensor and communicates with CCTDE using a standard Unix socket. CCTDE then reads Snort alerts and pcap packets from the Unix socket into memory. Finally, it becomes possible to correlate recorded data in order to detect specific network activities. 1. Recording packets into memory -------------------------------- Each time a packet arrives on the unix socket, the main select system call returns and the manage_tcp_connection() (tcp.c) is executed. This function manages the tcp network streams building flows identified by an index (calculated on the source and destination ip addresses and tcp port). Then, the consider_timeline() (storage.c) function is called to record informations about the current packet into memory. The consider_timeline() first searches the time period the packet belongs to, creates new time period if needed and call the store_flow() (storage.c) function. The store_flow() function creates two storage areas (one for the client and one for the server) and fill them with the packet informations (defined in (storage.h). These two elements are then used to add or update informations in the current client list (at the first level for the clients and at the second level for the servers) and in the server list. Memory storage description : -------------------------|---------------------- | | | o Client1-Server1 |----> o Client 1 | # |- | --#-> $ Server1-----| | # |- | --#---#-------------| | # | # # | | o Client1-Server2 |--| # # |-> o Server1 # |------#-> $ Server2 --| # # |------#---#-----------|-| # # # $ ... | | # # # | | # o Client2-Server2 |----> o Client 2 |-|-> o Server2 # |------#-> $ Server2 --| | # # |------#-----------------| # # # # o ... o ... o ... Current_Connections Client_List Server_List Various entry points to the recorded informations are available. For example, when a packet related to a current connection is processed, the current_connection list is first updated, then a backuped pointer to the corresponding Client_List element and Client_list's Server_List element is used (avoiding thus to browse the two whole lists) and a last backuped pointer stored into the server element is used to reach the related Server_ List element. The store_flow() function also manages the TopX lists creation and updates these lists in real time each time a packet is processed. The TopX lists link to clients in the Client_List or to servers in the Server_List. Note that adding/removing/updating/seeking informations in the various linked lists is done using the list.c framework functions. 2. Time periods --------------- When a time period ends (storage.c : consider_timeline() and storage_ timelines.c : manage_timelines()), the current client_list pointers are backuped in a new client_list beeing an element of a time_list and the main client_list, server_list and TopX_list are reseted (and memory is freed). Example : Start 10H00 - Period 30 minutes - Current Time 11H15 : -----------------------------|---------------- | | | | | o 10H30-11H00 o C1-S2 (start : 10H59) -----------------------> $ C1 | | | $ S2 | | | o C1-S1 (start : 11H15) ---> o C1 o 10H00-10H30 | | $ S1 $ C1 | | | $ S1 o ... o ... $ C2 Current_Connections Client_List Time_List When a current connection needs to access a client element to update the current informations, it uses the pointer presented in 1. Recording packets into memory to reach the related client which is now stored in a client list beeing part of the time list. Time_List elements can be removed from memory, flushed on disk etc. when their number exceeds a defined value (see C) 3.). However, they're kept in memory if an element of the Current_Connections list is linked to a client recorded in a time period. 3. Correlation and alerts ------------------------- Each time the main select call returns (when a packet arrives or each second), the correlate_and_alert() (correlator.c) function is called and the correlation process starts parsing the stored informations : First it tries to detect suspicious activities on the current_connection list (see C) 1.), then it parses the client and server lists (see C) 2.). B) HOW TO USE IT ---------------- 1. Build from sources --------------------- Download the snort sources and untar them in a directory, then: cd cctde-0.2 ./configure --with-snort-include={PATH_TO}/snort-2.1.1-RC1/ make The binary is src/cctde and configuration files are in conf/. 2. Snort Configuration ---------------------- The Snort configuration is located in the snort_conf directory. There are two Snort standard config files (config + rules) and a few starting scripts allowing to run Snort in logging mode and to process a tpcdump logfile. 3. CCTDE Configuration ---------------------- The CCTDE configuration is located in conf/cctde.conf. If you kill -SIGHUP the CCTDE process, the configuration file will be reloaded and directives starting with A_ will be updated. 4. Synopsis ----------- cctde - Covert Channels and Tunnels Detection Engine - v0.2.0 (21/02, 2004) Copyright (C) 2002,2003 Simon Castro - scastro@entreelibre.com Synopsis: --------- cctde -S snort_usock [-f config_file] [-p pidfilepath] [-l logfile]\ [-s start_time] [-R refresh] [-B not_before] [-A not_after] cctde -V cctde -h Parameters: ----------- cctde -S snort_usock Path to the Snort unix socket (ex: logs/snort_alert). cctde -f config_file Use the configuration file (if NS, use DEFAULT_CONF_FILE). cctde -p pidfilepath Log pid to pidfilepath/cctde.pid. cctde -l logfile Log to logfile. cctde -s start_time Do as if start_time was the current time (ex: 1074194932). cctde -R refresh Verbose refresh period delay (in secs). cctde -B not_before Don't take packet if tv_sec < not_before (ex: 1074194932). cctde -A not_after Don't take packet if tv_sec > not_after (ex: 1074194932). cctde -V Show version. cctde -h Show this help. Signals: -------- Sighup Reload configuration file. Sigusr1 Call display_cctde_infos. Sigusr2 Execute functions from A_COMMAND_FILE. 5. Displaying informations to stdout ------------------------------------ Using the Core displaying directives of the configuration file, you can set and update various displaying functions : o Display TCP connections (no,SYN/FIN,all packets); o Display Cold start and desync packets and timeout connections; o Display Snort messages (when the label is not yours : usefull when you log stdout to a logfile); o Display a resume (when the process ends, each -R period or when the process gets SIGUSR1) : $ Alert rules; $ Current connections; $ Clients (and associated servers) informations; $ Servers informations; $ TopX; $ Timelines (with or without recorded clients); o Display statistics about the current informations. C) SUSPECT FLOW DETECTION ------------------------- 1. Openned connections ---------------------- Each connection is monitored by CCTDE and stored as a network flow. When CCTDE ends processing a packet or each time the main select() system call returns, a detection function browse the recorded network flows and raise alerts if monitored values exceed configured thresholds. You can set using the A_MAX_UPCON directives in the config file : o The connection uptime (in seconds); o The number of packets received and/or sent; o The number of bytes received and/or sent; 2. Stored connections infos --------------------------- Each time a packet is received, CCTDE records informations about the client and the server into current period informations linked lists (See 3. Rotation periods). For each received packet or when the main select() returns, several specific detection functions are used to detect suspicious activies. Each detection function focuses on client or on server and browses an alert rules linked list. Two kind of alert rules may be used: ALERT_MAX_ - Monitor maximal values: ------------------------------------ An ALERT_MAX_ rule may be defined for clients or for servers in order to raise alerts if monitored values exceed next configured thresholds : o A number of TCP connection; o A connection uptime (in seconds); o A number of packets received and/or sent; o A number of bytes received and/or sent; o A number of servers (for clients only); o A number of 'silent' alerts; ALERT_ADV_ - Monitor advanced expressions: ------------------------------------------ An ALERT_ADV_ rule may be defined for clients or for server in order to raise alerts if relationships between monitored values exceed configured thresholds. The monitored values (MV) are : o A number of TCP connection; o A connection uptime (in seconds); o A number of packets received and/or sent; o A number of bytes received and/or sent; o A number of servers (for clients only); o A number of alerts; The two curently implemented relationships are : o if MV1 > x && MV1 > MV2 * y then generates an alert (Example: If the number of packets sent is greater than 100 and that the number of packet sent is double than the the number of packet received, then generates an alert). o if MV1 > x && MV1 / MV2 > y then generates an alert (Example : If the number of bytes sent is greater than 1Mb and that the number of bytes per connection is greater than 500Kb, then generates an alert). Alert rules are set with delay alert and percentage alert reminders. Each rule can be configured to alert once only (with the dal setting), to remind the alert each x seconds (with the dal setting too) or to remind the alert each time the monitored value change is greater than a percentage (with the pal setting). Alert rules can be 'silent' (with the silent setting) meaning that when the monitored value exceeds a thresholds or if a dal/pal setting is used, a counter is incremented but no alarm is raised to the operator. A rule can be silent for x alerts. Alert rules are represented by a unique number which is used if you want to dynamically add/display/remove a rule while the process is running (see D) EXECUTING DYNAMIC COMMANDS). 3. Rotation periods ------------------- Informations about clients and servers may be rotated on a periodic time. Each time the current period is over, the main informations structure is flushed into a specific period structure and the current period infos are reseted. You can then monitor specific values on a time line basis. Note that if a current network flow is active when a rotation period comes, then informations won't be updated for the current period struct but for the timeline period the connection started at. You can set the number of previous time period you want to keep in memory using the A_KEEP_TIMELINE directive and you can set what CCTDE has to do when previous period have to be removed using the A_FLUSH_TIMELINE directive (only free memory, free memory and display to stdout the related informations, free memory and write the timeline infos into a binary file (see E) BACKUPING / RESTORING) or free memory, display and write). 4. Top_X -------- Using the A_TOP_X directives in the configuration file you can generate and update on each packet clients and servers TOP X related to : o A number of TCP connection; o A connection uptime (in seconds); o A number of packets received and/or sent; o A number of bytes received and/or sent; o A number of servers (for clients only); o A number of alerts; If you start CCTDE without a TOP_X directive or if you want add / update / remove a TOP_X while running the process, you can use the dynamic command build_topX_[clients|servers]. D) EXECUTING DYNAMIC COMMANDS ----------------------------- Using the Sigusr2 signal you can tell the CCTDE process to look for commands in a specific command file. You can : o Display to stdout the clients infos, the clients and servers infos and the servers infos; o Display information for a specific client given its IP address; o Dynamically (re)build and/or display TopX clients and servers; o Backup the current informations (including timeline*S*) into a binary file. o Restore into memory the content of a previously backuped file; o Add/display/remove the alert rules. E) BACKUPING / RESTORING ------------------------ You can backup your information structure (current and timelines) in a file using the A_BACKUP_FILE directive (when the process gets SIGTERM) or each time you want using the dynamic backup_stats command. You can restore statistics from a backuped file using the dynamic read_stats_from command. F) WARNING ---------- This is a concept tool, so don't run it as root and so on... G) PLATFORMS ------------ Well, actually it runs on my Woody box. H) LICENCE ---------- GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. See the COPYING file for more informations. I) THANKS --------- My first thanks are going to Alex for all of the things we are discovering together each day and for his support and ideas for the CCTDE development and to the whole GW team. Next are flowing to Arun for his support since the publication of the cctde paper on his fantastic E-zine. I also thank Fryxar for his TcpStatFlow [5] release because he reminded me that 'code and see' was a good way ... And that thinking, thinking and thinking is sometime not enough... And Modu for his precious contributions and the 'spot that bug' party. Have a look at the ChangeLog file for further informations concerning the contributions. J) WEBOGRAPHY ------------- [1] - Covert Channel and Tunneling over the HTTP protocol Detection : GW implementation theoretical design - Simon Castro, July 2003. http://gray-world.net/projects/papers/cctde.txt You can find the last current working version of this paper in the doc/ directory of this archive. [2] - Web Shell Project - http://gray-world.net [3] - Firepass Project - http://gray-world.net [4] - Snort Project - http://www.snort.org [5] - Tcpstatflow Project - http://www.geocities.com/fryxar/ =============================================================================== CCTDE RESOURCES ================ A) Where is the latest CCTDE release ------------------------------------ You can get the last CCTDE release from http://gray-world.net or from a GW mirror : [France] : http://www.entreelibre.com/gray-world.net/ [Spain] : http://www.securitylabs.info/gray-world.net/ B) Discussion forum ------------------- You can post feature requests, bugs reports and discuss about CCTDE on an online forum located at : http://gray-world.net/board/ C) Patches ---------- Current version patches (if any) are announced on the discussion forum and available on: http://gray-world.net/projects/cctde/patches/ Simon Castro - scastro@entreelibre.com