What happened at IETF98 in Chicago, March 26-31?

Expecting an usual, not particular special IETF meeting, the 98th meeting two weeks ago in Chicago was actually quite exciting, in a positive sense. A lot of work, relevant to the architecture as well as measurement objectives of the MAMI project, made good progress and fostered interesting, still on-going discussions.

The meeting started off with the IP Performanec Metrics (IPPM) session Monday morning. A document on In-situ Operations, Administration, and Maintenance (IOAM) proposing a data model for telemetry and measurement data that can be applied to different (tunneling) protocols was discussed and received positive feedback from the group. This work is highly relevant to MAMI given measurement is one of the basic use cases for the Middlebox Cooperation protocol (MCP).

In the afternoon, there were several talks on middlebox benefits, state management, and privacy in the tsv-area meeting. Dave Dolsen presented draft-dolson-plus-middlebox-benefits and Brian Trammel (remotely) presented draft-trammell-plus-statefulness. Both drafts created quite some interest as well as discussion – any information that is exposed by an endpoint must first demonstrate benefits when consumed by the network.

Tuesday morning the IRTF Measurement and Analysis for Protocols (maprg) had a long slot with a number of presentations on security and privacy relevant topics. maprg usually sends out a Call for Contributions a couple of weeks ahead of the meeting. This time the group had a large number of very interesting submissions and could finally accept only about half of them to fit in the 2.5h slot. The talks presented data on DSCP, ECN, IPv6, Let’sEncrypt, broadcasted hostnames, weak keys in HTTPS, and censorship detected by the OONI project.

In the last session on Tuesdays Tommy Pauly presented Post Sockets in the Transport Services (taps) working group which is common work with the MAMI project updated after an one day workshop held in Feburary in Zürich.

Thursday started with the QUIC meeting were Mirja Kühlewind presented two drafts on Applicability and Managability as well as a proposal by Brian Trammell to add a packet number echo to the public QUIC header for RTT measurements. The draft received positive feedback to address one of the charter milestones and adoption will be confirmed on the mailing list. The proposal to add a packet number echo was lively discussed without reaching consensus yet. There was broad support for this addition but there were also privacy concerns that needs further discussion; similar to all information that will be exposed for network use in the header. QUIC will hold an interim meeting on June 6-8 in Paris.

Later that day, the Multipath TCP working group discussed the use of MPTCP for bandwidth aggregation in network scenarios where one low bandwidth fixed line link (DSL) is bonded with a high speed mobile link (LTE). Several approaches to address this use case have been proposed but there was so far no real consensus to move any of them forward. This time the discussion was led by the chairs attempting to compare the different proposals on a high level allowing the group to move forward. While there were also quite some people who indicated that they think this work should not be taken up by the IETF at all, a larger group of people were interested in this work and there was a clear indication for one of the proposals which is a solution that only requires a small addition to the SYN payload on the bonding link. The MAMI project is also focusing on this use case, using the MCP to signal preferences of the endhost to the bonding boxes in the network.

Thursday afternoon finally also the new MAMI T-shirts arrived (see photo above). Unfortunatly, they came only in to action during the bits-and-bytes and Friday morning that is traditionally rather quite. However, the next event is comig and the next IETF is already on the horizon in July in Prague! See you there! Good bye, Chicago!

Posted in Uncategorized | Leave a comment

Frist Prize at IETF98 Hackathon for “connection identifier” in DTLS group!

On Saturday and Sunday at the IETF there always is a Hackathon. This time at IETF-98 members of the MAMI consoritum were working on the implementation of the “connection identifier” in DTLS, a fairly recent proposal Hannes Tschofenig, Nikos Mavrogiannopoulos, and Thomas Fossati brought to the TLS working group.
The problem this proposal addresses is that an end-to-end DTLS session may silently die because an on-path Network Address Translation (NAT) middlebox dropped state after a (relatively) short period of quiescence. There is known trouble with UDP, as transport used by DTLS, that in-network state for this kind of traffic tends to vanish much more quickly than its TCP counterpart. As an example, the default timeout settings in the latest Linux kernel are 5 days for TCP and only 3 minutes for UDP — that is, three orders of magnitude!

© Stonehouse Photographic/Internet Society
IETF Hackathon, Chicago 25/03/2017

Obviously, there is a good reason for that: since UDP is connectionless, layer 4 devices have no way to possibly track a “connection” other than deep inspecting the flow, which is a pretty expensive activity. So it’s simpler for it to leave the onus of proving that a given UDP 4-tuple has an associated connection to the endpoints, by forcing them to regularly move bytes across. This state of affairs is clearly far from ideal for DTLS because, when a timeout happens in the NAT box, the victim endpoints need to re-negotiate a new crypto session context.

While this is generally annoying, it becomes even more nasty in cases where the client is a resource constrained, battery operated IoT device that woke up from its sleep cycle only to find its session doesn’t work anymore… The second nuisance is that the state that is dropped on the NAT box immediately becomes dead state in the server, consuming precious resources in vain. So, one popular workaround is to create synthetic “keep-alive” traffic, for example using the TLS heartbeat extension. This technique is a) not very robust (choosing the right keep-alive clocking depends on many external factors), and b) certainly, not affordable at smaller device scales, where waking up the “thing” to keep the NAT binding happy has the potential to quickly drain its battery.

© Stonehouse Photographic/Internet Society

Our solution – provide a connection identifier!  We propose to add a 32-bits blob that does one very simple thing: it decouples the DTLS session from the underlying 4-tuple, making it possible for the endpoints to dispatch incoming UDP traffic to the correct crypto session independently of any change in the underlying UDP address.

Sounds simple, right?  Yes, conceptually — apart from the birthday paradox hitting hard at large scales (see https://tools.ietf.org/html/draft-mavrogiannopoulos-tls-cid-00#section-4). What we discovered at the Hackathon is that backporting it to an existing stack (ARM’s mbedTLS) can be more difficult than expected if one needs to maintain API compatibility.

Another practical complication is signaling the wire format change to the receiving endpoint so that it can parse the incoming frame correctly.  This is easy when you can make breaking changes (for example, when transitioning from 1.2 to 1.3).  Not as much if you have to maintain backwards compatibility with existing and deployed versions of the protocol. We have been discussing a couple of possible solutions around this issue — namely: using the Version field, or moving to an extensible record layer format.  We are still undecided on what “the right” approach would be.

Note that there are interesting privacy implications related to using a visible and potentially long-term identifier due to the obvious linkability properties of such a construct. Even if not all the use cases are problematic in this respect, some of them are, and thus we designed (but not yet implemented) a privacy friendly connection identifier based on HMAC-based One-time Password (HOTP) which can be rotated at client’s will at any point in time.

Enough with the babbling! If you read up to this point you will be glad to know the hackathon judges rewarded our herculean effort with the first prize. Yay!

Posted in Uncategorized | Leave a comment

Post Sockets Workshop

MAMI partner ETH Zürich hosted a small workshop last month to discuss a topic central to our vision: how to move network programming beyond the 1980s-era BSD sockets API.  The workshop had two main goals. First, to sync up and share concepts about abstract programming interfaces among researchers and developers working on future networking stacks. Several partners from our H2020 sister project NEAT, which has a more explicit focus on how applications see the network and transport layer, were in attendance.

Our second goal with the workshop was to further develop the concepts in the Post Sockets abstract programming interface. Post Sockets is explicitly asynchronous and message oriented, which is a better match to most network applications and all future transport protocols than the single-streamed SOCK_STREAM service offered by TCP. It breaks the hard relationship between a connection (called a “carrier” in Post, since it carries messages) and the path(s) carrying it, and separates short-term transport layer state (“transients” which hold transport-protocol specific information such as current sequence numbers and flow control windows) from longer-term state (“associations” which hold state such as cryptographic resumption parameters), allowing generalized fast resumption of sessions for commonly-communicating endpoint pairs. Like the NEAT API, it is intended to be transport-protocol independent, and interoperating with NEAT’s policy engine and transport protocol selection machinery is a goal of MAMI’s Post Sockets work.

Our current work on Post Sockets in MAMI focuses on implementation details: how message carriers, associations, and transients bind to underlying transport protocol implementations, specifically in situations with multiple candidate transport and network layer protocols, and where network address translation makes rendezvous non-trivial. Tommy Pauly from Apple will present Post Sockets at the IETF taps session next week in Chicago (checkout remote participation). Also watch this space for future announcements.

We thank the workshop attendees for a very productive conversation, and look forward to working together in the future to drag network APIs into the twenty-first century!

Posted in Uncategorized | Leave a comment

Path Transparency Observatory (PTO) goes live!

One of the core objectives of the MAMI project is to perform measurements and collect measurement data that can help to quantify middlebox impairments on the Internet. While various impairments on different layers of the protocol stack have been detected over the years, the provided measurement data are often not sufficient to assess how much of a problem there actually is. This information is needed to guide development of protocols extension or new protocols such as the Middlebox Cooperation Protocol (MCP) as proposed by MAMI.

Over the last year the MAMI project developed and implemented the Path Transparency Observatory (PTO), an open-source and publicly accessible repository for measurements of path transparency. An Internet path between a vantage point and a target is consider transparent if no impairment has been observed on that path and therefore the packet was successfully received without modification. The goal of the PTO is to collect data from different sources usually in different formats and represent them in a comparable way. To achieve this we apply a pre-processing step that transform the raw data in observations of a network condition c on a path p during a time interval t.

In parallel we also developed PATHspider (see below), an active measurement tool for A/B testing which now also integrates upload facilities for the PTO. We used and are using PATHspider as well as other measurements tools such as tracebox to continuously run measurements to test TCP extensions such as ECN or TFO as well as impairments on protocols on other layers such as the use of the DiffServ codepoint. Currently, only observations for our ECN measurement campaigns of the last couple of month are available over the public PTO interface. However, we have collected more raw data and are currently running the needed processing steps to successively release more observation data to be publicly available in the PTO: https://observatory.mami-project.eu/

For more information check out the ANRW PTO paper and Brian Trammell’s talk at the CAIDA AIMS workshop today! And of course, we will announce on twitter when more data is available!

Posted in Uncategorized | Leave a comment

2 Years of PATHspider Development

The current incarnation of PATHspider‘s first Git commit occurred on the 22nd of February 2015, now just over 2 years ago.

Since then, PATHspider has gained its own target list resolvers and has generalised for performing far more than just the ECN-dependent connectivity failures it was originally created to measure. Through plugins it can now perform measurements relating to other TCP features such as TCP Fast Open, and IP features such as DiffServ.

Active development of PATHspider continues, with the roadmap for PATHspider 2.0 provisionally laid out in the GitHub issue tracker. This includes:

  • seamless MONROE integration,
  • support for workflows integrating with the MAMI Path Transparency Observatory,
  • …and of course, new measurement plugins.

The following video is a visualisation of the last two years of work, featuring commits from the PATHspider authors: myself, Brian Trammell, Elio Gubser, Mirja Kühlewind, Andreas Germann, Piet De Vaere and Ana Custura.

Posted in News | Leave a comment

Technical Plenay in Seville

On Feb 7/8 the MAMI project met in Seville for the third technical plenary meeting. Besides the great Spanish food and tapas crawl organized and guided by our technical project coordinator Diego Lopez, we made good progress on the technical aspects of the of the project.

We discussed next steps on PATHspider, both using PATHspider for more measurements as well as the development of new features in PATHspider. Currently the integrating into the MONROE testbed is underway. Further, potential integration with OONI was discussed to extend censorship measurements to low layer mechanisms. We plan for a new release in the next couple of month. Check out GitHub for the current implementation status and details on planned extensions.

The MAMI middlebox taxonomy that will be published end of June in D2.1 feeds into the development of PATHspider and other measurement tools by providing input to define a formalized test description and conditions to measured. Measurement data describing conditions of path transparency observations will soon be publicly available over the Path Transparency Observatory frontend webpage.

The MAMI project is also working on the specification of the Middlebox Cooperation Protocol (see draft-trammell-plus-spec), planning an endpoint implementation using QUIC as transport layer protocol for an HTTP/2 web service example use case and a fd.io based middlebox implementation supporting network diagnostics and differential treatment for low latency services.

Posted in Uncategorized | Leave a comment

Slow going for TCP Fast Open

As part of our continued effort to measure Internet path transparency with PathSpider, we’ve taken a look at the state of deployment and potential impairments to TCP Fast Open. TCP Fast Open is an extension to TCP that allows data to be placed on the first (SYN) packet of the TCP handshake, eliminating a round-trip time from TCP connections. It uses a TCP Option to exchange a cookie to be used on subsequent fast open connection attempts, to reduce the risk of TFO-based denial of service attacks.Interference with this option could cause path impairment of TFO, and indeed Christoph Paasch has reported that this is the case on about 20% of the access networks he observed.

We set out to measure possible impairment on content provision networks and in the Internet core, and found instead that TFO deployment on popular Web servers is mostly limited to Google, who invented TFO. Of 939,680 web servers taken from the Public Targets List (PTL), only 866 (0.092%) negotiated TFO in measurements taken this week. 690 (about 79.7%) of these are Google servers. Compared to measurements taken in October 2016, we see no appreciable change; then 563 of 635,681 web servers (0.086%) negotiated TFO. This is unsurprising, given that TFO requires significant changes to both client-side and server-side application logic as well as kernel support on both endpoints, we expect slow adoption compared, e.g., to ECN.

The story on DNS, where TFO is part of an effort to improve query privacy by using TLS and TCP for DNS, is similar: of the 53,267 authoritative name servers taken from the PTL, 56 (0.105\%) negotiate TFO, only three of which are not Google name servers; two of those three use an experimental ID, and fail to ACK data on the SYN.

Posted in Uncategorized | Leave a comment

A Public Targets List

Comparability of measurements and repeatability of experiments is key to science, and active Internet measurement studies are no different. Active measurement studies need not only open descriptions of the experiment (preferably with open source code and analysis, as we use in the MAMI project) but a comparable set of targets against to run experiments.

For some time, the Alexa top million web domains list has acted as a de facto standard of this comparable set of targets, and we’ve used it ourselves in several studies. Recent announcements by Amazon have made it clear that the Alexa top million list will no longer be freely available, and the announced cost for API access have made periodic resolution of the list too expensive for most active measurement studies. Therefore, we have started compilation of a public targets list to replace this de-facto standard.

The initial MAMI Public Targets List is available in GitHub; see the README there for details.

Posted in Uncategorized | Leave a comment


MAMI was represented by ETH Zürich at the 97th meeting of the Internet Engineering Task Force in Seoul.


The biggest news this time around was the first meeting of the QUIC working group, which will standardize a next-generation, encrypted transport protocol encapsulated in UDP based on Google’s QUIC and TLS version 1.3. Brian Trammell presented a concept for a transport-independent state machine for middleboxes at the meeting, to start the discussion about how QUIC’s wire image should interact with on-path devices, both present and future. While it’s not clear how much of the proposed transport protocol mechanism will be adopted into QUIC, discussion during and after the working group meeting has led to further refinement thereof.

Measurement and Analysis for Protocols (MAP) RG

Measurement and Analysis for Protocols (MAP) research group meeting

Mirja Kühlewind chaired a meeting of the Measurement and Analysis for Protocols Research Group (MAPRG), the first meeting since the group was officially chartered. The four presentations included techniques for passive delay measurements, a study of broadband access peformance using M-Lab, a study of the performance gain associated with HTTP2, and a characterization of traffic rate policing in the Internet.

Post Sockets, the API concept atop MAMI’s flexible transport layer (FTL), was discussed at the TAPS working group meeting. Tommy Pauly of Apple, a co-author of the Post draft, presented a quite similar approach. Post is very much a work in progress, but we’re happy to see broad interest in the concept, and look forward to developing it further with a broad group of collaborators both inside and outside the project.

Banana BoF (thanks @MeganRKruse)

Standing room only at the BANANA BoF (thanks @MeganRKruse)

The Bandwidth Aggregation for Networked Access (BANANA) BoF looked into standardizing approaches to share bandwidth on a customer network across two access links (usually one mobile and one terrestrial), as we explored in our ANRW paper last year. There is a lot of interest in doing work in this space, but not yet a lot of agreement as to what that work is yet; as is often the case, discussion continues on the mailing list. The MAMI project will also look into providing cooperative signaling for such approaches.

Yaron Sheffer presented a solution to the problem addressed in last meeting’s LURK BoF using short-term, automatically renewable certificates provisioned using the ACME protocol to the ACME working group meeting. The draft has a good chance of being adopted in the timeframe of the next IETF meeting, and work is progressing in parallel on a prototype.

Mirja Kühlewind and Brian Trammell led a discussion on protocol transitions in transport protocols at the Transport Area’s open meeting, both as an open forum on transition in an area full of efforts to deploy new work at Internet scale, and as input for an IAB document on the topic.

We’re back in Zürich now, and the jet lag is finally over. We’re already busy preparing for IETF 98 in March in Chicago, and the QUIC working group’s interim in Tokyo in January!

Posted in Uncategorized | Leave a comment

PATHspider Plugins

In today’s Internet we see an increasing deployment of middleboxes. While middleboxes provide in-network functionality that is necessary to keep networks manageable and economically viable, any packet mangling — whether essential for the needed functionality or accidental as an unwanted side effect — makes it more and more difficult to deploy new protocols or extensions of existing protocols.

For the evolution of the protocol stack, it is important to know which network impairments exist and potentially need to be worked around. While classical network measurement tools are often focused on absolute performance values, PATHspider performs A/B testing between two different protocols or different protocol extensions to perform controlled experiments of protocol-dependent connectivity problems as well as differential treatment.

PATHspider 1.0.1 has been released today and is now available from GitHub, PyPI and Debian unstable. This is a small stable update containing a documentation fix for the example plugin.

PATHspider now contains 3 built-in plugins for measuring path transparency to explicit congestion notification, DiffServ code points and TCP Fast Open. It’s easy to write your own plugins, and if they’re good enough, they may be included in the PATHspider distribution at the next feature release.

We have a GitHub repository you can fork that has a premade directory structure for new plugins. You’ll need to implement logic for performing the two connections, for the A and the B tests. Once you’ve verified your connection logic is working with Wireshark, you can move on to writing Observer functions to analyse the connections made in real time as PATHspider runs. The final step is to merge the results of the connection logic (e.g. did the operating system report a timeout?) with the results of your observer functions (e.g. was ECN successfully negotiated?) and write out the final result.

We have dedicated a section of the manual to the development of plugins and we really see plugins as first-class citizens in the PATHspider ecosystem. While future releases of PATHspider may contain new plugins, we’re also making it easier to write plugins by providing reusable library functions such as the tcp_connect() function of the SynchronisedSpider that allows for easy A/B testing of TCP connections with any globally configured state set. We also provide reusable observer functions for simple tasks such as determining if a 3-way handshake completed or if there was an ICMP unreachable message received.

Visit the PATHspider website to learn more.

Posted in Uncategorized | Leave a comment