Thanks and Good Bye!

As everything good also the MAMI project comes to an end, or actually came to an end, as it is officially over since Dec 2018. Last week we also had our final review meeting and that now really means, it’s done! I guess that’s good and bad news.

The (b/s)ad news, first of all, is that this will be last post on this blog. However, we will make sure that the webpage stays available for another few years, so you can come back read the existing posts over and over again, e.g. about PATHspider (plugin, update, and 2.0), ECN (here and here), IETF meeting reports (95, 96, ANRW-96, 97, 98, hackathon-98, 99, 100, 101, 102, 103), and more meetings, events, and measurement results… also keep watching our twitter account (@mamiproject) as we will keep you posted with interesting news about Internet evolution and the usual events (IETF meetings, research conferences, and there is another edition of the MNM workshop coming up co-located with TMA’19 in Paris)!

The good news is that I think we can call the project a success. Looking back at the goals we were aiming for when we started the project three years ago, it’s not quite were we are now, but many good things have happened that we can call our achievement. The project was mainly focused on problems with protocol ossification on the Internet and subsequently arising limitations for protocol and service evolution. These problems are still there, and it was never the assumption that we could make them go away in (just) three years. However, we do understand the problem and potential solutions to it much better and many small steps have been taken.

Our initial proposal as a way forward was the introduction of a shim layer for path cooperation, where the layer boundary is also the encryption boundary to clearly separate path signaling from end-to-end information. Chances seemed promising to deploy our approach together with the introduction of a completely new transport protocol, QUIC, when we started the project but at the end it didn’t happen that way. Still the idea of a carefully designed wire image that provides explicit signals to the path has spread the word and is now defined in standards and applied by the Spin Bit in QUIC.

Discussion at IETF QUIC meeting about the Spin Bit.

And then there are also a couple of lessons learnt, or let’s call it take-aways, that, I’m pretty sure of, not only the MAMI partners will further apply in their future research work and standardization efforts but that have also influenced the community as a whole in how we think about the Internet and protocol design. For example, the main take-away from the M3S workshop, that we held last year with a bunch of industry stakeholders, is that future in-network function need to be visible to the endpoints and will need consensus from both sides (also see the M3S report). That doesn’t sound like a groundbreaking insight, however, it is a change in how we designed and operated the Internet so far, as without the wide-spread deployment of encryption or at least (integrity) protection, the easiest and cheapest way has often not followed this principle.

Another effort by the MAMI project, that has impacted the way we design and discuss protocols, is our measurement work. Our approach here was always to base design decisions on measured evidence. The challenge here is not only to measure the right thing at the right point of time but to also to collect enough measurement data in order to draw a signification conclusion whether an observed impairment will be barrier for deployment or not. Good engineering decisions require not only information about the types of impairments that can arise, but estimates of the risk of encountering them on any particular type of network. Over the last three year we have performed a whole bunch of measurement studies, many of them with our own tool for path transparency measurements PATHspider that will further be used and maintained by an ex-MAMIan who is now working with the TOR project. We published our findings as papers, in blog posts, or presented them directly at standardization events (see e.g. IRTF maprg) and other industry meetings. And our results on e.g. ECN, DSCP, PMTU, or IP checksum calculation have impacted and are still impacting work in the IETF tsvwg (this, this, and this) and tcpm as well as operations directly (see cloudefare blog).

However, doing measurements is hard work and costs time and money. Therefore we always had this vision of utilizing the measurement data as a whole that we have collected as a community over so many years. This idea has driven the work on our own repository system, called the Path Transparency Observatory (PTO), as well as the commitment of some of the MAMI partners to support and drive efforts to increase repeatability and reproduciblility of measurement results or actually research results in general in the CS networking community (see here, here, and here).

M&Ms and MAMI T-shirt at Mobile Network Measurement (MNM) workshop.

So while we will all not work together as a project anymore (and distribute nice stickers and M&Ms), you can see that the work of the project will not stop with end of the project funding. I think I can well speak for everybody in the project and say that we really enjoyed working together and are at least a little proud of what we have achieved in the last three years. Thanks for everybody who has worked on the project, as well as our “MAMI friends” who have not only joint us for many fun dinners and lunches (e.g. see draft beer tour in Berlin) but also collaboratively worked with us (see post socket/taps) or provided good and valuable advise (e.g. the EAB)! Other than that, all that’s left to say is: keep enjoying our stickers if you still have some and see you soon no matter what!

Last “official” MAMI lunch at IETF-103 in Bangkok.

Observing the Evolution of ECN Support with the PTO

If you’ve followed the MAMI project over the past three years, you know that we are big fans of Explicit Congestion Notification (ECN). First, it provides an illustrative case study of the deployment of an optional transport protocol feature: standardized in 2001, deployment lagged for a decade due to reports of its signaling crashing certain routers, and rolled out slowly on the server side due to defaults in the Linux kernel, until deployment accelerated due to client-side defaults in Mac OS X. Second, manipulation of ECN is indicative of other TCP and IP manipulations, especially of DSCP.

Let’s have a look at the evolution of ECN support and problems with ECN in the MAMI Path Transparency Observatory front-end. The front page shows a matrix, from which we can see that we have ECN measurements back to 2014 (including measurements taken before the project, and before the development of PATHspider):

The charts page for ECN has charts for raw observations of ECN-dependent connectivity failure (ecn.connectivity conditions), as well as charts of ECN negotiation. Let’s look at connectivity first:

The top chart shows the proportion of observations we have of each condition, and the bottom chart shows the absolute number of measurements. Green ( is good: indeed, attempting to ECN does not often cause problems with connectivity1. By clicking on the conditions above the chart, we can disable the display of certain states; by doing this for all conditions except ecn.connectivity.broken (i.e., attempts to negotiate ECN cause connectivity failure), we can see the following clear downward trend:

Clicking on one of these bars brings up a detail view of the observations from which it is derived (If the observation query for that bar has not already been cached, you may have to wait). To learn more about each target, clicking on the IP address will pull up the RIPEstat page for that address.

For many measurements, we have data taken from multiple vantage points toward the same targets, in order to attempt to find “on-path” (as opposed to “near-server”) interference with ECN; these are represented in ecn.multipoint conditions:

Here we see another unsurprising up-and-to-the-right trend: due in large part to the dominance of Linux web servers and reasonable practices on upgrading them, of about 800000 measurements we took from three vantage points (in Toronto, Amsterdam, and Singapore) this week, more than 86% support ECN negotiation. By zooming in on on ecn.multipoint.negotiation.path_dependent conditions, we can see that there are still a few servers accessible via paths with TCP header manipulation deployed:

We invite you to explore the PTO. To learn more about the PTO itself, have a look at our Deliverable 1.2, which details its design. All of the software behind the PTO is available open source: the core PTO server and tools for running normalization and analysis are in the pto3-go repository, and the web front-end in pto3-web. Normalizers and analyzers for ECN are in pto3-ecn.

1: The grey in the low-volume measurement during the week of 14 January represents a large number of “offline” measurements, where a site was not reachable with or without ECN; this represents a transient connectivity We can click on the

MAMI Active Measurement Hackathon in Aberdeen, Scotland

On the 5th December 2018 we kicked off a two-day hackathon with participants from MAMI and the Open Observatory of Network Interference at the University of Aberdeen. The goal of the hackathon was to share experiences in Internet measurement between our projects.

We started with presentations from the participants to get everyone up to speed on what each other had been working on. Iain Learmonth presented PATHspider, Simone Basso presented ooniprobe and Raffaele Zullo presented tracemore.

While the MAMI measurements have had an Internet engineering focus attempting to find broken behavior in the Internet, OONI measurements focus on finding censorship in the Internet. Typically both broken behavior and censorship can be found implemented in middleboxes and so there is overlap between these areas of interest.

In Tracking transport-layer evolution with PATHspider, we identified with PATHspider that protocol-dependent connectivity failures can be correlated with networks that have large-scale censorship infrastructure. This is probably not deliberate, but a side effect of misconfiguration or poorly written software running on these boxes. To allow for OONI to leverage this discovery, three of the PATHspider tests for discovering broken middleboxes have been added to the OONI test specifications repository.

The tests we have added are for protocol-dependent connectivity failure of: ECN, TCP Fast Open and H2. Support for executing these tests in measurement-kit, the measurement library developed for use in ooniprobe, has already been added for TCP Fast Open and H2.

Ana Custura added support for using arbitrary socket options for DNS-based plugins in PATHspider that will enable new measurements that previously could only be performed with TCP.

One issue with PATHspider is that many of the tests require privileged access to the device, while OONI would like to have volunteers run measurements from their Android devices without requiring “rooting” or other techniques. We explored whether the ability to add eBPF programs to sockets used for measurements might be a substitute for privileged access and our initial thought experiments looked promising, however we did not have time to implement any experiments.

The MAMI project would like to thank Tor Project for partially supporting the costs for this event.

The Future of Middlebox Cooperation in the Internet Protocol Stack

As you may have read in our last post, the MAMI project is celebrating the hard-won, long-discussed consensus to add a spin bit to QUIC. The spin bit was conceived as the minimum useful explicit signal one could add to a transport protocol, and we think the benefit for the overhead is quite worth it. Though it exposes “just” RTT, latency (together with data rate, which is available in QUIC simply by counting packets and bytes on the wire) is possibly the most important metric for understanding transport layer performance diagnosing all matter of transport-relevant network problems, and the spin signal itself can also be observed to infer loss and other issues with network treatment of a packet stream. The definition and deployment of the spin bit is therefore a victory for making network protocols more measurable while preserving the privacy gains from encryption, and a victory for network operations and management.

When we started the MAMI project, we had a much more extensive vision for middlebox cooperation: our Path Layer UDP Substrate (PLUS) proposal provides a generalized signaling mechanism allowing signals from the sender to the path element, as well as a mechanism for allowing on-path elements to communicate information to the receiver (and back to the sender with the receiver’s assistance) in a safe, limited way, all with end-to-end integrity protection. However, we failed to achieve consensus to move forward in the IETF with this proposal by forming a working group, so we refocused our efforts on a more modest implementation of the explicit signaling concept.

Selective exposure, however, has a pretty toxic reputation in the IETF, as it “breaks” TLS, by adding a third-party to a two-party protocol and changing its security properties, reducing or eliminating integrity and confidentiality protection with respect to a set of entities in the network that may or may not be known to one or both endpoints. In our recently published white paper, the outcome of the M3S meeting, we argued that all approaches in this space must provide transparency and control to the endpoints.

In hindsight, more important to the controversy around PLUS was (in our opinion) a fundamental misunderstanding about what is meant by the term “middlebox cooperation”, as there are two broad approaches to supporting on-path operations on encrypted traffic being pursued in the industry:

  • Explicit signaling approaches like PLUS, where information about encrypted traffic is carried separately from the encrypted traffic itself.
  • Selective exposure approaches like mcTLS, mbTLS, eTLS and so on, which share secrets from the end-to-end cryptographic context with middleboxes, allowing them to decrypt (and in some cases, modify) traffic between the endpoints.

We focused on explicit signaling in MAMI, in part due to the difficulty of building selective exposure approaches that still provide useful security properties for the end-to-end communication. However, large parts of industry are more interested in selective exposure, because building and deploying selective exposure approaches promises to be more easily backward-compatible with existing protocols and network hardware. PLUS, after all, envisioned deploying a new transport protocol stack. We examine the tradeoffs in these approaches in another white paper, released today.

Among the arguments presented against PLUS at our BoF in Berlin in July 2016 were doubts that the protocol was safe in general: that it provided new side-channels for data exfiltration or hooks for coercion. We looked into this as part of our own internal study of integrity-protected, encrypted-payload middlebox cooperation approaches like PLUS, finding in a white paper, also released today, that the additional attack surface presented by PLUS is negligible.

The looming end of the project does not mean the end of the problems we set out to solve, or interest among the project participants and the field in general in improving privacy and evolvability through encryption while retaining limited support for in network functions. So, we ask ourselves: what’s next?

Several threads of work in this space will outlive MAMI:

Our efforts to add measurability to QUIC, which appears to represent the first real hope of deployable transport evolution in the last three decades, will of course continue. And QUIC itself will continue to evolve. The group of implementors and network folks we’ve helped pull together around the spin bit will keep experimenting with the best ways to make it possible to do passive measurement and diagnosis, as QUIC traffic increasingly supplants TCP.

Our work on experimentation with the Loss/Latency Tradeoff (LoLa) signal continues, as we noted in the previous blog bost. LoLa may enable better performance for real-time communications traffic and was first specified as a bit in the PLUS header continues, with a possible implementation as a DSCP codepoint.

The IRTF Path Aware Networking Research Group (PANRG), founded by the project, is pursuing work both in understanding why previous attempts to consider middleboxes in transport layer interactions have failed, and in exploring a set of properties of end-to-end paths on which future explicit signaling methods could be built.

Further, there is ongoing work on UDP options in the IETF TSVWG and a discussion about support for proxying of UDP-based, encrypted protocol started in the at IETF-103 with some initial presentations about requirements and use cases in the tsvarea session.

We are very happy that this important discussion is continued and that we as a project could provide a valuable input based on measurement and experimentation. And while the project’s time had to come to an end at some point, the MAMI contributors will keep working on this in order to keep the Internet operational as well as improve  its robustness in operation, performance, and security.


IETF 103 Bangkok

The MAMI project was back at the IETF for the last time — as a project, anyway — last month at IETF 103 in Bangkok. Join us on a trip through the week!

the IETF 103 venue in Bangkok

Hackathon: Loss-Latency Tradeoff in Mobile Networks

Most of us arrived on the Friday before the meeting in order to attend the now already traditional hackathon. This time we set up a table to pull together a few different projects related to measurement and measurability. Friends of the project Fabio Gulgarella and Mauro Cociglio ran an experiment running on a 2-bit variant of the spin-bit they dubbed “delay sample”. Thomas Fossati, as main representative of the MAMI project, has been doing some measurements on the impact of the LoLa bit on the mobile network.

This is another instance of the “one-bit signalling” topic we’ve been working on in MAMI – notably, a Loss/Latency Tradeoff (LoLa) bit was present in the original PLUS proposal. These experiments are concerned with mapping the LoLa signal to a dedicated low-latency bearer in the LTE core and air interface. This is different from how mobile networks are deployed today, to actively ignoring or bleaching any end-user signal, and bundling all flows to and from the Internet into a single “default” bearer, whose buffering characteristics that are not compatible with low-latency traffic. The established behaviour is partly rooted in the desire to prioritise operators’ voice services over competing over-the-top services, but that market consideration has lost relevance in the recent years.

It looks like the incentives are now aligned in the direction of allowing more suitable treatment of Internet real-time flows. However, a couple of preconditions need to be satisfied before we can move on from the status quo. First, the real-time flows must be efficiently identified so that they can be put on the right queue especially nearby the bottleneck link, which in 4G mobile networks is typically the air interface. This is at odds with the rise of encrypted and multiplexed transports, which has the potential of increasing the cost/accuracy ratio of DPI over the acceptable threshold. LoLa instead is this simple clear-text signal set by the endpoints which would enable a super efficient classifier. The question is how can we trust the endpoints to set the LoLa signal correctly? What if they lie? Would they get an advantage at the expenses of other flows? The hackathon experiments were aimed at answering these core questions. On one hand we wanted to see whether giving mobile devices a low-latency bearer in addition to the default one would make a difference in terms of QoE. And secondly, we wanted to measure what happens when an endpoint lies about the nature of its flows. So Thomas spent his Saturday and Sunday hacking around the LTE module of NS-3 and getting these fine results which he also presented at the HotRFC session on Sunday night. We got some positive feedback about the idea from a few mobile operators in the room.  Thomas, Gorry Fairhurst and Mirja Kühlewind together with Vz, AT&T, Orange and Three managed to schedule a side meeting for later in the week to discuss both the finer points and an overall coordination strategy also involving and the 3GPP liaison Georg Mayer and Florin.

Hackathon: Path MTU Signaling in QUIC

In addition, Tom Jones worked on Path MTU signaling mechanisms for QUIC at the QUIC hackathon table. Tom’s draft describing “Packetization Layer Path MTU Discovery for Datagram Transports” will also be incorporated in QUIC. MTU discovery is also under discussion in the 6MAN Working Group, where there are two competing proposals to add new mechanisms that allow better signalling of MTU by routers a long the path, Truncate and Hop By Hop:

  • The Truncate proposal adds a MTU Destination Option, the intention of this option is that when a router has to discard a packet for being too large it also truncates the datagram. The MTU Destination Option carries the original packet length allowing the end host to detect this packet as ‘truncated’, and the end host uses this packet to send an ICMP PTB message.
  • The Hop By Hop proposal creates a new IPv6 Hop By Hop option. This option is added to small packets, when a router forwards a packet with this option it examines the MTU field in the option, if the MTU is larger than the routers out going link it updates the MTU field to match this link size.

Tom implemented the HBH proposal at the IETF 103 Hackathon to get a sense of how practical it was to add to a real network stack — he had previously implemented the Truncate proposal on the flight back from IETF102 in Montreal. His inplementation in FreeBSD achieved self-interop between end host and router in the two days. Running code is a core part of the IETF ethos, after IETF 103 we are able to compare how the HBH and Truncate proposals compare and differ in implementation.

Monday: Evolving Transport

On Monday, the official sessions started with an interesting discussion about the evolution of transport and the role of proxies in the Transport area open meeting featured with presentations from Lucas Pardue and Ben Schwartz’s team on HiNT and Helium and a presentation from Thomas on PEPs role and fate in an end-to-end encrypted Internet. One of the takeaways of the session is that if we want to keep PEPs alive, which looks like a sensible thing to do at least in some scenarios (cfr. satellite), we need to re-think the security model in particular how the transport security association is modelled and what is its relationship with the security context that sits at the application layer. QUIC in particular, collapses the two into a single inextricable blob, making proxying an all-or-nothing option, which is far from ideal. The “transport proxy” gang (including us, Lucas and Ben’s team) is going to chat further again in December to discuss next steps.

Then the first TSVWG session on Monday touched a little on the PMTUD work happening in other working groups, as already mentioned above. And we had a presentation by friend-of-the-project Colin Perkins on the Transport Header Encryption Draft, recently adopted by the WG, which explores transport-specific considerations of the trend toward more ubiquitous encryption and encryption deeper down the stack.

Tuesday: Measurement and Analysis for Protocols and the MAMI Lunch

The founded-by-MAMI Measurement and Analaysis for Protocols Research Group (MAPRG) met on Tuesday. This time we had an 2-hour slot (slightly shorter than usually) with interesting talk about UDP checksum issues, QUIC performance over  satellite, comparison of CoAP/MQTT/HTTP in vehicular scenarios, as well as measurements on Certificate Transparency and OCSP Must Staple deployment.

Tuesday also saw our traditional lunch for MAMI and friends:

Tuesday and Wednesday: QUIC and the Spin Bit

Since MAMI is largely focused on transport evolution, and transport evolution work in the IETF is now focused on the completion and deployment of the IETF standard version of QUIC, the “main event” of the IETF for the project was the QUIC working group. QUIC met on Tuesday and Wednesday. Tuesday’s meeting covered smaller issues, including the applicability and manageability drafts, presented by Mirja.

After 18 months of heated discussion and thorough experimentation, the latency spin bit was  been accepted during the second session on Wednesday as a feature of the base QUIC protocol. Implementation and participation in signaling are optional, though several large endpoint implementors indicated they will implement and enable the bit by default. The consensus is to add just the spin bit, without the Valid Edge counter described in our recent IMC paper.

QUIC hums on the Spin Bit (image: @csperkins)

This proposal has been pretty controversial, not so much technically but rather from a political perspective – being a proxy for the wider (and highly flammable) discussion around the tension between protocol encryption and network functions in the IETF. One key takeaway of this whole experience (and a key result of the MAMI project as a whole) is that getting the right balance between opaqueness and transparency a protocol’s wire image should provide is a very tough job and requires a great deal of analysis and experimentation along many different axes.

After three years working on these topics, and despite the hyper-sensitivity to privacy issues in the Internet many of us in the project have, we are even more then ever convinced that a world of completely opaque wires is not a place where we would like to spend our future, and that total privacy and total privatisation of the Internet infrastructure are two sides of the same coin. The Internet is a common good and we need to keep it that way, finding the right balance between what is visible and what is not is a tough job but we need to resist the temptation to use the encryption hammer for things that are not necessarily privacy related – e.g., as anti-ossification techniques. We very much hope that this decision enables now more experimentation and will lead to more experience with these kind of explicit signals which can be the basis for more discussion about additional features that would support measurability.

Wednesday: UDP Options, MARC, and Encrypted SNI

Wednesday morning also saw the second TSVWG Working Group session discussing the status of UDP Options, with Tom presenting an implementation report. Tom also followed up on questions asked on the TSVWG mailing list about the future and suitability of FRAG, LITE and AE UDP Options. This led to an enthusiastic discussion about the suitability and use of UDP Options at all. Tom continued with another presentation, this time on Datagram PLPMTUD, this slot was grouped with presentations from the authors of the 6MAN drafts attempting to address PTMUD. These presentations led to very productive working group discussion on both PLPMTUD and PMTUD, with finding the right MTU for a network still being an open problem.

The video interest group (GGIE) side meeting we presented MARC (mobile assisted rate controller), the congestion controller developed by Morteza and Gorry in the project that uses (among others) the mobile throughput guidance signal from the eNodeB to adapt the sending rate to the (very quickly and widely) changing conditions of the mobile network. There was an entertaining discussion around the topic including a nice digression on how to carry the MTG signal in QUIC using UDP options.

In TLS there was more discussion about the encrypted SNI proposal which is trying to remove one privacy-critical bit of clear-text from the TLS wire image. Of course SNI is relevant to policing middleboxes (enterprise TLS proxies, censorship and parental control filters), hence some of the usual scuffle. There are a couple of issues with the current ESNI proposal, one of which is pretty substantial in terms of the ability to reliably deploy the feature, but there is a huge amount of energy being poured into finding a good solution so hopefully we are going to get something done soon-ish. On another topic, the DTLS 1.3 protocol specification is complete and ready for review by the working group. We need some security analysis there which will take a fair amount of cycles. In the meantime, the DTLS 1.2 connection identifier is done and is currently in working group last call. So this should provide an OK solution (although not ideal because of the higher correlation properties compared to 1.3) until 1.3 is finalised and deployed at scale.

Wednesday: Transport Services (TAPS)

The Transport Services (TAPS) working group, in which MAMI heavily contributes, continues its work to standardize an interface to a modern Internet transport layer. The group had a productive session, solving some of the open issues for the API document. In addition Tommy Pauly, another friend of the project, presented a mapping of the TAPS interface to QUIC, leading to a discussion about mappings for different DNS variants (DoT, DoH, … Do*) as a quite interesting use case for taps, given it could actually provide an opportunity to simplify future mappings of DNS to other transports.

If you’d like to help the TAPS effort as well, it’s now the right time to review the architecture document as the first of the taps trilogy, to be published around IETF 104 in March 2019.

Thursday: ACME, Round Tables, and Not-So-Secret Cabals

The very last session of the meeting was ACME. MAMI has a couple of drafts there: the STAR document, which is going through a second LC as we speak due to some adjustments we had to do related to a pretty big last minute change in the base ACME spec, and the delegation draft, which we have re-written as an ACME profile (as per request in Montreal) which – good news everyone! – we got consensus to adopt by the working group. As per IETF process, the official CfA confirmation is currently on the mailing list and should be over in a few days.

On Thursday, the long-running Post Sockets Secret Cabal met, though now this meeting is less an insurgency bent on overthrowing BSD Sockets and more merely an afterparty for the TAPS meeting where we distribute issues to work on before the next interim.

The Post Sockets Not So Secret Cabal (image: @csperkins)

Thursday also saw our LoLa round table. The meeting went well and it was decided that the coordination for the various activities – which are likely to span more than one SDO – should happen in the Internet Group in GSMA. The activity is in progress, and last week I presented the plan at the GSMA IG plenary.

After that we had a de-brief with the spin-bit folks, including Brian joining remotely via Thomas’ laptop :-), to plan the next steps in measurement experimentation on QUIC. This was a fitting end to the technical part of the meeting in Bangkok, as well as the MAMI project’s official engagement in the IETF.  For us, the IETF crowd of the project, it definitely has been an incredible pleasure working in MAMI and progress these important topics in standardization. MAMI has been a fantastic project, which tackled a core technical and societal tussle with an abundance of courage (and/or recklessness, you choose), with a truly amazing group of people! Thanks everybody!

The MAMI Management and Measurement Summit

Just before IETF 101 in London in March, the MAMI project hosted an invitation-only MAMI Management and Measurement Summit (M3S), bringing together researchers, engineers, and vendors for a focused discussion on how to meet the challenges posed to network measurement and network management by the increasing deployment of strong encryption and the extension of encryption down the stack. Today, we release “Challenges in Network Management of Encrypted Traffic”, a white paper covering the discussions and distilling the recommendations that came out of the meeting.

This discussion has played out in multiple forums, including the IETF, for some time, underpinning discussions and debates from the (failed) proposal to include static keys in TLS, to continue to provide for “business as usual” monitoring, to the spin bit proposal in QUIC, which replaces implicit passive measurability of RTT with an explicit signal. Recognizing that neither business as usual, nor forging forward with the deployment of strong crypto down the stack and invalidating most of the current practice of network management, are tenable positions, the attendees converged on a set of recommendations for future protocol design and network architecture to partially meet these challenges:

  1. Protocols and networks must provide for independent measurability of important metrics when these measurements may be contested: one outcome of increasing encryption is that existing independent passive measurement techniques will become less effective.
  2. Future secure protocols should support different security associations at different layers: approaches that integrate transport and application-layer security (such as QUIC) make limited or no provision for network management that need to interact with the transport protocol while not breaking application layer security, in contrast to the TLS-over-TCP status quo.
  3. Transparent middleboxes should be replaced with middlebox transparency: the dominant architectural pattern for in-network functions today is that of the “transparent middlebox”, which attempts to the extent possible to be undetectable to the endpoint(s). While this has benefits for initial deployment, it makes it impossible to build cooperative protocols, where the middlebox and its functions are visible to the endpoints, and the endpoints have some control over how their traffic is treated by the network (in the last instance by detecting a middlebox with which they do not wish to cooperate, and cease using the path).

IETF 102 Montreal

The IETF week started on Saturday with the hackthon. Diego, Pedro and I (Thomas) have been working on a variety of topics: a “STAR Requests” slide deck for the upcoming SecDispatch meeting, an architecture design for an end-to-end ACME STAR demo, and the one-bit experiment harness- reproducing and analysing a bug in iperf3 which prevents our trafic to produce reliable / precise emulation of a throughput-seeking flow.  This was pretty quickly done and we got eventually absorbed by the SUIT / TEEP table.  The back-story for this is I had brought from Cambridge to Montreal 40 boards with me, done especially for the hackathon by ARM – Cambridge is a very small city where everyone seems to know everyone else. Though we didn’t get to do much with the boards and the TEEP protocol, we got to know a bit better the technology, talk to people involved and, in my case, share my experience with fellow Nokians IoT folks.  The QUIC spin-bit table, with Marcus, Al, Roni and Emile was also active on experimenting with 1-bit spin signal and a bunch of heuristics to reject bad RTT samples in presence of reordering. See Marcus presentations [1] and [2].)  This was, as usual, very useful, ludic, extemporary and genuinely fun!

On Monday, at the SecDispatch session Diego spoke about “Generating Certificate Requests for STAR Certificates“.  This complements ACME STAR and is a building block needed for closing the loop in the identity-owner controlled name delegation workflow using X.509 certs we designed in MAMI.  We need to find a sensible home for this document so that it can receive a thorough security analysis. The outcome of the discussion was quite surprising but, IMO, pretty good – even though there is a fair amount of running code that I’m going to trash as a result.  Martin and Ted noted that STAR request could be re-written in terms of pure ACME, provided the proof of the identifier ownership is not requested by the IdO.  So, the subsequent step was to go back to ACME and discuss adoption of a STAR Request there, which is what we’ve done on Tuesday (see below).  Earlier that day, at the TLS session, we discussed DTLS connection ID for 1.2 and 1.3, handling an interesting fallout from the recent major header refactoring in DTLS 1.3 on to the content type allocation (and its repercussions on the marking strategy envisaged for CID in 1.2.).  The discussion is not settled yet, and continues on GitHub [1], [2], [3].

On Tuesday in ACME, STAR officially entered WGLC, which means, finger crossed, we should be pretty close to finalising this one.  In relation to STAR Requests, instead, we were asked to produce a new version of the protocol as an ACME profile, which can then be called for adoption by the WG.  So here it is, waiting for the base ACME draft to do the final adjustments to make it through IESG.

At the same time, in HTTPbis, HELIUM & HiNT were introduced.  This work lead by Google and BBC R&D has some interesting overlap with MAMI in that it provides a tunnelling protocol and side channel with a proxy which might end up being useful in tuning CC for mobile networks in presence of fully encrypted end-to-end traffic.  I especially liked the alternative tagline “a proxy is an honest middlebox” in Ben’s presentation.

Discussion in taps, later that afternoon, many focused on the API draft. While there are still a whole bunch of open issues, thers broad agreement on the general concepts and a lively discussion making good progress to resolve these issues. The taps working group will hold a vitual interim on September 12.

QUIC met on Wednesday morning. Work is winding up on the IETF standard version of this new transport protocol, which means it’s time to start talking about how it will interact with the rest of the Internet. Brian presented the “operations” drafts for the first time, though they’ve been under construction since shortly after the working group was chartered. The first of these, the applicability document, gives pointers on building applications (other than HTTP) on top of QUIC, and the other, the manageability document, provides an independent guide to the protocol for operators of the networks its traffic will be carried over. Discussion in the WG pointed out that this division might not be precisely the right one — pointedly, the definition of an abstract interface from the application down to QUIC is missing, and doesn’t necessarily belong in either. Work on this will ramp up in the weeks before Bangkok.

The Wednesday concluded with the tsvarea session where Brian and Ted (Hardie) presented two IAB documents defining the wire image and path signals. Further Christoper Paasch and Ian Sweet provided some insights on implementation/deployment challenges for MPTCP and QUIC.

Thursday started of with the usual maprg (Measurement and Analysis for Protocols research group) session. The agenda cover a broad spectrum of topic including reordering in QUIC, as well privacy issue of RTT measurements/bufferbloat presented by Brian and measurements of PMTU by Gorry.

Later on in LAMPS we discussed “STAR at large” for adoption.  Yoav’s presentation is very neat, and does a good job discussing the non-web use cases for STAR certs (e.g., VPNs, Software defined storage, but also autonomic networks).  The WG asks us to clarify a few points, in particular how to make the semantics of a no-revo cert explicit to relying parties, before taking it as an working item.  So we are back to the desk on this one.

Last but not least, the Path Aware Networking (PAN) Research Group met midday on Friday, during the last slot of the meeting, for the first time as a no-longer-proposed-but-actually-chartered research group. In addition to proposed talks, discussion centered around two work items: first, a review of bad ideas in transport-path cooperation, and second, a start on answering the research group’s first question about path property definition and representation.

We had the usual amount of fun, high-bandwidth, high-energy hallway discussions that make the IETF meeting a pretty special thing.  Next round, sadly the last one under the aegis of MAMI, is going to be Bangkok. Stay tuned!

Summer School on Internet Path Transparency Measurements

On June the 11th the Electronics Research Group hosted the MAMI Summer
School on Internet Path Transparency Measurements in Aberdeen, Scotland.
This consisted of a few hands-on workshops, with participation
both on-site and remote via video conference.

The summer school started with Korian and Justin demonstrating Tracebox
through a variety of topologies. The participants then worked on their
own trying to uncover middleboxes and hidden topologies using a variety
of tools, including tracebox and paris-traceroute.

To follow, the history and development of PATHspider were presented by
Iain Learmonth, one of the creators of PATHspider. Iain also delivered
an interactive Scapy tutorial followed by teaching students how to use
Scapy to create PATHspider plugins using the Evil Bit as an example.
The participants ran plugins against a list of real targets and produced

To finish off, Brian Trammel delivered a presentation on the PTO (Path
Transparency Observatory), and helped the participants upload to it the
results they collected in the previous session. The participants then
learned how to query the PTO in order to examine the data they just
uploaded. In an interesting twist of events, the measurements found ECN
manipulation by a middlebox in the eduroam network in Aberdeen – mission

If you missed the summer school, all slides and materials can be found

2. Edition of the MAMI/MONROE workshop on Mobile Network Measuremnt (MNM’18)

On June 25, we held another edition of the Mobile Network Measuremnt (MNM’18) workshop together with the MONROE project. As last year the workshop was co-located with the Network Traffic Measurement and Analysis Conference (TMA 2018) which was held this year in Vienna. We started the day with a keynote from Varun Singh, the CEO of – a start-up from Helsinki that collects and analyses WebRTC measurements. While he was giving us some interesting insights in the technical work and challenges of monitoring real-time communications, his elaborations about the challenges and problems as a young start-up were as least as exciting!

The technical agenda of the workshop were split between 6 talks into two sessions, basically between the layers: first focusing on Network QoS and (mobile) Coverage, and then Multipath and Application Performance (in mobile networks). Further Özgü Alay, the co-ordinator of the MONROE project, and Iain Learmonth, the main maintainer of PATHSPider, gave an introduction on the use of the MONROE platform and mobile measurements using PATHspider.

One more important thing to mention: We did print MAMI labelled M&M’s for the MNM workshop… of course! I’d say those where a big success but we gladly have still some left. So if you meet us the next time at some event (e.g. come to our SIGCOMM tutorial on Repeatability and Comparability in Measurement (RCM) where we introduce tracebox, PATHspider, and the PTO), chances are good to get some as well!

Of course we also stayed for the rest of the TMA meeting in Vienna! Beside our two paper on Exploring usable Path MTU in the Internet and Tracing Internet Path Transparency, there where a lot of great talks and keynotes, including an expert summit on Tuesday of the week. After all a big thanks to the organizors for the great meeting as well as great social events (every night of the entire week)!

IETF101 in London, March 17-23

Last month, the IETF returned to the Hilton Metropole near Paddington Station in London for its 101st meeting.

Hello, London!

While it is always nice to go to an IETF meeting in Europe (and therefore suffer less from jet-lag), in this specific hotel the challenge is to find your way around and actually make it to your meeting in time. The meeting room are distributed over three “wings” in the first as well as ground floors as well as in the upper third and “lower third level” (i.e., the sub-basement, next to the Underground), with a less than optimal elevator configuration:

However, the meeting itself was very productive, despite the labyrinth! As is now customary, the week started early on Saturday and Sunday already with the Hackathon where MAMI was present with two projects and a total of 6 people working on Scalable, Privacy-preserving In-Network Measurement (i.e., the QUIC Latency Spin Bit) and a testbed for 1-bit optimisations for the mobile access network based on the Loss-Latency tradeoff.

Monday, we mainly focused on transport topics with a presentation of the soon-to-be-finished AccECN TCP extension in tcpm, an interesting discussion about framing in QUIC (i.e. whether or not to use DTLS as QUIC’s wire image), and a general discussion about TCP encapsulation in tsvarea.

On Tuesday both of the research groups that have grown out of the MAMI project met: theMeasurement and Analysis for Protocols (MAP) and the proposed Path Aware Networking (PAN) research groups. MAPRG’s two and a half hour slot contained many interesting presentations, covering both papers from, e.g., IMC as well as “previews” of work presented at PAM 2018 the following week in Berlin.

PANRG met for the third time as a proposed RG, which means that the process of actually forming the group officially is underway now. The meeting had a productive discussion and a lot of positive feedback, indicating that there is interest in continuing work in the group. There seem to be two broad areas of research the group will tackle going forward: exploring how to add “path awareness” to the Internet architecture (in the vein of the PLUS work pursued by the MAMI project), and continuing work on various not-yet-ready-for-standardization techniques to use path information at the transport layer.

The MAMI project, together with the H2020 NEAT project and engineers and researchers from Apple, the University of Glasgow, and TU Berlin, proposed a new architecture for the Transport Services working group, and an abstract interface for that architecture based in large part on MAMI’s Post Sockets and flexible transport layer work. These drafts were adopted by the TAPS working group, and will form the basis of a new standard abstract API for the transport layer.

The new TAPS cabal, working out the details after the adoption of the new architecture drafts. (Thanks Colin Perkins for the photo!)

MAMI was also busy in TLS, presenting a proposal to extend the DTLS header and discussing the nuances of the DTLS connection Id encoding, and in ACME where we asked for WGLC of the STAR document.

The “main event” for the project, so to say, took place on Thursday morning with a discussion of the QUIC Spin Bit, a facility for supporting passive round-trip time measurement despite the encryption of the QUIC header. This discussion took the majority of a two and a half hour session, and was quite lively: for the first time in our experience at an IETF meeting, the microphones at an IETF meeting had to be moved to keep the line from running out the door.

“How many engineers does it take to spin one bit?”

While the working group still could not come to consensus to add the spin bit directly to the protocol at this time, the outcome was a good one for the project (and for the concept of explicit measurability and in our opinion, for the Internet at large): one bit has been reserved for experimentation with the spin bit, with a directive to reserve a further two for experimentation with additional signaling such as the Valid Edge Counter (VEC) presented at the meeting, with a draft to be published under working group change control for coordinating larger-scale experimentation.

All in all, it was a great week in London, and we’re already looking forward to July’s IETF 102 meeting in Montreal!

PATHspider has exciting new features in release 2.0.0

PATHspider is a free-software extensible path transparency measurement tool that performs path transparency measurements using either real network stacks or packet forging. A new major version, 2.0.0, has just been released and is packed with new features to expand the range of measurement tasks it can perform.

For the evolution of the Internet’s protocol stack, it is important to know which network impairments exist and potentially need to be worked around. PATHspider performs A/B testing between two different protocols or different protocol extensions to perform controlled experiments of protocol-dependent connectivity problems as well as differential treatment.

One new feature that can simplify the creation of tests for path transparency to new protocols in development is the inclusion of a packet forging framework, Scapy (thanks to Ēriks Dobelis for his work on porting Scapy to Python 3, without which this would not have been possible).

This means that even before the specification for a new protocol or extension is written, before any code exists, you can already be testing for possible issues. This feature was already used to explore the possibilities for a new DiffServ codepoint for lower effort traffic and reported on at an IRTF MAPRG meeting.

The API for developing plugins for new measurements has also been greatly simplified. A lot of effort has gone into refactoring large chunks of the codebase to remove code duplication in plugins and ensure plugin authors only have to write the code that they need to.

When using real network stacks, connection helpers are now provided for HTTP and DNS using pycurl and dnslib. This greatly simplifies the creation of plugins that are toggling kernel options or iptables rules as you now only need to write the function to perform the system-wide configuration and the traffic generation is handled for you.

Analysis of data is now also simplified as PATHspider no longer outputs individual flows but instead waits for all the flows to be available and performs an automated analysis to generate conditions that apply to a path, for example whether or not the use of a feature has broken connectivity. It will also determine your public IP address if behind a NAT, and look up the ASN of the vantage point and include these in the computed network path in the output.

A completely new feature in PATHspider allows the use of the built-in flow meter without actively generating any traffic. This can be used to examine another device that is generating traffic or to examine traffic on a link aggregating many devices to discover how clients behave and what typical Internet traffic looks like with regard to the protocol features in use.

PATHspider includes comprehensive documentation to help you get started. If you have Vagrant installed, you can have a working PATHspider environment as simply as “vagrant up”. Other installation methods are described in the documentation.

In the near future, there will be more work on the test suites that allow you to verify the installation of PATHspider is working correctly and the addition of a benchmarking command to optimise the speed at which PATHspider is running on a particular machine to balance speed with dropped packets. There will also be more extensive documentation on packaging your plugins so that they can be more easily shared other researchers and deployed to remote measurement vantage points.

If you have any ideas for interesting plugins, you could file a GitHub issue or send a tweet to @iainlearmonth. If it sounds interesting, it may make it into the next release. If you’d like to follow PATHspider development, you can also join #pathspider on

In order to complete the inclusion of pycurl for traffic generation, it was necessary to add support to pycurl for a couple of additional features (thanks to Oleg Pudeyev for reviewing and merging those changes). Unfortunately these changes were only recently released and so it may be necessary to install pycurl from source. If using the Vagrantfile this will be done for you, on a Debian system the following will get you going:

apt-get install python3-libtrace python3-sphinx python3-straight.plugin python3-setuptools pylint3 python3-pep8 python3-pyroute2 python3-pip unzip python3-nose python3-stem
apt-get build-dep python3-pycurl
pip3 install 'pycurl>='
pip3 install 'pathspider>=2.0.0'

IETF 100

Good morning, Singapore!

A few of us from the MAMI project started arriving in Singapore on Friday, in order to participate in the hackathon, which has become an IETF tradition. A few hours later we were at our tables, Thomas Fossati and James Bulmer working on a STAR Requests implementation, and Brian Trammell sitting with the QUIC and TLS tables to work on passive measurability of the protocol with mokumkuoren. We had a day and a half of fun coding, patching specs, improving test coverage, and generally hacking about.  Pretty good progress indeed, a few nice chats with old and new friends and great food and beers.  Thank you hackathon organisers!

Test coverage improved by the end of the hackathon…

The next step for STAR is to get the e2e demo up & running on our dear Blue Box with a miniaturised but fully functional CDN talking to an ACME STAR CA in time for IETF 101 in London in March.

Back to the hotel we posted acme-star-01, since we had missed the pre-IETF cutoff date and we needed people to get a chance to glance through the changes which had been quite abundant. The updated draft includes  an “implementation status” section (as per BCP205) documenting the work that Diego De Aguilar Cañellas has done on top of Boulder and Certbot (LetsEncrypt’s server and EFF’s client, respectively) to add the new STAR flow in ACME.

Then Monday arrived, and the meeting started. Diego went to TRANS to talk about ACME STAR (slides) and discuss the cost that an increase in log ingestion of one or two orders of magnitude poses on the Certificate Transparency infrastructure. Based on the observation that all STARs belonging to the same ACME order are basically equivalent modulo their validity dates and serial number, we also prepared and presented a napkin design that uses a new SCT type to address the scale problem. The discussion (youtube link) was, as often happens in the IETF, very instructive but inconclusive. We went away without a clear answer whether this is going to cause troubles or not.  The reactions up to this point are scattered all over the spectrum, ranging from “omg, this will melt the world” to “nah, the log can cope” to “meh, future problems”.

Monday’s session of the Measurement and Analysis of Protocols research group (MAPRG), co-chaired by Mirja Kühlewind, included a presentation by Brian of Principles for Measurability in Protocol Design (slides). This paper articulates our vision for measurement as a first-class function of the protocol stack.

Another meeting of the not-very-secret Post Sockets cabal

Tuesday started off with the Transport Services (TAPS) WG, where discussion focused on whether the working group should take on work in defining abstract programming interfaces for applications atop a dynamic . Here, discussion focused on Post Sockets, a realization of MAMI’s flexible transport layer (FTL). We came to no conclusion, but will schedule a meeting in the margins of our upcoming plenary in Cambridge in January to further develop Post Sockets into an architecture for flexible transport services.

The QUIC mic line (©Stonehouse Photography)

The first session of QUIC was Tuesday afternoon. A slightly congested mic line and very robust discussion surrounded the wrap up of the design team for the “spin bit”, designed to provide explicit passive measurability of end-to-end latency in QUIC flows, replacing TCP timestamps for this purpose. While the design team itself was unable to come to consensus to add the spin but to the protocol (though it did conclude that passive latency measurement poses no known threat to privacy), there was a balance of support in the room for adding passive latency measurability to the protocol, and a sense that the spin bit is a good method for doing so. However, work to achieve consensus is ongoing; watch this space for future posts about our experiences with implementation and use of the spin bit.

Thursday was definitely a busy day.  In TLS we did the call for adoption for the connection identifier for (D)TLS.  That went really smooth and the draft has been adopted – pending confirmation on the mailing list, obviously.  The co-authors have slightly different opinions on a few key points, including implicit vs explicit signals and the protocol friendliness to troubleshooting (deja-vu?).  But we all agree this solves the big issues related to connection migration and NAT rebinding that we already discussed in a previous post and the important thing here is that the TLS working group reckons this is worth spending working group cycles on.

Thursday afternoon saw the second meeting of the Path Aware Networking research group (PANRG), and included presentations on path property dissemination and interfaces for path control (hello again, Post Sockets), as well as an examination of open questions in bringing path awareness to the Internet architecture. We see the general area of path-aware networking as being an unexpected legacy of the project.

Later, in the ACME session we presented the updates we’d been working on in the months following Prague.  The document is in good shape, the protocol flow should be stable and my impression is that once we complete the security and operational analysis, the document should be ready for last call. After ACME, we had another informal STAR-centred meeting organised by Yoav to talk about generic short-lived certificates that automatically renew which may or may not depend on the ACME ecosystem – for example, based on ANIMA, or on proprietary systems – and may or may not address the HTTPS use case and address instead IPsec, non-web uses of TLS & SSH in enterprise and datacentre-type environments.  The meeting was well attended with more than 20 people at the table (a couple of CDNs, middlebox vendors, web folks, mobile network operators, academia, other SDOs) all bringing their own experience and perspective on the issues related to certificate revocation (one of the core motivations to look into STAR) and the solution space.  The discussion was great – with use cases in NSF, vehicle-to-vehicle, SAN, IPsec, and of course the Web – though a tad too short: many had to run, including Diego and I to our traditional MAMI dinner 🙂  One core thing that was concluded is that the “short” in short-term is a very fluid concept and must be defined on a case by case basis.  In fact, the exact definition of “short” should match the time it takes to the revocation information (CRL and/or OCSP) to propagate to the relying parties.  We hope to continue the exchange on the SAAG list or maybe in an ad-hoc list.

Goodbye, Singapore!

With that, we bid farewell to Singapore! See you all at IETF 101 in London!

MAMI @ IMC’17 in London

The MAMI project was represented at the Internet Measurement Conference (IMC) 2017 last week in London wiht two posters:

Mirja Kühlewind, Brian Trammell, and Tobias Bühler at IMC’17

While we were of course most excited about the positive feedback we got for presenting the MAMI Path Transparency Observatory (PTO) and our planned work on measurability of encrypted traffic, there were a lot of very interesting presentations covering main transport protocol- and MAMI-related topics, e.g TCP Congestion Signatures, Large-Scale Scanning of TCP’s Initial Window, and Taking a Long Look at QUIC: An Approach for Rigorous Evaluation of Rapidly Evolving Transport Protocols. Check out the full program here.

Is Internet RTT reliable for geolocation?

Short, short answer: nope, don’t bother. While this is probably obvious to any of you with network engineering experience, we thought we’d use RIPE Atlas to have a look into this question anyway.

In the context of an ongoing conversation about the explicit exposure of RTT information to devices on path in the IETF standard version of the QUIC protocol, we’ve briefly looked into how much of a threat Internet-observable per-path RTT is to geoprivacy of one of the endpoints. It turns out that the old network operations rule of thumb that a millisecond of RTT is 100km long adds a whole lot of uncertainty — a fact which also confounded some recent work on RTT-based anycast detection by Cicalese et al. Only in cases where one is very, very lucky — microseconds lucky — in the placement of the vantage points from which RTT measurements are taken can one use RTT measurements for elimination-based geolocation.

Our full white paper — which is also an experiment in “runnable papers” using Jupyter notebooks — is available on GitHub.

IETF99: QUIC, TAPS, PAN(P)RG, MAPRG, ACME, BANANA, IPPM, … a busy week in Prague!

Two weeks ago, 16-21 July 2017, the IETF returned to Prague, as it apparently does every few years now, and the MAMI project went with it.

Charles Bridge at night

Piet de Vaere presents new results with PATHspider at ANRW ’17

As with last summer’s IETF (96, in Berlin), our meeting started a day early with the ACM/IRTF Applied Networking Research Workshop (ANRW), with two MAMI papers on the program: Korian Edeline presented copycat, a differential TCP/UDP treatment tool; and Piet De Vaere presented our latest results with PATHspider on ECN, DCSP and TFO measurements. In addition, putting the “applied” in ANRW, discussions at the final panel may lead to efforts to do some simple standardization for data interchange formats for very simple measurements; watch this space for future announcements.

The IETF meeting proper kicked off on Monday morning with TCPM where most discussion was focused around TCP’s Explicit Congestion Nodification (ECN). The MAMI project is working on an extension for more accurate ECN feedback (AccECN) that can be used as input for future, more advantage congestion control schemes. With the ECN deployment efforts from Apple and hopefully ECN support by default in QUIC, this can provide an interesting new space for research and experimentation.

Also on Monday, Brian Trammell of MAMI partner ETH co-chaired a second BoF on bandwidth aggregation approaches for multiply connected networks (BANANA); the discussion was much more focused than that in Seoul, and we anticipate a decision as to whether a working group will be formed soon.

Work on Post Sockets-related drafts continued at the TAPS working group meeting on Tuesday. Discussions in the working group focused on how to add security to the model of transport services worked out in RFC 8095. It has also become clear that discussions about the details of transport policies (addressed in depth by our sister NEAT project) will be central to the usability of a flexible transport layer (FTL) as envisioned by the MAMI project, and we will work together with the TAPS working group to define common policy models for future APIs. The authors of the Post Sockets draft also met after the WG meeting to discuss next steps with the document and bringing it up to date with our recently published paper. In addition, the MAMI project started new work in operation with Apple on security features in the transport stack.

The most important track at the IETF is the hallway track: meeting in the atrium about Post Sockets

Going on, the IPPM working group decided on Wednesday to attempt to change its charter to allow it to work, among other things, on the OAM work discussed in Chicago, which, as we noted, addresses some of the goals of MAMI MCP.

Wednesday afternoon marked the first meeting of the Path Aware Networking (PAN) proposed Research Group, co-chaired by Brian Trammell of MAMI partner ETH and Jen Linkova, a Google network engineer. PAN expands the question addressed by the MAMI project somewhat: what can we do with network architectures, protocols, and applications, when the endpoints are made explicitly aware of the paths between them and their properties? The creation of an IRTF research group as a venue to have these discussions will, we believe, be a major unanticipated outcome of the MAMI project, so we’ll go into more depth about PAN in a future blog post.

Thursday morning started with the usual MAPRG session chaired by Mirja Kühlewind, the MAMI project coordinator. Other than the last time, there was no Call for Contributions as the list of proposed presentation was just growing continuously. Check out the agenda for various talks on IPv6, DNS hijacking, or latency measurements. Or watch the recording. Please also consider to announce your measurement work in the maprg mailing list or use the mailing to check out if someone has that to share that might help your research work!

While the MAMI project was initially focused on the definition of a common wire image for encrypted transport protocols, it has become clear that QUIC is the currently-important such protocol under standardization in the IETF, so we focus our efforts on applying the principles we work out in the project to QUIC, as well. Thursday’s QUIC session was largely focused on discussion of the addition of explicit round-trip-time measurability to the protocol. For such a basic observable metric as latency, this discussion was surprisingly contentious, showing that emotions continue to run high in the IETF on the question of support for network management functions. On this particular question, we anticipate closure at the next IETF meeting in Singapore in November; watch this space for a future blog post on the details of the question.

While you might already got the impression that the meeting was packed, there was  more stuff to report also on Friday. Beside a second QUIC meeting where among other things ECN support in QUIC was discussed, there was the ACME session. Work on certificate delegation that was adopted by the working group at the last interim was presented there. Further it should be noted that the MAMI project was also presented at the IETF hackathon on Sunday, with ACME STAR and LoLa.

MAMI partners and advisors at a project lunch in Prague

It was, as always, an interesting, enlightening, and exhausting week. We’ll see the IETF again in November in Singapore, and we look forward to the IETF’s return to Prague in 2019.

Project coordinator Mirja Kühlewind waves goodbye to IETF99 in Prague.

Joint MAMI/MONROE workshop on Mobile Network Measurements (MNM’17) held on June 20 in Dublin

On June 20, we had a joint workshop with the EU-H2020 MONROE project on Mobile Network Measurements (MNM). The workshop was held in conjunction with TMA Conference 2017 in Dublin/Maynooth which was a great fit for this workshop and our project given the strong focus on measurements we have in WP1. For us the goal of the workshop was two-fold: of course it’s a good opportunity to disseminate the goals and results of our two projects but it was also a great chance to meet up with people that use or plan to use the MONROE testbed as well as PATHspider which is available on MONROE and build a focused community  around this group of people.

The workshop received 16 paper contributions, and 10 6-page papers were selected by the TPC, consisting of participants from both projects as well as MAMI EAB members, for presentation at the workshop. At this point again a big thanks to all TPC members, providing reviews within an incredibly short period of only two weeks!

The workshop was organized in three technical sessions focusing on network performance in mobile networks (like congestion forecasting or available bandwidth estimation) as well as application performance (e.g. Video QoE) over a mobile network and middlebox mangling in mobile networks (such as DCSP rewrites and NAT). The technical session where supplemented by a keynote on “From packets to knowledge: applying data science approaches to traffic measurement” held by Marco Mellia, Politecnico di Torino, Italy.

With a total of 18 registered participants, and a couple of visitors from the parallel workshops, the workshop provided lively discussion and a good opportunity to make people aware of MAMI’s measurement efforts, raising interest in a focused community of MONROE users as well as other researchers working on Internet measurements in academia and industry. And we are already discussing to have another workshop next year again!

QUIC Interim WG Meeting, Paris, June 6-8

The QUIC transport protocol, developed by Google and currently under standardization in the IETF, is of central interest to our project. QUIC is an encrypted transport protocol encapsulated in UDP, as those we aim to support with our MCP; indeed, our pilot MCP implementation targets QUIC as its overlying transport. So we are naturally very interested in QUIC’s development, and MAMI partners ETH (Mirja Kühlewind and Brian Trammell, in person) and UoA (Gorry Fairhurst, remotely) attended an interim meeting of the QUIC WG last week in Paris.

The meeting focused on laying out the features of a draft version of the protocol for the first interop test. As the interface between QUIC and TLS is entirely new and somewhat complex, interoperability will focus on connection establishment with cryptographic handshaking. Also on the agenda were discussions about the measurability of QUIC: how much information should QUIC explicitly radiate about its operation toward devices on path in order to support measurement, similar as questions we discuss for MCP. Here discussion is ongoing, but it seems that consensus on a set of explicit mechanisms for limited measurement and operations task is closer than it has been.

Chasing the Big NAT across Europe and the U.S. with NAT Revelio

In light of the IPv4 address scarcity problem, one approach towards prolonging the life of current IPv4 address allocations is to deploy Carrier Grade NATs (CGNs), where Internet Service Providers (ISPs) share the same public IPv4 address across multiple end users.  CGNs may bring a number of challenges for end users, service providers, content providers and government authorities. For example, there is some evidence that CGNs can cause dropped services in peer-to-peer applications, and lead to low performance of file transfer and video streaming sessions. Despite all this, CGNs offer an immediate relief to the IPv4 address scarcity problem, so it is likely that their popularity will increase over time.

Given the potentially disruptive impact of what seems a likely future scenario, it behooves policymakers, ISPs and Internet users to monitor the extent of CGN deployment in the Internet. But like many aspects of Internet structure, systematic measurement and monitoring of CGN deployment in the wide area is challenging. The MAMI project through Simula Research Laboratory, together with external collaborators at University Carlos III of Madrid and CAIDA/UCSD worked towards addressing this challenge. We built and perfected NAT Revelio, a tool that enables us to actively determine from within residential networks the type of upstream network address translation, namely NAT at the home gateway (customer-grade NAT) or NAT in the ISP (Carrier Grade NAT). Check our talk at PAM 2016 for an overview of how Revelio works.

We deployed Revelio on two large-scale hardware-based measurement platforms – RIPE Atlas in Europe and the FCC Measuring Broadband America (FCC-MBA) in the U.S. – with a total of 5,121 vantage points in over 60 ISPs. The FCC-MBA deployment consisted of 2,477 home routers operated by SamKnows in 21 large residential broadband Internet access service providers in the U.S. We also executed the Revelio tests from 2,644 Atlas probes in 43 ISPs mainly active in Europe. We ran the measurement campaign in two phases (May 2016 and August 2016) on both platforms. Based on the experimental results from the first phase (May 2016), we enhanced the test suite to account for a wide diversity of home network topologies and various access technologies. In the second phase of the measurement campaign (August 2016) we deployed the evolved Revelio suite to investigate the state of CGN deployment in broadband networks.

Our results show that 10% (6 out of 64) of the ISPs we tested have some form of CGN deployment. In particular, one ISP has a large-scale deployment where Revelio detected upstream CGN deployment from all 76 vantage points in that ISP. In the other 5 ISPs we observed evidence of a localized deployment limited to a subset of customers. We verified our results with representatives of the ISPs to validate our positive and negative inferences at the IP level. We confirmed the results for 4 of the 6 positive ISPs by personal communications with ISP representatives. The combination of the FCC-MBA and RIPE Atlas study represents (to the best of our knowledge) the largest active measurement study to date with confirmed CGN deployments in broadband networks at the IP-level granularity.

For a more in-depth analysis of our measurements please visit the openly available technical report.

What happened at IETF98 in Chicago, March 26-31?

Expecting an usual, not particular special IETF meeting, the 98th meeting two weeks ago in Chicago was actually quite exciting, in a positive sense. A lot of work, relevant to the architecture as well as measurement objectives of the MAMI project, made good progress and fostered interesting, still on-going discussions.

The meeting started off with the IP Performanec Metrics (IPPM) session Monday morning. A document on In-situ Operations, Administration, and Maintenance (IOAM) proposing a data model for telemetry and measurement data that can be applied to different (tunneling) protocols was discussed and received positive feedback from the group. This work is highly relevant to MAMI given measurement is one of the basic use cases for the Middlebox Cooperation protocol (MCP).

In the afternoon, there were several talks on middlebox benefits, state management, and privacy in the tsv-area meeting. Dave Dolsen presented draft-dolson-plus-middlebox-benefits and Brian Trammel (remotely) presented draft-trammell-plus-statefulness. Both drafts created quite some interest as well as discussion – any information that is exposed by an endpoint must first demonstrate benefits when consumed by the network.

Tuesday morning the IRTF Measurement and Analysis for Protocols (maprg) had a long slot with a number of presentations on security and privacy relevant topics. maprg usually sends out a Call for Contributions a couple of weeks ahead of the meeting. This time the group had a large number of very interesting submissions and could finally accept only about half of them to fit in the 2.5h slot. The talks presented data on DSCP, ECN, IPv6, Let’sEncrypt, broadcasted hostnames, weak keys in HTTPS, and censorship detected by the OONI project.

In the last session on Tuesdays Tommy Pauly presented Post Sockets in the Transport Services (taps) working group which is common work with the MAMI project updated after an one day workshop held in Feburary in Zürich.

Thursday started with the QUIC meeting were Mirja Kühlewind presented two drafts on Applicability and Managability as well as a proposal by Brian Trammell to add a packet number echo to the public QUIC header for RTT measurements. The draft received positive feedback to address one of the charter milestones and adoption will be confirmed on the mailing list. The proposal to add a packet number echo was lively discussed without reaching consensus yet. There was broad support for this addition but there were also privacy concerns that needs further discussion; similar to all information that will be exposed for network use in the header. QUIC will hold an interim meeting on June 6-8 in Paris.

Later that day, the Multipath TCP working group discussed the use of MPTCP for bandwidth aggregation in network scenarios where one low bandwidth fixed line link (DSL) is bonded with a high speed mobile link (LTE). Several approaches to address this use case have been proposed but there was so far no real consensus to move any of them forward. This time the discussion was led by the chairs attempting to compare the different proposals on a high level allowing the group to move forward. While there were also quite some people who indicated that they think this work should not be taken up by the IETF at all, a larger group of people were interested in this work and there was a clear indication for one of the proposals which is a solution that only requires a small addition to the SYN payload on the bonding link. The MAMI project is also focusing on this use case, using the MCP to signal preferences of the endhost to the bonding boxes in the network.

Thursday afternoon finally also the new MAMI T-shirts arrived (see photo above). Unfortunatly, they came only in to action during the bits-and-bytes and Friday morning that is traditionally rather quite. However, the next event is comig and the next IETF is already on the horizon in July in Prague! See you there! Good bye, Chicago!

Frist Prize at IETF98 Hackathon for “connection identifier” in DTLS group!

On Saturday and Sunday at the IETF there always is a Hackathon. This time at IETF-98 members of the MAMI consoritum were working on the implementation of the “connection identifier” in DTLS, a fairly recent proposal Hannes Tschofenig, Nikos Mavrogiannopoulos, and Thomas Fossati brought to the TLS working group.
The problem this proposal addresses is that an end-to-end DTLS session may silently die because an on-path Network Address Translation (NAT) middlebox dropped state after a (relatively) short period of quiescence. There is known trouble with UDP, as transport used by DTLS, that in-network state for this kind of traffic tends to vanish much more quickly than its TCP counterpart. As an example, the default timeout settings in the latest Linux kernel are 5 days for TCP and only 3 minutes for UDP — that is, three orders of magnitude!

© Stonehouse Photographic/Internet Society
IETF Hackathon, Chicago 25/03/2017

Obviously, there is a good reason for that: since UDP is connectionless, layer 4 devices have no way to possibly track a “connection” other than deep inspecting the flow, which is a pretty expensive activity. So it’s simpler for it to leave the onus of proving that a given UDP 4-tuple has an associated connection to the endpoints, by forcing them to regularly move bytes across. This state of affairs is clearly far from ideal for DTLS because, when a timeout happens in the NAT box, the victim endpoints need to re-negotiate a new crypto session context.

While this is generally annoying, it becomes even more nasty in cases where the client is a resource constrained, battery operated IoT device that woke up from its sleep cycle only to find its session doesn’t work anymore… The second nuisance is that the state that is dropped on the NAT box immediately becomes dead state in the server, consuming precious resources in vain. So, one popular workaround is to create synthetic “keep-alive” traffic, for example using the TLS heartbeat extension. This technique is a) not very robust (choosing the right keep-alive clocking depends on many external factors), and b) certainly, not affordable at smaller device scales, where waking up the “thing” to keep the NAT binding happy has the potential to quickly drain its battery.

© Stonehouse Photographic/Internet Society

Our solution – provide a connection identifier!  We propose to add a 32-bits blob that does one very simple thing: it decouples the DTLS session from the underlying 4-tuple, making it possible for the endpoints to dispatch incoming UDP traffic to the correct crypto session independently of any change in the underlying UDP address.

Sounds simple, right?  Yes, conceptually — apart from the birthday paradox hitting hard at large scales (see What we discovered at the Hackathon is that backporting it to an existing stack (ARM’s mbedTLS) can be more difficult than expected if one needs to maintain API compatibility.

Another practical complication is signaling the wire format change to the receiving endpoint so that it can parse the incoming frame correctly.  This is easy when you can make breaking changes (for example, when transitioning from 1.2 to 1.3).  Not as much if you have to maintain backwards compatibility with existing and deployed versions of the protocol. We have been discussing a couple of possible solutions around this issue — namely: using the Version field, or moving to an extensible record layer format.  We are still undecided on what “the right” approach would be.

Note that there are interesting privacy implications related to using a visible and potentially long-term identifier due to the obvious linkability properties of such a construct. Even if not all the use cases are problematic in this respect, some of them are, and thus we designed (but not yet implemented) a privacy friendly connection identifier based on HMAC-based One-time Password (HOTP) which can be rotated at client’s will at any point in time.

Enough with the babbling! If you read up to this point you will be glad to know the hackathon judges rewarded our herculean effort with the first prize. Yay!

Post Sockets Workshop

MAMI partner ETH Zürich hosted a small workshop last month to discuss a topic central to our vision: how to move network programming beyond the 1980s-era BSD sockets API.  The workshop had two main goals. First, to sync up and share concepts about abstract programming interfaces among researchers and developers working on future networking stacks. Several partners from our H2020 sister project NEAT, which has a more explicit focus on how applications see the network and transport layer, were in attendance.

Our second goal with the workshop was to further develop the concepts in the Post Sockets abstract programming interface. Post Sockets is explicitly asynchronous and message oriented, which is a better match to most network applications and all future transport protocols than the single-streamed SOCK_STREAM service offered by TCP. It breaks the hard relationship between a connection (called a “carrier” in Post, since it carries messages) and the path(s) carrying it, and separates short-term transport layer state (“transients” which hold transport-protocol specific information such as current sequence numbers and flow control windows) from longer-term state (“associations” which hold state such as cryptographic resumption parameters), allowing generalized fast resumption of sessions for commonly-communicating endpoint pairs. Like the NEAT API, it is intended to be transport-protocol independent, and interoperating with NEAT’s policy engine and transport protocol selection machinery is a goal of MAMI’s Post Sockets work.

Our current work on Post Sockets in MAMI focuses on implementation details: how message carriers, associations, and transients bind to underlying transport protocol implementations, specifically in situations with multiple candidate transport and network layer protocols, and where network address translation makes rendezvous non-trivial. Tommy Pauly from Apple will present Post Sockets at the IETF taps session next week in Chicago (checkout remote participation). Also watch this space for future announcements.

We thank the workshop attendees for a very productive conversation, and look forward to working together in the future to drag network APIs into the twenty-first century!

Path Transparency Observatory (PTO) goes live!

One of the core objectives of the MAMI project is to perform measurements and collect measurement data that can help to quantify middlebox impairments on the Internet. While various impairments on different layers of the protocol stack have been detected over the years, the provided measurement data are often not sufficient to assess how much of a problem there actually is. This information is needed to guide development of protocols extension or new protocols such as the Middlebox Cooperation Protocol (MCP) as proposed by MAMI.

Over the last year the MAMI project developed and implemented the Path Transparency Observatory (PTO), an open-source and publicly accessible repository for measurements of path transparency. An Internet path between a vantage point and a target is consider transparent if no impairment has been observed on that path and therefore the packet was successfully received without modification. The goal of the PTO is to collect data from different sources usually in different formats and represent them in a comparable way. To achieve this we apply a pre-processing step that transform the raw data in observations of a network condition c on a path p during a time interval t.

In parallel we also developed PATHspider (see below), an active measurement tool for A/B testing which now also integrates upload facilities for the PTO. We used and are using PATHspider as well as other measurements tools such as tracebox to continuously run measurements to test TCP extensions such as ECN or TFO as well as impairments on protocols on other layers such as the use of the DiffServ codepoint. Currently, only observations for our ECN measurement campaigns of the last couple of month are available over the public PTO interface. However, we have collected more raw data and are currently running the needed processing steps to successively release more observation data to be publicly available in the PTO:

For more information check out the ANRW PTO paper and Brian Trammell’s talk at the CAIDA AIMS workshop today! And of course, we will announce on twitter when more data is available!

2 Years of PATHspider Development

The current incarnation of PATHspider‘s first Git commit occurred on the 22nd of February 2015, now just over 2 years ago.

Since then, PATHspider has gained its own target list resolvers and has generalised for performing far more than just the ECN-dependent connectivity failures it was originally created to measure. Through plugins it can now perform measurements relating to other TCP features such as TCP Fast Open, and IP features such as DiffServ.

Active development of PATHspider continues, with the roadmap for PATHspider 2.0 provisionally laid out in the GitHub issue tracker. This includes:

  • seamless MONROE integration,
  • support for workflows integrating with the MAMI Path Transparency Observatory,
  • …and of course, new measurement plugins.

The following video is a visualisation of the last two years of work, featuring commits from the PATHspider authors: myself, Brian Trammell, Elio Gubser, Mirja Kühlewind, Andreas Germann, Piet De Vaere and Ana Custura.

Technical Plenay in Seville

On Feb 7/8 the MAMI project met in Seville for the third technical plenary meeting. Besides the great Spanish food and tapas crawl organized and guided by our technical project coordinator Diego Lopez, we made good progress on the technical aspects of the of the project.

We discussed next steps on PATHspider, both using PATHspider for more measurements as well as the development of new features in PATHspider. Currently the integrating into the MONROE testbed is underway. Further, potential integration with OONI was discussed to extend censorship measurements to low layer mechanisms. We plan for a new release in the next couple of month. Check out GitHub for the current implementation status and details on planned extensions.

The MAMI middlebox taxonomy that will be published end of June in D2.1 feeds into the development of PATHspider and other measurement tools by providing input to define a formalized test description and conditions to measured. Measurement data describing conditions of path transparency observations will soon be publicly available over the Path Transparency Observatory frontend webpage.

The MAMI project is also working on the specification of the Middlebox Cooperation Protocol (see draft-trammell-plus-spec), planning an endpoint implementation using QUIC as transport layer protocol for an HTTP/2 web service example use case and a based middlebox implementation supporting network diagnostics and differential treatment for low latency services.

Slow going for TCP Fast Open

As part of our continued effort to measure Internet path transparency with PathSpider, we’ve taken a look at the state of deployment and potential impairments to TCP Fast Open. TCP Fast Open is an extension to TCP that allows data to be placed on the first (SYN) packet of the TCP handshake, eliminating a round-trip time from TCP connections. It uses a TCP Option to exchange a cookie to be used on subsequent fast open connection attempts, to reduce the risk of TFO-based denial of service attacks.Interference with this option could cause path impairment of TFO, and indeed Christoph Paasch has reported that this is the case on about 20% of the access networks he observed.

We set out to measure possible impairment on content provision networks and in the Internet core, and found instead that TFO deployment on popular Web servers is mostly limited to Google, who invented TFO. Of 939,680 web servers taken from the Public Targets List (PTL), only 866 (0.092%) negotiated TFO in measurements taken this week. 690 (about 79.7%) of these are Google servers. Compared to measurements taken in October 2016, we see no appreciable change; then 563 of 635,681 web servers (0.086%) negotiated TFO. This is unsurprising, given that TFO requires significant changes to both client-side and server-side application logic as well as kernel support on both endpoints, we expect slow adoption compared, e.g., to ECN.

The story on DNS, where TFO is part of an effort to improve query privacy by using TLS and TCP for DNS, is similar: of the 53,267 authoritative name servers taken from the PTL, 56 (0.105\%) negotiate TFO, only three of which are not Google name servers; two of those three use an experimental ID, and fail to ACK data on the SYN.

A Public Targets List

Comparability of measurements and repeatability of experiments is key to science, and active Internet measurement studies are no different. Active measurement studies need not only open descriptions of the experiment (preferably with open source code and analysis, as we use in the MAMI project) but a comparable set of targets against to run experiments.

For some time, the Alexa top million web domains list has acted as a de facto standard of this comparable set of targets, and we’ve used it ourselves in several studies. Recent announcements by Amazon have made it clear that the Alexa top million list will no longer be freely available, and the announced cost for API access have made periodic resolution of the list too expensive for most active measurement studies. Therefore, we have started compilation of a public targets list to replace this de-facto standard.

The initial MAMI Public Targets List is available in GitHub; see the README there for details.


MAMI was represented by ETH Zürich at the 97th meeting of the Internet Engineering Task Force in Seoul.


The biggest news this time around was the first meeting of the QUIC working group, which will standardize a next-generation, encrypted transport protocol encapsulated in UDP based on Google’s QUIC and TLS version 1.3. Brian Trammell presented a concept for a transport-independent state machine for middleboxes at the meeting, to start the discussion about how QUIC’s wire image should interact with on-path devices, both present and future. While it’s not clear how much of the proposed transport protocol mechanism will be adopted into QUIC, discussion during and after the working group meeting has led to further refinement thereof.

Measurement and Analysis for Protocols (MAP) RG

Measurement and Analysis for Protocols (MAP) research group meeting

Mirja Kühlewind chaired a meeting of the Measurement and Analysis for Protocols Research Group (MAPRG), the first meeting since the group was officially chartered. The four presentations included techniques for passive delay measurements, a study of broadband access peformance using M-Lab, a study of the performance gain associated with HTTP2, and a characterization of traffic rate policing in the Internet.

Post Sockets, the API concept atop MAMI’s flexible transport layer (FTL), was discussed at the TAPS working group meeting. Tommy Pauly of Apple, a co-author of the Post draft, presented a quite similar approach. Post is very much a work in progress, but we’re happy to see broad interest in the concept, and look forward to developing it further with a broad group of collaborators both inside and outside the project.

Banana BoF (thanks @MeganRKruse)

Standing room only at the BANANA BoF (thanks @MeganRKruse)

The Bandwidth Aggregation for Networked Access (BANANA) BoF looked into standardizing approaches to share bandwidth on a customer network across two access links (usually one mobile and one terrestrial), as we explored in our ANRW paper last year. There is a lot of interest in doing work in this space, but not yet a lot of agreement as to what that work is yet; as is often the case, discussion continues on the mailing list. The MAMI project will also look into providing cooperative signaling for such approaches.

Yaron Sheffer presented a solution to the problem addressed in last meeting’s LURK BoF using short-term, automatically renewable certificates provisioned using the ACME protocol to the ACME working group meeting. The draft has a good chance of being adopted in the timeframe of the next IETF meeting, and work is progressing in parallel on a prototype.

Mirja Kühlewind and Brian Trammell led a discussion on protocol transitions in transport protocols at the Transport Area’s open meeting, both as an open forum on transition in an area full of efforts to deploy new work at Internet scale, and as input for an IAB document on the topic.

We’re back in Zürich now, and the jet lag is finally over. We’re already busy preparing for IETF 98 in March in Chicago, and the QUIC working group’s interim in Tokyo in January!

PATHspider Plugins

In today’s Internet we see an increasing deployment of middleboxes. While middleboxes provide in-network functionality that is necessary to keep networks manageable and economically viable, any packet mangling — whether essential for the needed functionality or accidental as an unwanted side effect — makes it more and more difficult to deploy new protocols or extensions of existing protocols.

For the evolution of the protocol stack, it is important to know which network impairments exist and potentially need to be worked around. While classical network measurement tools are often focused on absolute performance values, PATHspider performs A/B testing between two different protocols or different protocol extensions to perform controlled experiments of protocol-dependent connectivity problems as well as differential treatment.

PATHspider 1.0.1 has been released today and is now available from GitHub, PyPI and Debian unstable. This is a small stable update containing a documentation fix for the example plugin.

PATHspider now contains 3 built-in plugins for measuring path transparency to explicit congestion notification, DiffServ code points and TCP Fast Open. It’s easy to write your own plugins, and if they’re good enough, they may be included in the PATHspider distribution at the next feature release.

We have a GitHub repository you can fork that has a premade directory structure for new plugins. You’ll need to implement logic for performing the two connections, for the A and the B tests. Once you’ve verified your connection logic is working with Wireshark, you can move on to writing Observer functions to analyse the connections made in real time as PATHspider runs. The final step is to merge the results of the connection logic (e.g. did the operating system report a timeout?) with the results of your observer functions (e.g. was ECN successfully negotiated?) and write out the final result.

We have dedicated a section of the manual to the development of plugins and we really see plugins as first-class citizens in the PATHspider ecosystem. While future releases of PATHspider may contain new plugins, we’re also making it easier to write plugins by providing reusable library functions such as the tcp_connect() function of the SynchronisedSpider that allows for easy A/B testing of TCP connections with any globally configured state set. We also provide reusable observer functions for simple tasks such as determining if a 3-way handshake completed or if there was an ICMP unreachable message received.

Visit the PATHspider website to learn more.

Web Performance is in the Eye(org) of the User

Tremendous effort is undergoing to make the Web faster. However, quantifying speed on the Web is complex: usually we are attempting to capture human perception with a computer-generated metric. In many studies, participants are simply shown a page loading, in person, in a controlled environment, which has a clear scalability problem. MAMI partners at Telefonica Research (in collaboration with Carnegie Mellon University) took a different approach and built Eyeorg, an automated system for crowdsourcing Web Quality of Experience (QoE) measurements. Eyeorg uses crowdsourced participants to scale and shows videos of pages loading to provide a consistent experience to all participants, regardless of their network connections and device configurations. In their paper, to be published at CONEXT 2016, they present hands-on experience from using Eyeorg to 1) study the quality of several PLT metrics, 2) compare HTTP/1.1 and HTTP/2 performance, and 3) assess the impact of online advertisements and ad blockers on user experience. A key result they observed is that many videos have two modes, one for participants who consider the pages “ready” when the primary content is in place and one for those who wait for auxiliary content like advertisement (see below). These results show the potential of Eyeorg to measure the impact changes to the web have on people. For example, Eyeorg can be used to evaluate TCP vs. QUIC, TLS 1.2 vs TLS 1.3, HTTP/2 push/priority strategies, web design techniques like domain sharding or image spriting, browser plugins, or even in-network services like Google’s Flywheel compression proxy.


Some sites exhibit multiple modes; here, some participants consider the site “ready” before the ads load.




The MAMI project was out in force at last week’s IETF 96 meeting in Berlin. The Measurement and Analysis for Protocols Research Group, founded by MAMI partner ETH and chaired by coordinator Mirja Kühlewind and external advisor Dave Plonka from Akamai, was officially chartered as a research group of the Internet Research Task Force during the meeting. MAPRG provides a place to discuss protocol-design-relevant measurement techniques and results. MAPRG’s Monday evening meeting included several interesting presentations on ongoing measurements, including an interesting CDN-based survey of active IPv4 space and dynamic address allocation policies by Phillip Richter.


Full room at the QUIC BoF on Wednesday morning, IETF 96, Berlin

The biggest event on the IETF calendar this time was the QUIC Birds of a Feather (BoF) session on Wednesday morning, where on the order of 400 participants — about a third of the attendees of the IETF as a whole — discussed the formation of a working group to standardize the QUIC UDP-based transport protocol for HTTP and HTTP-like applications developed by Google. It seems likely that a working group will be formed in the coming weeks. Brian Trammell of ETH co-chaired the BoF. MAMI’s measurements of UDP impairment in the Internet are relevant to the deployability of QUIC, and the project will participate in the development of the protocol on the background of this measurement.

Another BoF of interest was a second Limited Use of Remote Keys (LURK) BoF, which decided not to form a working group to handle key management and delegation within content delivery networks, but rather to solve the problem using short-lived certificates, perhaps provisioned using the ACME protocol.

The Transport Area Open Meeting on Monday saw a presentation by Volker Sypli of Germany’s telecom regulator BNetzA, representing the European association of telecom regulators BEREC, invited by MAMI project coordinator and Transport Area Director Mirja Kühlewind, to explain the BEREC network neutrality guidelines. The discussion was interesting and spirited. While MAMI is not concerned with network neutrality per se, path impairment and neutrality violations are related, and work on the Path Transparency Observatory may contribute to the development of measurement tools for network neutrality, as well.

At the Transport Services (TAPS) working group on Thursday, Brian Trammell presented Post Sockets, a potential API for the MAMI flexible transport layer. Discussion following the presentation indicates some interest in defining next generation APIs for transport, and the project will follow up with interested collaborators.


Brian Trammell explains the Path Layer, PLUS BoF, IETF 96, Berlin

Most important for the MAMI project as a whole, though, was the Path Layer UDP Substrate (PLUS) BoF on Thursday morning, which discussed and aimed to form a working group to standardize explicit cooperation approaches over UDP, informed by MAMI’s Middlebox Cooperation Protocol (MCP) development. While more work will be needed before a working group can be formed, there was significant interest in the room in continuing work on the effort, and we received valuable feedback from the community with respect to the scope and use cases, the details of the protocol mechanisms, and the privacy characteristics of explicit cooperation approaches in general. A presentation detailing the abstract mechanisms of the present proposal can be seen here. Internet-Drafts describing the PLUS proposal in more detail will appear in the coming weeks. Watch this space for an announcement!

Applied Networking Research Workshop at IETF-96

The first ACM, IRTF & ISOC Applied Networking Research Workshop 2016 (ANRW ‘16) was held in co-location with IETF-96 on July 16, 2016, in Berlin. The MAMI project conributed with one full paper and three short papers describing multipath bonding as an use case for MCP, PATHspider’s inital release, as well as the basic structure of MAMI’s Path Transparency Observatory (see ANRW’16 webpage for papers and presentations):

  • Multipath bonding at Layer 3. (Full)
    Maciej Bednarek (ETH Zurich), Guillermo Barrenetxea Kobas (Swisscom), Mirja Kühlewind (ETH Zurich), and Brian Trammell (ETH Zurich).
  • Towards a Multipath TCP Aware Load Balancer. (Short)
    Simon Liénardy (Université de Liège) and Benoit Donnet (Université de Liège).
  • PATHspider: A tool for active measurement of path transparency. (Short)
    Iain R. Learmonth (University of Aberdeen), Brian Trammell (ETH Zurich), Mirja Kuhlewind (ETH Zurich), and Gorry Fairhurst (University of Aberdeen).
  • Towards an Observatory for Network Transparency Research. (Short)
    Stephan Neuhaus (Zürcher Hochschule für Angewandte Wissenschaften), Roman Müntener (Zürcher Hochschule für Angewandte Wissenschaften), Korian Edeline (Université de Liège), Benoit Donnet (Université de Liège), and Elio Gubser (ETH Zurich).

anrw-postersWhile this was a great opertunity to present the work of the MAMI project, there have been a number of very interesting and related papers. To learn more about an extended API for Multipath TCP, multi-homing in IPv6, the effects and cost of Happy-Eyeballs,  and measurement on IPv6 and DSCP usage, check out the ANRW’16 webpage.

Thanks to Lars and Colin for organizing this very interesting and interactive workshop!

2. MAMI Plenary Meeting in Berlin

The MAMI project held its second technical plenary on July 13-15, 2016, in Berlin. This time also members of the External Advisory Board (EAB) participated. Thanks for your time, fruitful discussions and very valuable input! Also special thanks Joe Hildebrand for providing meeting space in the Cisco openBerlin Innovation Center (as well as the free barista team building event..)!


We started on our first afternoon with an open discussion slot focusing on two main topics of the project: test specifications and Middlebox Cooperation Protocol (MCP) design. While the MAMI project is currently running a number of path transparency tests on the Internet, we decided to also provide a specifications of current and planned tests to enable external parties to run similar measurements on other testbeds. Stay tuned, we will provide further measurement results as well as the specs soon!

Just before the meeting, we released the first PATHspider version “Phidippus audax” (0.9.0). In case you wonder why PATHspider only has six legs, there is a sad story to tell which involves a compression middlebox…


After submitting our first technical deliverable D3.1 on Use Cases and Requirements for MCP, we now enter the actual protocol design phase. We discussed and agreed on the basic mechanisms for signaling from and to middleboxes, as also outlined at the PLUS BoF meeting at IETF96 in the following week (see slides here).

Further, work package 2 on Middlebox Classification and Modelling started this month. This work will be informed by our own middlebox measurements and testing as well as additional tests we plan to run in cooperation with our EAB member Paul Hoffman on the ICANN middlebox testlab.

All in all, we had a very productive and fruitful meeting, including an old-fashioned french dinner, and a great tour of craft breweries in bars throughout Berlin.

70% of popular Web sites support ECN

One of the primary goals of MAMI’s measurement work is to quantify path transparency in the Internet: how likely a given transport protocol or feature is to work on which paths, and how these features break. Earlier work by MAMI partners ETH and the University of Aberdeen on this topic focused on Explicit Congestion Notification (ECN) in TCP, a feature that allows congestion to be detected without packet loss. Our paper, based on measurements in August and September 2014 and published at PAM 2015, found that 56% of IPv4 and 65% of IPv6 hosts serving the Alexa top million websites would negotiate ECN if the client requested it, which at the time was not the default in any major client operating system. ECN negotiation attempts could lead to connectivity issues and fallback to non-ECN usage for 0.42% of IPv4 and 0.05% of IPv6 servers in the top million.

In the meantime, Apple has added ECN negotiation by default on the client side in developer previews of Mac OS X and iOS, and our patch adding fallback in the case of ECN failure to non-ECN usage, as specified in RFC 3168, has been added to the Linux kernel. The tooling for the 2015 paper is evolving into a generic path impairment measurement tool called PathSpider. So what’s the state of the Alexa top million today?

ecn-trendWe recently ran a measurement from a single vantage point, a DigitalOcean server in Amsterdam, to the set of unique IPv4 and IPv6 addresses serving the top million websites, and found that 432544 of 617873 (70.005%) of IPv4 addresses and 20262 of 24472 (82.797%) IPv6 addresses will negotiate ECN. This continues a trend ETH started observing in 2013, shown here.

The proportion of servers requiring fallback has not changed appreciably: 0.44% of IPv4 and 0.11% of IPv6 servers. This reflects the two different forces at work: ECN support on the server side generally follows the operating system defaults, and web hosting machines generally run a recent Linux, the first operating system with server side ECN on by default. Connectivity problems, however, are often a function of faulty middleboxes, which are more slowly replaced, or firewall rules explicitly disabling ECN traffic for dubious reasons.

Detailed analysis behind this blog post is available here; the raw data it runs on will be made available shortly.


MAMI was represented at RIPE 72 in Copenhagen with two presentations by ETH. First, Mirja Kühlewind presented a possible application of MAMI’s Middlebox Cooperation Protocol (MCP) in “What if we designed measurement as a first-order service?”, an exploration of what it would take to build Internet measurement on protocol foundations stronger than ping and a collection of ingenious hacks.

Brian Trammell also presented “Internet Path Transparency Measurements using RIPE Atlas”, on MAMI’s use of the RIPE Atlas platform to find differential treatment between UDP and TCP, and the incidence of UDP blocking on access networks. Here, we found about 3% of Atlas probes to be on networks where UDP is severely impaired. More in-depth analysis of this question will appear in an upcoming paper, currently under submission.


IETF95 was held last week (3-8 April) in Buenos Aires, Argentina, and the MAMI project was out in force. First and foremost, project coordinator Mirja Kühlewind assumed the office of IETF Transport Area Director at the plenary meeting on Wednesday evening.


Congratulations on the yellow dot!

Brian Trammell presented a small research study performed with RIPE Atlas at the Measurement and Analysis for Protocols (MAP) proposed Research Group, on the deployability of UDP encapsulation-based approaches to deploying new transport protocols, as MAMI will explore with its middlebox cooperation protocol (MCP) and flexible transport layer (FTL). He also presented a potential approach to supporting low-latency service signaling using a new IP Differentiated Services codepoint at the “Alternatives to Content Classification for Operator Resource Deployment” (ACCORD) BoF, where explicit cooperation with respect to radio access network were discussed.

The Transport Services (TAPS) working group was interesting as well: our abstract work on decomposing transport protocols into features, to be published shortly, has led to the first concrete proposals on implementation.The NEAT project presented their proposed API for an adaptive transport layer, and MAMI will bring an FTL API proposal to the next meeting in Berlin.

The Limited Use of Remote Keys (LURK) BoF was also of special interest to the project, as key management protocols useful in CDN and operator networks may present a way forward for cooperation with trusted middleboxes. MAMI partner TID is working on an implementation of a key server for LURK, and we’ll be watching continued developments in this space.

Being in Buenos Aires, the project made sure to enjoy the local cuisine (thanks Oscar Gonzalez for local arrangements!)


And in the marketing department, MAMI’s measurement / experimentation / architecture stickers were a big hit.


We’re all back home and mostly recovered from the jet lag, but here are only three months left until the next IETF meeting in Berlin in July. We’ll be hard at work preparing, including a potential Birds of a Feather session to discuss the standardization of approaches like MAMI’s MCP. Watch this space!

MAMI Promo Tour started

Over the last few weeks, we went around and gave a couple of presentations about the current and planned work in MAMI. Feedback was very positive and we identified more friendly people to work with. If you want to learn more about MAMI (illustrated with some nice and colorful picture), check out the slides on the Publications page. Especially the following presentations will provide you a good overview:

MAMI Kickoff Meeting in Zürich

The MAMI project held its first plenary meeting at the Networked Systems Group at ETH Zürich this week.


The technical agenda included detailed discussions of measurement tool design, the beginnings of the detailed specification of our Path Transparency Observatory (as initially described in our RAIM paper), and further refinement of the use cases, requirements, and security model for our Middlebox Cooperation Protocol.

The nontechnical agenda included the obligatory-for-Switzerland-in-winter cheese fondue.


We also began technical coordination with two related Horizon 2020 projects with whom we share partners: MONROE is a FIRE testbed we’ll use for measurements as well as middlebox cooperation experiments, and NEAT aims to redesign application-transport interactions in the Internet. We look forward to a productive collaboration with both projects.

In addition, we formally appointed our External Advisory Board: Joe Hildebrand, Paul Hoffman, Dirk Kutscher, Allison Mankin, Szilveszter Nadas, and Dave Plonka. Welcome aboard, and many thanks to our advisors!

MAMI webpage updated

The MAMI webpages has been updated to now contain more information on the project’s objectives, work package structure as well as standardization efforts.
Also check out our github at

Next thing to do is our first plenary meeting on Feb 8-10 in Zürich. Stay tuned to find further information here on current and planned activities after the meeting!

Dagstuhl Seminar on Global Internet Measurement

Mirja Kühlewind (ETH) and Brian Trammell (ETH) from the MAMI project kicked off the new year by participating in a seminar at Schloss Dagstuhl on Global Measurement Practice and Experience. Dagstuhl seminars are always thought provoking, but in addition to informing the design of MAMI’s forthcoming Internet Path Transparency Observatory, two specific initiatives started there that we hope will ease the creation of measurement studies like the one we’re undertaking: first, a common platform for active measurement platforms on small hardware (e.g. Raspberry Pi); and second, a cooperative, Internet-measurement-focused “storage cloud” for reliable distribution of archival data generated by such platforms. Watch this space for future developments.

Welcome to MAMI

MAMI (“Measurement and Architecture for a Middleboxed Internet”) is a European Commission Horizon 2020 funded research project. Our consortium is made up of seven European universities, research laboratories, and industrial partners. We aim to rearchitect the Internet to allow explicit cooperation between endpoints and middleboxes, restoring the promise and innovation potential of the original end-to-end architecture of the Internet while enabling appropriate in-network services to ease management and scalability of ever more demanding applications.

A central tussle in today’s Internet is that between the desire for privacy, which requires strong encryption to protect, and the need to efficiently manage network traffic. Current approaches to traffic management typically require access to plaintext and application payload, which is fundamentally incompatible with the privacy goal. By replacing implicit cooperation (e.g. middleboxes which assume they have access to traffic and understand its semantics) with explicit cooperation (endpoint and application signaling to middleboxes and vice-versa), we aim to break this tussle.

MAMI has two broad goals:

  • empirical research to develop what kinds of middleboxes perform what kinds of manipulations on what paths, in order to understand the present Internet: both the extent and variety of in-network services as well as the degrees of freedom for incremental deployment of new protocols; and
  • development of a Middlebox Cooperation Protocol (MCP) that would enable explicit cooperation between endpoints and middleboxes, transport innovation, and ubiquitous encryption simultaneously.

MAMI will officially start in January 2016, but we’re already hard at work:

  • the How Ossified is the Protocol Stack? (HOPS) proposed research group within the Internet Research Task Force (IRTF) met at IETF 92 in Prague and will meet again at IETF 93 in Yokohama; the RG provides a forum for exchanging information on measurements of middlebox impairment in the Internet and tools for researching it
  • A paper on an initial proposal for the MAMI Middlebox Observatory will be presented at the IRTF/ISOC workshop on Research and Applications of Internet Measurements (RAIM).

Keep an eye on this blog for news about the project as we go forward.