The Future of Middlebox Cooperation in the Internet Protocol Stack

As you may have read in our last post, the MAMI project is celebrating the hard-won, long-discussed consensus to add a spin bit to QUIC. The spin bit was conceived as the minimum useful explicit signal one could add to a transport protocol, and we think the benefit for the overhead is quite worth it. Though it exposes “just” RTT, latency (together with data rate, which is available in QUIC simply by counting packets and bytes on the wire) is possibly the most important metric for understanding transport layer performance diagnosing all matter of transport-relevant network problems, and the spin signal itself can also be observed to infer loss and other issues with network treatment of a packet stream. The definition and deployment of the spin bit is therefore a victory for making network protocols more measurable while preserving the privacy gains from encryption, and a victory for network operations and management.

When we started the MAMI project, we had a much more extensive vision for middlebox cooperation: our Path Layer UDP Substrate (PLUS) proposal provides a generalized signaling mechanism allowing signals from the sender to the path element, as well as a mechanism for allowing on-path elements to communicate information to the receiver (and back to the sender with the receiver’s assistance) in a safe, limited way, all with end-to-end integrity protection. However, we failed to achieve consensus to move forward in the IETF with this proposal by forming a working group, so we refocused our efforts on a more modest implementation of the explicit signaling concept.

Selective exposure, however, has a pretty toxic reputation in the IETF, as it “breaks” TLS, by adding a third-party to a two-party protocol and changing its security properties, reducing or eliminating integrity and confidentiality protection with respect to a set of entities in the network that may or may not be known to one or both endpoints. In our recently published white paper, the outcome of the M3S meeting, we argued that all approaches in this space must provide transparency and control to the endpoints.

In hindsight, more important to the controversy around PLUS was (in our opinion) a fundamental misunderstanding about what is meant by the term “middlebox cooperation”, as there are two broad approaches to supporting on-path operations on encrypted traffic being pursued in the industry:

  • Explicit signaling approaches like PLUS, where information about encrypted traffic is carried separately from the encrypted traffic itself.
  • Selective exposure approaches like mcTLS, mbTLS, eTLS and so on, which share secrets from the end-to-end cryptographic context with middleboxes, allowing them to decrypt (and in some cases, modify) traffic between the endpoints.

We focused on explicit signaling in MAMI, in part due to the difficulty of building selective exposure approaches that still provide useful security properties for the end-to-end communication. However, large parts of industry are more interested in selective exposure, because building and deploying selective exposure approaches promises to be more easily backward-compatible with existing protocols and network hardware. PLUS, after all, envisioned deploying a new transport protocol stack. We examine the tradeoffs in these approaches in another white paper, released today.

Among the arguments presented against PLUS at our BoF in Berlin in July 2016 were doubts that the protocol was safe in general: that it provided new side-channels for data exfiltration or hooks for coercion. We looked into this as part of our own internal study of integrity-protected, encrypted-payload middlebox cooperation approaches like PLUS, finding in a white paper, also released today, that the additional attack surface presented by PLUS is negligible.

The looming end of the project does not mean the end of the problems we set out to solve, or interest among the project participants and the field in general in improving privacy and evolvability through encryption while retaining limited support for in network functions. So, we ask ourselves: what’s next?

Several threads of work in this space will outlive MAMI:

Our efforts to add measurability to QUIC, which appears to represent the first real hope of deployable transport evolution in the last three decades, will of course continue. And QUIC itself will continue to evolve. The group of implementors and network folks we’ve helped pull together around the spin bit will keep experimenting with the best ways to make it possible to do passive measurement and diagnosis, as QUIC traffic increasingly supplants TCP.

Our work on experimentation with the Loss/Latency Tradeoff (LoLa) signal continues, as we noted in the previous blog bost. LoLa may enable better performance for real-time communications traffic and was first specified as a bit in the PLUS header continues, with a possible implementation as a DSCP codepoint.

The IRTF Path Aware Networking Research Group (PANRG), founded by the project, is pursuing work both in understanding why previous attempts to consider middleboxes in transport layer interactions have failed, and in exploring a set of properties of end-to-end paths on which future explicit signaling methods could be built.

Further, there is ongoing work on UDP options in the IETF TSVWG and a discussion about support for proxying of UDP-based, encrypted protocol started in the at IETF-103 with some initial presentations about requirements and use cases in the tsvarea session.

We are very happy that this important discussion is continued and that we as a project could provide a valuable input based on measurement and experimentation. And while the project’s time had to come to an end at some point, the MAMI contributors will keep working on this in order to keep the Internet operational as well as improve  its robustness in operation, performance, and security.


Posted in Uncategorized | Leave a comment

IETF 103 Bangkok

The MAMI project was back at the IETF for the last time — as a project, anyway — last month at IETF 103 in Bangkok. Join us on a trip through the week!

the IETF 103 venue in Bangkok

Hackathon: Loss-Latency Tradeoff in Mobile Networks

Most of us arrived on the Friday before the meeting in order to attend the now already traditional hackathon. This time we set up a table to pull together a few different projects related to measurement and measurability. Friends of the project Fabio Gulgarella and Mauro Cociglio ran an experiment running on a 2-bit variant of the spin-bit they dubbed “delay sample”. Thomas Fossati, as main representative of the MAMI project, has been doing some measurements on the impact of the LoLa bit on the mobile network.

This is another instance of the “one-bit signalling” topic we’ve been working on in MAMI – notably, a Loss/Latency Tradeoff (LoLa) bit was present in the original PLUS proposal. These experiments are concerned with mapping the LoLa signal to a dedicated low-latency bearer in the LTE core and air interface. This is different from how mobile networks are deployed today, to actively ignoring or bleaching any end-user signal, and bundling all flows to and from the Internet into a single “default” bearer, whose buffering characteristics that are not compatible with low-latency traffic. The established behaviour is partly rooted in the desire to prioritise operators’ voice services over competing over-the-top services, but that market consideration has lost relevance in the recent years.

It looks like the incentives are now aligned in the direction of allowing more suitable treatment of Internet real-time flows. However, a couple of preconditions need to be satisfied before we can move on from the status quo. First, the real-time flows must be efficiently identified so that they can be put on the right queue especially nearby the bottleneck link, which in 4G mobile networks is typically the air interface. This is at odds with the rise of encrypted and multiplexed transports, which has the potential of increasing the cost/accuracy ratio of DPI over the acceptable threshold. LoLa instead is this simple clear-text signal set by the endpoints which would enable a super efficient classifier. The question is how can we trust the endpoints to set the LoLa signal correctly? What if they lie? Would they get an advantage at the expenses of other flows? The hackathon experiments were aimed at answering these core questions. On one hand we wanted to see whether giving mobile devices a low-latency bearer in addition to the default one would make a difference in terms of QoE. And secondly, we wanted to measure what happens when an endpoint lies about the nature of its flows. So Thomas spent his Saturday and Sunday hacking around the LTE module of NS-3 and getting these fine results which he also presented at the HotRFC session on Sunday night. We got some positive feedback about the idea from a few mobile operators in the room.  Thomas, Gorry Fairhurst and Mirja Kühlewind together with Vz, AT&T, Orange and Three managed to schedule a side meeting for later in the week to discuss both the finer points and an overall coordination strategy also involving and the 3GPP liaison Georg Mayer and Florin.

Hackathon: Path MTU Signaling in QUIC

In addition, Tom Jones worked on Path MTU signaling mechanisms for QUIC at the QUIC hackathon table. Tom’s draft describing “Packetization Layer Path MTU Discovery for Datagram Transports” will also be incorporated in QUIC. MTU discovery is also under discussion in the 6MAN Working Group, where there are two competing proposals to add new mechanisms that allow better signalling of MTU by routers a long the path, Truncate and Hop By Hop:

  • The Truncate proposal adds a MTU Destination Option, the intention of this option is that when a router has to discard a packet for being too large it also truncates the datagram. The MTU Destination Option carries the original packet length allowing the end host to detect this packet as ‘truncated’, and the end host uses this packet to send an ICMP PTB message.
  • The Hop By Hop proposal creates a new IPv6 Hop By Hop option. This option is added to small packets, when a router forwards a packet with this option it examines the MTU field in the option, if the MTU is larger than the routers out going link it updates the MTU field to match this link size.

Tom implemented the HBH proposal at the IETF 103 Hackathon to get a sense of how practical it was to add to a real network stack — he had previously implemented the Truncate proposal on the flight back from IETF102 in Montreal. His inplementation in FreeBSD achieved self-interop between end host and router in the two days. Running code is a core part of the IETF ethos, after IETF 103 we are able to compare how the HBH and Truncate proposals compare and differ in implementation.

Monday: Evolving Transport

On Monday, the official sessions started with an interesting discussion about the evolution of transport and the role of proxies in the Transport area open meeting featured with presentations from Lucas Pardue and Ben Schwartz’s team on HiNT and Helium and a presentation from Thomas on PEPs role and fate in an end-to-end encrypted Internet. One of the takeaways of the session is that if we want to keep PEPs alive, which looks like a sensible thing to do at least in some scenarios (cfr. satellite), we need to re-think the security model in particular how the transport security association is modelled and what is its relationship with the security context that sits at the application layer. QUIC in particular, collapses the two into a single inextricable blob, making proxying an all-or-nothing option, which is far from ideal. The “transport proxy” gang (including us, Lucas and Ben’s team) is going to chat further again in December to discuss next steps.

Then the first TSVWG session on Monday touched a little on the PMTUD work happening in other working groups, as already mentioned above. And we had a presentation by friend-of-the-project Colin Perkins on the Transport Header Encryption Draft, recently adopted by the WG, which explores transport-specific considerations of the trend toward more ubiquitous encryption and encryption deeper down the stack.

Tuesday: Measurement and Analysis for Protocols and the MAMI Lunch

The founded-by-MAMI Measurement and Analaysis for Protocols Research Group (MAPRG) met on Tuesday. This time we had an 2-hour slot (slightly shorter than usually) with interesting talk about UDP checksum issues, QUIC performance over  satellite, comparison of CoAP/MQTT/HTTP in vehicular scenarios, as well as measurements on Certificate Transparency and OCSP Must Staple deployment.

Tuesday also saw our traditional lunch for MAMI and friends:

Tuesday and Wednesday: QUIC and the Spin Bit

Since MAMI is largely focused on transport evolution, and transport evolution work in the IETF is now focused on the completion and deployment of the IETF standard version of QUIC, the “main event” of the IETF for the project was the QUIC working group. QUIC met on Tuesday and Wednesday. Tuesday’s meeting covered smaller issues, including the applicability and manageability drafts, presented by Mirja.

After 18 months of heated discussion and thorough experimentation, the latency spin bit was  been accepted during the second session on Wednesday as a feature of the base QUIC protocol. Implementation and participation in signaling are optional, though several large endpoint implementors indicated they will implement and enable the bit by default. The consensus is to add just the spin bit, without the Valid Edge counter described in our recent IMC paper.

QUIC hums on the Spin Bit (image: @csperkins)

This proposal has been pretty controversial, not so much technically but rather from a political perspective – being a proxy for the wider (and highly flammable) discussion around the tension between protocol encryption and network functions in the IETF. One key takeaway of this whole experience (and a key result of the MAMI project as a whole) is that getting the right balance between opaqueness and transparency a protocol’s wire image should provide is a very tough job and requires a great deal of analysis and experimentation along many different axes.

After three years working on these topics, and despite the hyper-sensitivity to privacy issues in the Internet many of us in the project have, we are even more then ever convinced that a world of completely opaque wires is not a place where we would like to spend our future, and that total privacy and total privatisation of the Internet infrastructure are two sides of the same coin. The Internet is a common good and we need to keep it that way, finding the right balance between what is visible and what is not is a tough job but we need to resist the temptation to use the encryption hammer for things that are not necessarily privacy related – e.g., as anti-ossification techniques. We very much hope that this decision enables now more experimentation and will lead to more experience with these kind of explicit signals which can be the basis for more discussion about additional features that would support measurability.

Wednesday: UDP Options, MARC, and Encrypted SNI

Wednesday morning also saw the second TSVWG Working Group session discussing the status of UDP Options, with Tom presenting an implementation report. Tom also followed up on questions asked on the TSVWG mailing list about the future and suitability of FRAG, LITE and AE UDP Options. This led to an enthusiastic discussion about the suitability and use of UDP Options at all. Tom continued with another presentation, this time on Datagram PLPMTUD, this slot was grouped with presentations from the authors of the 6MAN drafts attempting to address PTMUD. These presentations led to very productive working group discussion on both PLPMTUD and PMTUD, with finding the right MTU for a network still being an open problem.

The video interest group (GGIE) side meeting we presented MARC (mobile assisted rate controller), the congestion controller developed by Morteza and Gorry in the project that uses (among others) the mobile throughput guidance signal from the eNodeB to adapt the sending rate to the (very quickly and widely) changing conditions of the mobile network. There was an entertaining discussion around the topic including a nice digression on how to carry the MTG signal in QUIC using UDP options.

In TLS there was more discussion about the encrypted SNI proposal which is trying to remove one privacy-critical bit of clear-text from the TLS wire image. Of course SNI is relevant to policing middleboxes (enterprise TLS proxies, censorship and parental control filters), hence some of the usual scuffle. There are a couple of issues with the current ESNI proposal, one of which is pretty substantial in terms of the ability to reliably deploy the feature, but there is a huge amount of energy being poured into finding a good solution so hopefully we are going to get something done soon-ish. On another topic, the DTLS 1.3 protocol specification is complete and ready for review by the working group. We need some security analysis there which will take a fair amount of cycles. In the meantime, the DTLS 1.2 connection identifier is done and is currently in working group last call. So this should provide an OK solution (although not ideal because of the higher correlation properties compared to 1.3) until 1.3 is finalised and deployed at scale.

Wednesday: Transport Services (TAPS)

The Transport Services (TAPS) working group, in which MAMI heavily contributes, continues its work to standardize an interface to a modern Internet transport layer. The group had a productive session, solving some of the open issues for the API document. In addition Tommy Pauly, another friend of the project, presented a mapping of the TAPS interface to QUIC, leading to a discussion about mappings for different DNS variants (DoT, DoH, … Do*) as a quite interesting use case for taps, given it could actually provide an opportunity to simplify future mappings of DNS to other transports.

If you’d like to help the TAPS effort as well, it’s now the right time to review the architecture document as the first of the taps trilogy, to be published around IETF 104 in March 2019.

Thursday: ACME, Round Tables, and Not-So-Secret Cabals

The very last session of the meeting was ACME. MAMI has a couple of drafts there: the STAR document, which is going through a second LC as we speak due to some adjustments we had to do related to a pretty big last minute change in the base ACME spec, and the delegation draft, which we have re-written as an ACME profile (as per request in Montreal) which – good news everyone! – we got consensus to adopt by the working group. As per IETF process, the official CfA confirmation is currently on the mailing list and should be over in a few days.

On Thursday, the long-running Post Sockets Secret Cabal met, though now this meeting is less an insurgency bent on overthrowing BSD Sockets and more merely an afterparty for the TAPS meeting where we distribute issues to work on before the next interim.

The Post Sockets Not So Secret Cabal (image: @csperkins)

Thursday also saw our LoLa round table. The meeting went well and it was decided that the coordination for the various activities – which are likely to span more than one SDO – should happen in the Internet Group in GSMA. The activity is in progress, and last week I presented the plan at the GSMA IG plenary.

After that we had a de-brief with the spin-bit folks, including Brian joining remotely via Thomas’ laptop :-), to plan the next steps in measurement experimentation on QUIC. This was a fitting end to the technical part of the meeting in Bangkok, as well as the MAMI project’s official engagement in the IETF.  For us, the IETF crowd of the project, it definitely has been an incredible pleasure working in MAMI and progress these important topics in standardization. MAMI has been a fantastic project, which tackled a core technical and societal tussle with an abundance of courage (and/or recklessness, you choose), with a truly amazing group of people! Thanks everybody!

Posted in Uncategorized | Leave a comment

The MAMI Management and Measurement Summit

Just before IETF 101 in London in March, the MAMI project hosted an invitation-only MAMI Management and Measurement Summit (M3S), bringing together researchers, engineers, and vendors for a focused discussion on how to meet the challenges posed to network measurement and network management by the increasing deployment of strong encryption and the extension of encryption down the stack. Today, we release “Challenges in Network Management of Encrypted Traffic”, a white paper covering the discussions and distilling the recommendations that came out of the meeting.

This discussion has played out in multiple forums, including the IETF, for some time, underpinning discussions and debates from the (failed) proposal to include static keys in TLS, to continue to provide for “business as usual” monitoring, to the spin bit proposal in QUIC, which replaces implicit passive measurability of RTT with an explicit signal. Recognizing that neither business as usual, nor forging forward with the deployment of strong crypto down the stack and invalidating most of the current practice of network management, are tenable positions, the attendees converged on a set of recommendations for future protocol design and network architecture to partially meet these challenges:

  1. Protocols and networks must provide for independent measurability of important metrics when these measurements may be contested: one outcome of increasing encryption is that existing independent passive measurement techniques will become less effective.
  2. Future secure protocols should support different security associations at different layers: approaches that integrate transport and application-layer security (such as QUIC) make limited or no provision for network management that need to interact with the transport protocol while not breaking application layer security, in contrast to the TLS-over-TCP status quo.
  3. Transparent middleboxes should be replaced with middlebox transparency: the dominant architectural pattern for in-network functions today is that of the “transparent middlebox”, which attempts to the extent possible to be undetectable to the endpoint(s). While this has benefits for initial deployment, it makes it impossible to build cooperative protocols, where the middlebox and its functions are visible to the endpoints, and the endpoints have some control over how their traffic is treated by the network (in the last instance by detecting a middlebox with which they do not wish to cooperate, and cease using the path).
Posted in Uncategorized | Leave a comment

IETF 102 Montreal

The IETF week started on Saturday with the hackthon. Diego, Pedro and I (Thomas) have been working on a variety of topics: a “STAR Requests” slide deck for the upcoming SecDispatch meeting, an architecture design for an end-to-end ACME STAR demo, and the one-bit experiment harness- reproducing and analysing a bug in iperf3 which prevents our trafic to produce reliable / precise emulation of a throughput-seeking flow.  This was pretty quickly done and we got eventually absorbed by the SUIT / TEEP table.  The back-story for this is I had brought from Cambridge to Montreal 40 boards with me, done especially for the hackathon by ARM – Cambridge is a very small city where everyone seems to know everyone else. Though we didn’t get to do much with the boards and the TEEP protocol, we got to know a bit better the technology, talk to people involved and, in my case, share my experience with fellow Nokians IoT folks.  The QUIC spin-bit table, with Marcus, Al, Roni and Emile was also active on experimenting with 1-bit spin signal and a bunch of heuristics to reject bad RTT samples in presence of reordering. See Marcus presentations [1] and [2].)  This was, as usual, very useful, ludic, extemporary and genuinely fun!

On Monday, at the SecDispatch session Diego spoke about “Generating Certificate Requests for STAR Certificates“.  This complements ACME STAR and is a building block needed for closing the loop in the identity-owner controlled name delegation workflow using X.509 certs we designed in MAMI.  We need to find a sensible home for this document so that it can receive a thorough security analysis. The outcome of the discussion was quite surprising but, IMO, pretty good – even though there is a fair amount of running code that I’m going to trash as a result.  Martin and Ted noted that STAR request could be re-written in terms of pure ACME, provided the proof of the identifier ownership is not requested by the IdO.  So, the subsequent step was to go back to ACME and discuss adoption of a STAR Request there, which is what we’ve done on Tuesday (see below).  Earlier that day, at the TLS session, we discussed DTLS connection ID for 1.2 and 1.3, handling an interesting fallout from the recent major header refactoring in DTLS 1.3 on to the content type allocation (and its repercussions on the marking strategy envisaged for CID in 1.2.).  The discussion is not settled yet, and continues on GitHub [1], [2], [3].

On Tuesday in ACME, STAR officially entered WGLC, which means, finger crossed, we should be pretty close to finalising this one.  In relation to STAR Requests, instead, we were asked to produce a new version of the protocol as an ACME profile, which can then be called for adoption by the WG.  So here it is, waiting for the base ACME draft to do the final adjustments to make it through IESG.

At the same time, in HTTPbis, HELIUM & HiNT were introduced.  This work lead by Google and BBC R&D has some interesting overlap with MAMI in that it provides a tunnelling protocol and side channel with a proxy which might end up being useful in tuning CC for mobile networks in presence of fully encrypted end-to-end traffic.  I especially liked the alternative tagline “a proxy is an honest middlebox” in Ben’s presentation.

Discussion in taps, later that afternoon, many focused on the API draft. While there are still a whole bunch of open issues, thers broad agreement on the general concepts and a lively discussion making good progress to resolve these issues. The taps working group will hold a vitual interim on September 12.

QUIC met on Wednesday morning. Work is winding up on the IETF standard version of this new transport protocol, which means it’s time to start talking about how it will interact with the rest of the Internet. Brian presented the “operations” drafts for the first time, though they’ve been under construction since shortly after the working group was chartered. The first of these, the applicability document, gives pointers on building applications (other than HTTP) on top of QUIC, and the other, the manageability document, provides an independent guide to the protocol for operators of the networks its traffic will be carried over. Discussion in the WG pointed out that this division might not be precisely the right one — pointedly, the definition of an abstract interface from the application down to QUIC is missing, and doesn’t necessarily belong in either. Work on this will ramp up in the weeks before Bangkok.

The Wednesday concluded with the tsvarea session where Brian and Ted (Hardie) presented two IAB documents defining the wire image and path signals. Further Christoper Paasch and Ian Sweet provided some insights on implementation/deployment challenges for MPTCP and QUIC.

Thursday started of with the usual maprg (Measurement and Analysis for Protocols research group) session. The agenda cover a broad spectrum of topic including reordering in QUIC, as well privacy issue of RTT measurements/bufferbloat presented by Brian and measurements of PMTU by Gorry.

Later on in LAMPS we discussed “STAR at large” for adoption.  Yoav’s presentation is very neat, and does a good job discussing the non-web use cases for STAR certs (e.g., VPNs, Software defined storage, but also autonomic networks).  The WG asks us to clarify a few points, in particular how to make the semantics of a no-revo cert explicit to relying parties, before taking it as an working item.  So we are back to the desk on this one.

Last but not least, the Path Aware Networking (PAN) Research Group met midday on Friday, during the last slot of the meeting, for the first time as a no-longer-proposed-but-actually-chartered research group. In addition to proposed talks, discussion centered around two work items: first, a review of bad ideas in transport-path cooperation, and second, a start on answering the research group’s first question about path property definition and representation.

We had the usual amount of fun, high-bandwidth, high-energy hallway discussions that make the IETF meeting a pretty special thing.  Next round, sadly the last one under the aegis of MAMI, is going to be Bangkok. Stay tuned!

Posted in Uncategorized | Leave a comment

Summer School on Internet Path Transparency Measurements

On June the 11th the Electronics Research Group hosted the MAMI Summer
School on Internet Path Transparency Measurements in Aberdeen, Scotland.
This consisted of a few hands-on workshops, with participation
both on-site and remote via video conference.

The summer school started with Korian and Justin demonstrating Tracebox
through a variety of topologies. The participants then worked on their
own trying to uncover middleboxes and hidden topologies using a variety
of tools, including tracebox and paris-traceroute.

To follow, the history and development of PATHspider were presented by
Iain Learmonth, one of the creators of PATHspider. Iain also delivered
an interactive Scapy tutorial followed by teaching students how to use
Scapy to create PATHspider plugins using the Evil Bit as an example.
The participants ran plugins against a list of real targets and produced

To finish off, Brian Trammel delivered a presentation on the PTO (Path
Transparency Observatory), and helped the participants upload to it the
results they collected in the previous session. The participants then
learned how to query the PTO in order to examine the data they just
uploaded. In an interesting twist of events, the measurements found ECN
manipulation by a middlebox in the eduroam network in Aberdeen – mission

If you missed the summer school, all slides and materials can be found

Posted in Uncategorized | Leave a comment

2. Edition of the MAMI/MONROE workshop on Mobile Network Measuremnt (MNM’18)

On June 25, we held another edition of the Mobile Network Measuremnt (MNM’18) workshop together with the MONROE project. As last year the workshop was co-located with the Network Traffic Measurement and Analysis Conference (TMA 2018) which was held this year in Vienna. We started the day with a keynote from Varun Singh, the CEO of – a start-up from Helsinki that collects and analyses WebRTC measurements. While he was giving us some interesting insights in the technical work and challenges of monitoring real-time communications, his elaborations about the challenges and problems as a young start-up were as least as exciting!

The technical agenda of the workshop were split between 6 talks into two sessions, basically between the layers: first focusing on Network QoS and (mobile) Coverage, and then Multipath and Application Performance (in mobile networks). Further Özgü Alay, the co-ordinator of the MONROE project, and Iain Learmonth, the main maintainer of PATHSPider, gave an introduction on the use of the MONROE platform and mobile measurements using PATHspider.

One more important thing to mention: We did print MAMI labelled M&M’s for the MNM workshop… of course! I’d say those where a big success but we gladly have still some left. So if you meet us the next time at some event (e.g. come to our SIGCOMM tutorial on Repeatability and Comparability in Measurement (RCM) where we introduce tracebox, PATHspider, and the PTO), chances are good to get some as well!

Of course we also stayed for the rest of the TMA meeting in Vienna! Beside our two paper on Exploring usable Path MTU in the Internet and Tracing Internet Path Transparency, there where a lot of great talks and keynotes, including an expert summit on Tuesday of the week. After all a big thanks to the organizors for the great meeting as well as great social events (every night of the entire week)!

Posted in Uncategorized | Leave a comment

IETF101 in London, March 17-23

Last month, the IETF returned to the Hilton Metropole near Paddington Station in London for its 101st meeting.

Hello, London!

While it is always nice to go to an IETF meeting in Europe (and therefore suffer less from jet-lag), in this specific hotel the challenge is to find your way around and actually make it to your meeting in time. The meeting room are distributed over three “wings” in the first as well as ground floors as well as in the upper third and “lower third level” (i.e., the sub-basement, next to the Underground), with a less than optimal elevator configuration:

However, the meeting itself was very productive, despite the labyrinth! As is now customary, the week started early on Saturday and Sunday already with the Hackathon where MAMI was present with two projects and a total of 6 people working on Scalable, Privacy-preserving In-Network Measurement (i.e., the QUIC Latency Spin Bit) and a testbed for 1-bit optimisations for the mobile access network based on the Loss-Latency tradeoff.

Monday, we mainly focused on transport topics with a presentation of the soon-to-be-finished AccECN TCP extension in tcpm, an interesting discussion about framing in QUIC (i.e. whether or not to use DTLS as QUIC’s wire image), and a general discussion about TCP encapsulation in tsvarea.

On Tuesday both of the research groups that have grown out of the MAMI project met: theMeasurement and Analysis for Protocols (MAP) and the proposed Path Aware Networking (PAN) research groups. MAPRG’s two and a half hour slot contained many interesting presentations, covering both papers from, e.g., IMC as well as “previews” of work presented at PAM 2018 the following week in Berlin.

PANRG met for the third time as a proposed RG, which means that the process of actually forming the group officially is underway now. The meeting had a productive discussion and a lot of positive feedback, indicating that there is interest in continuing work in the group. There seem to be two broad areas of research the group will tackle going forward: exploring how to add “path awareness” to the Internet architecture (in the vein of the PLUS work pursued by the MAMI project), and continuing work on various not-yet-ready-for-standardization techniques to use path information at the transport layer.

The MAMI project, together with the H2020 NEAT project and engineers and researchers from Apple, the University of Glasgow, and TU Berlin, proposed a new architecture for the Transport Services working group, and an abstract interface for that architecture based in large part on MAMI’s Post Sockets and flexible transport layer work. These drafts were adopted by the TAPS working group, and will form the basis of a new standard abstract API for the transport layer.

The new TAPS cabal, working out the details after the adoption of the new architecture drafts. (Thanks Colin Perkins for the photo!)

MAMI was also busy in TLS, presenting a proposal to extend the DTLS header and discussing the nuances of the DTLS connection Id encoding, and in ACME where we asked for WGLC of the STAR document.

The “main event” for the project, so to say, took place on Thursday morning with a discussion of the QUIC Spin Bit, a facility for supporting passive round-trip time measurement despite the encryption of the QUIC header. This discussion took the majority of a two and a half hour session, and was quite lively: for the first time in our experience at an IETF meeting, the microphones at an IETF meeting had to be moved to keep the line from running out the door.

“How many engineers does it take to spin one bit?”

While the working group still could not come to consensus to add the spin bit directly to the protocol at this time, the outcome was a good one for the project (and for the concept of explicit measurability and in our opinion, for the Internet at large): one bit has been reserved for experimentation with the spin bit, with a directive to reserve a further two for experimentation with additional signaling such as the Valid Edge Counter (VEC) presented at the meeting, with a draft to be published under working group change control for coordinating larger-scale experimentation.

All in all, it was a great week in London, and we’re already looking forward to July’s IETF 102 meeting in Montreal!

Posted in Uncategorized | Leave a comment

PATHspider has exciting new features in release 2.0.0

PATHspider is a free-software extensible path transparency measurement tool that performs path transparency measurements using either real network stacks or packet forging. A new major version, 2.0.0, has just been released and is packed with new features to expand the range of measurement tasks it can perform.

For the evolution of the Internet’s protocol stack, it is important to know which network impairments exist and potentially need to be worked around. PATHspider performs A/B testing between two different protocols or different protocol extensions to perform controlled experiments of protocol-dependent connectivity problems as well as differential treatment.

One new feature that can simplify the creation of tests for path transparency to new protocols in development is the inclusion of a packet forging framework, Scapy (thanks to Ēriks Dobelis for his work on porting Scapy to Python 3, without which this would not have been possible).

This means that even before the specification for a new protocol or extension is written, before any code exists, you can already be testing for possible issues. This feature was already used to explore the possibilities for a new DiffServ codepoint for lower effort traffic and reported on at an IRTF MAPRG meeting.

The API for developing plugins for new measurements has also been greatly simplified. A lot of effort has gone into refactoring large chunks of the codebase to remove code duplication in plugins and ensure plugin authors only have to write the code that they need to.

When using real network stacks, connection helpers are now provided for HTTP and DNS using pycurl and dnslib. This greatly simplifies the creation of plugins that are toggling kernel options or iptables rules as you now only need to write the function to perform the system-wide configuration and the traffic generation is handled for you.

Analysis of data is now also simplified as PATHspider no longer outputs individual flows but instead waits for all the flows to be available and performs an automated analysis to generate conditions that apply to a path, for example whether or not the use of a feature has broken connectivity. It will also determine your public IP address if behind a NAT, and look up the ASN of the vantage point and include these in the computed network path in the output.

A completely new feature in PATHspider allows the use of the built-in flow meter without actively generating any traffic. This can be used to examine another device that is generating traffic or to examine traffic on a link aggregating many devices to discover how clients behave and what typical Internet traffic looks like with regard to the protocol features in use.

PATHspider includes comprehensive documentation to help you get started. If you have Vagrant installed, you can have a working PATHspider environment as simply as “vagrant up”. Other installation methods are described in the documentation.

In the near future, there will be more work on the test suites that allow you to verify the installation of PATHspider is working correctly and the addition of a benchmarking command to optimise the speed at which PATHspider is running on a particular machine to balance speed with dropped packets. There will also be more extensive documentation on packaging your plugins so that they can be more easily shared other researchers and deployed to remote measurement vantage points.

If you have any ideas for interesting plugins, you could file a GitHub issue or send a tweet to @iainlearmonth. If it sounds interesting, it may make it into the next release. If you’d like to follow PATHspider development, you can also join #pathspider on

In order to complete the inclusion of pycurl for traffic generation, it was necessary to add support to pycurl for a couple of additional features (thanks to Oleg Pudeyev for reviewing and merging those changes). Unfortunately these changes were only recently released and so it may be necessary to install pycurl from source. If using the Vagrantfile this will be done for you, on a Debian system the following will get you going:

apt-get install python3-libtrace python3-sphinx python3-straight.plugin python3-setuptools pylint3 python3-pep8 python3-pyroute2 python3-pip unzip python3-nose python3-stem
apt-get build-dep python3-pycurl
pip3 install 'pycurl>='
pip3 install 'pathspider>=2.0.0'
Posted in Uncategorized | Leave a comment

IETF 100

Good morning, Singapore!

A few of us from the MAMI project started arriving in Singapore on Friday, in order to participate in the hackathon, which has become an IETF tradition. A few hours later we were at our tables, Thomas Fossati and James Bulmer working on a STAR Requests implementation, and Brian Trammell sitting with the QUIC and TLS tables to work on passive measurability of the protocol with mokumkuoren. We had a day and a half of fun coding, patching specs, improving test coverage, and generally hacking about.  Pretty good progress indeed, a few nice chats with old and new friends and great food and beers.  Thank you hackathon organisers!

Test coverage improved by the end of the hackathon…

The next step for STAR is to get the e2e demo up & running on our dear Blue Box with a miniaturised but fully functional CDN talking to an ACME STAR CA in time for IETF 101 in London in March.

Back to the hotel we posted acme-star-01, since we had missed the pre-IETF cutoff date and we needed people to get a chance to glance through the changes which had been quite abundant. The updated draft includes  an “implementation status” section (as per BCP205) documenting the work that Diego De Aguilar Cañellas has done on top of Boulder and Certbot (LetsEncrypt’s server and EFF’s client, respectively) to add the new STAR flow in ACME.

Then Monday arrived, and the meeting started. Diego went to TRANS to talk about ACME STAR (slides) and discuss the cost that an increase in log ingestion of one or two orders of magnitude poses on the Certificate Transparency infrastructure. Based on the observation that all STARs belonging to the same ACME order are basically equivalent modulo their validity dates and serial number, we also prepared and presented a napkin design that uses a new SCT type to address the scale problem. The discussion (youtube link) was, as often happens in the IETF, very instructive but inconclusive. We went away without a clear answer whether this is going to cause troubles or not.  The reactions up to this point are scattered all over the spectrum, ranging from “omg, this will melt the world” to “nah, the log can cope” to “meh, future problems”.

Monday’s session of the Measurement and Analysis of Protocols research group (MAPRG), co-chaired by Mirja Kühlewind, included a presentation by Brian of Principles for Measurability in Protocol Design (slides). This paper articulates our vision for measurement as a first-class function of the protocol stack.

Another meeting of the not-very-secret Post Sockets cabal

Tuesday started off with the Transport Services (TAPS) WG, where discussion focused on whether the working group should take on work in defining abstract programming interfaces for applications atop a dynamic . Here, discussion focused on Post Sockets, a realization of MAMI’s flexible transport layer (FTL). We came to no conclusion, but will schedule a meeting in the margins of our upcoming plenary in Cambridge in January to further develop Post Sockets into an architecture for flexible transport services.

The QUIC mic line (©Stonehouse Photography)

The first session of QUIC was Tuesday afternoon. A slightly congested mic line and very robust discussion surrounded the wrap up of the design team for the “spin bit”, designed to provide explicit passive measurability of end-to-end latency in QUIC flows, replacing TCP timestamps for this purpose. While the design team itself was unable to come to consensus to add the spin but to the protocol (though it did conclude that passive latency measurement poses no known threat to privacy), there was a balance of support in the room for adding passive latency measurability to the protocol, and a sense that the spin bit is a good method for doing so. However, work to achieve consensus is ongoing; watch this space for future posts about our experiences with implementation and use of the spin bit.

Thursday was definitely a busy day.  In TLS we did the call for adoption for the connection identifier for (D)TLS.  That went really smooth and the draft has been adopted – pending confirmation on the mailing list, obviously.  The co-authors have slightly different opinions on a few key points, including implicit vs explicit signals and the protocol friendliness to troubleshooting (deja-vu?).  But we all agree this solves the big issues related to connection migration and NAT rebinding that we already discussed in a previous post and the important thing here is that the TLS working group reckons this is worth spending working group cycles on.

Thursday afternoon saw the second meeting of the Path Aware Networking research group (PANRG), and included presentations on path property dissemination and interfaces for path control (hello again, Post Sockets), as well as an examination of open questions in bringing path awareness to the Internet architecture. We see the general area of path-aware networking as being an unexpected legacy of the project.

Later, in the ACME session we presented the updates we’d been working on in the months following Prague.  The document is in good shape, the protocol flow should be stable and my impression is that once we complete the security and operational analysis, the document should be ready for last call. After ACME, we had another informal STAR-centred meeting organised by Yoav to talk about generic short-lived certificates that automatically renew which may or may not depend on the ACME ecosystem – for example, based on ANIMA, or on proprietary systems – and may or may not address the HTTPS use case and address instead IPsec, non-web uses of TLS & SSH in enterprise and datacentre-type environments.  The meeting was well attended with more than 20 people at the table (a couple of CDNs, middlebox vendors, web folks, mobile network operators, academia, other SDOs) all bringing their own experience and perspective on the issues related to certificate revocation (one of the core motivations to look into STAR) and the solution space.  The discussion was great – with use cases in NSF, vehicle-to-vehicle, SAN, IPsec, and of course the Web – though a tad too short: many had to run, including Diego and I to our traditional MAMI dinner 🙂  One core thing that was concluded is that the “short” in short-term is a very fluid concept and must be defined on a case by case basis.  In fact, the exact definition of “short” should match the time it takes to the revocation information (CRL and/or OCSP) to propagate to the relying parties.  We hope to continue the exchange on the SAAG list or maybe in an ad-hoc list.

Goodbye, Singapore!

With that, we bid farewell to Singapore! See you all at IETF 101 in London!

Posted in Uncategorized | Leave a comment

Is Internet RTT reliable for geolocation?

Short, short answer: nope, don’t bother. While this is probably obvious to any of you with network engineering experience, we thought we’d use RIPE Atlas to have a look into this question anyway.

In the context of an ongoing conversation about the explicit exposure of RTT information to devices on path in the IETF standard version of the QUIC protocol, we’ve briefly looked into how much of a threat Internet-observable per-path RTT is to geoprivacy of one of the endpoints. It turns out that the old network operations rule of thumb that a millisecond of RTT is 100km long adds a whole lot of uncertainty — a fact which also confounded some recent work on RTT-based anycast detection by Cicalese et al. Only in cases where one is very, very lucky — microseconds lucky — in the placement of the vantage points from which RTT measurements are taken can one use RTT measurements for elimination-based geolocation.

Our full white paper — which is also an experiment in “runnable papers” using Jupyter notebooks — is available on GitHub.

Posted in Uncategorized | Leave a comment