Is Internet RTT reliable for geolocation?

Short, short answer: nope, don’t bother. While this is probably obvious to any of you with network engineering experience, we thought we’d use RIPE Atlas to have a look into this question anyway.

In the context of an ongoing conversation about the explicit exposure of RTT information to devices on path in the IETF standard version of the QUIC protocol, we’ve briefly looked into how much of a threat Internet-observable per-path RTT is to geoprivacy of one of the endpoints. It turns out that the old network operations rule of thumb that a millisecond of RTT is 100km long adds a whole lot of uncertainty — a fact which also confounded some recent work on RTT-based anycast detection by Cicalese et al. Only in cases where one is very, very lucky — microseconds lucky — in the placement of the vantage points from which RTT measurements are taken can one use RTT measurements for elimination-based geolocation.

Our full white paper — which is also an experiment in “runnable papers” using Jupyter notebooks — is available on GitHub.

Posted in Uncategorized | Leave a comment

IETF99: QUIC, TAPS, PAN(P)RG, MAPRG, ACME, BANANA, IPPM, … a busy week in Prague!

Two weeks ago, 16-21 July 2017, the IETF returned to Prague, as it apparently does every few years now, and the MAMI project went with it.

Charles Bridge at night

Piet de Vaere presents new results with PATHspider at ANRW ’17

As with last summer’s IETF (96, in Berlin), our meeting started a day early with the ACM/IRTF Applied Networking Research Workshop (ANRW), with two MAMI papers on the program: Korian Edeline presented copycat, a differential TCP/UDP treatment tool; and Piet De Vaere presented our latest results with PATHspider on ECN, DCSP and TFO measurements. In addition, putting the “applied” in ANRW, discussions at the final panel may lead to efforts to do some simple standardization for data interchange formats for very simple measurements; watch this space for future announcements.

The IETF meeting proper kicked off on Monday morning with TCPM where most discussion was focused around TCP’s Explicit Congestion Nodification (ECN). The MAMI project is working on an extension for more accurate ECN feedback (AccECN) that can be used as input for future, more advantage congestion control schemes. With the ECN deployment efforts from Apple and hopefully ECN support by default in QUIC, this can provide an interesting new space for research and experimentation.

Also on Monday, Brian Trammell of MAMI partner ETH co-chaired a second BoF on bandwidth aggregation approaches for multiply connected networks (BANANA); the discussion was much more focused than that in Seoul, and we anticipate a decision as to whether a working group will be formed soon.

Work on Post Sockets-related drafts continued at the TAPS working group meeting on Tuesday. Discussions in the working group focused on how to add security to the model of transport services worked out in RFC 8095. It has also become clear that discussions about the details of transport policies (addressed in depth by our sister NEAT project) will be central to the usability of a flexible transport layer (FTL) as envisioned by the MAMI project, and we will work together with the TAPS working group to define common policy models for future APIs. The authors of the Post Sockets draft also met after the WG meeting to discuss next steps with the document and bringing it up to date with our recently published paper. In addition, the MAMI project started new work in operation with Apple on security features in the transport stack.

The most important track at the IETF is the hallway track: meeting in the atrium about Post Sockets

Going on, the IPPM working group decided on Wednesday to attempt to change its charter to allow it to work, among other things, on the OAM work discussed in Chicago, which, as we noted, addresses some of the goals of MAMI MCP.

Wednesday afternoon marked the first meeting of the Path Aware Networking (PAN) proposed Research Group, co-chaired by Brian Trammell of MAMI partner ETH and Jen Linkova, a Google network engineer. PAN expands the question addressed by the MAMI project somewhat: what can we do with network architectures, protocols, and applications, when the endpoints are made explicitly aware of the paths between them and their properties? The creation of an IRTF research group as a venue to have these discussions will, we believe, be a major unanticipated outcome of the MAMI project, so we’ll go into more depth about PAN in a future blog post.

Thursday morning started with the usual MAPRG session chaired by Mirja Kühlewind, the MAMI project coordinator. Other than the last time, there was no Call for Contributions as the list of proposed presentation was just growing continuously. Check out the agenda for various talks on IPv6, DNS hijacking, or latency measurements. Or watch the recording. Please also consider to announce your measurement work in the maprg mailing list or use the mailing to check out if someone has that to share that might help your research work!

While the MAMI project was initially focused on the definition of a common wire image for encrypted transport protocols, it has become clear that QUIC is the currently-important such protocol under standardization in the IETF, so we focus our efforts on applying the principles we work out in the project to QUIC, as well. Thursday’s QUIC session was largely focused on discussion of the addition of explicit round-trip-time measurability to the protocol. For such a basic observable metric as latency, this discussion was surprisingly contentious, showing that emotions continue to run high in the IETF on the question of support for network management functions. On this particular question, we anticipate closure at the next IETF meeting in Singapore in November; watch this space for a future blog post on the details of the question.

While you might already got the impression that the meeting was packed, there was  more stuff to report also on Friday. Beside a second QUIC meeting where among other things ECN support in QUIC was discussed, there was the ACME session. Work on certificate delegation that was adopted by the working group at the last interim was presented there. Further it should be noted that the MAMI project was also presented at the IETF hackathon on Sunday, with ACME STAR and LoLa.

MAMI partners and advisors at a project lunch in Prague

It was, as always, an interesting, enlightening, and exhausting week. We’ll see the IETF again in November in Singapore, and we look forward to the IETF’s return to Prague in 2019.

Project coordinator Mirja Kühlewind waves goodbye to IETF99 in Prague.

Posted in Uncategorized | Leave a comment

Joint MAMI/MONROE workshop on Mobile Network Measurements (MNM’17) held on June 20 in Dublin

On June 20, we had a joint workshop with the EU-H2020 MONROE project on Mobile Network Measurements (MNM). The workshop was held in conjunction with TMA Conference 2017 in Dublin/Maynooth which was a great fit for this workshop and our project given the strong focus on measurements we have in WP1. For us the goal of the workshop was two-fold: of course it’s a good opportunity to disseminate the goals and results of our two projects but it was also a great chance to meet up with people that use or plan to use the MONROE testbed as well as PATHspider which is available on MONROE and build a focused community  around this group of people.

The workshop received 16 paper contributions, and 10 6-page papers were selected by the TPC, consisting of participants from both projects as well as MAMI EAB members, for presentation at the workshop. At this point again a big thanks to all TPC members, providing reviews within an incredibly short period of only two weeks!

The workshop was organized in three technical sessions focusing on network performance in mobile networks (like congestion forecasting or available bandwidth estimation) as well as application performance (e.g. Video QoE) over a mobile network and middlebox mangling in mobile networks (such as DCSP rewrites and NAT). The technical session where supplemented by a keynote on “From packets to knowledge: applying data science approaches to traffic measurement” held by Marco Mellia, Politecnico di Torino, Italy.

With a total of 18 registered participants, and a couple of visitors from the parallel workshops, the workshop provided lively discussion and a good opportunity to make people aware of MAMI’s measurement efforts, raising interest in a focused community of MONROE users as well as other researchers working on Internet measurements in academia and industry. And we are already discussing to have another workshop next year again!

Posted in Uncategorized | Leave a comment

QUIC Interim WG Meeting, Paris, June 6-8

The QUIC transport protocol, developed by Google and currently under standardization in the IETF, is of central interest to our project. QUIC is an encrypted transport protocol encapsulated in UDP, as those we aim to support with our MCP; indeed, our pilot MCP implementation targets QUIC as its overlying transport. So we are naturally very interested in QUIC’s development, and MAMI partners ETH (Mirja Kühlewind and Brian Trammell, in person) and UoA (Gorry Fairhurst, remotely) attended an interim meeting of the QUIC WG last week in Paris.

The meeting focused on laying out the features of a draft version of the protocol for the first interop test. As the interface between QUIC and TLS is entirely new and somewhat complex, interoperability will focus on connection establishment with cryptographic handshaking. Also on the agenda were discussions about the measurability of QUIC: how much information should QUIC explicitly radiate about its operation toward devices on path in order to support measurement, similar as questions we discuss for MCP. Here discussion is ongoing, but it seems that consensus on a set of explicit mechanisms for limited measurement and operations task is closer than it has been.

Posted in Uncategorized | Leave a comment

Chasing the Big NAT across Europe and the U.S. with NAT Revelio

In light of the IPv4 address scarcity problem, one approach towards prolonging the life of current IPv4 address allocations is to deploy Carrier Grade NATs (CGNs), where Internet Service Providers (ISPs) share the same public IPv4 address across multiple end users.  CGNs may bring a number of challenges for end users, service providers, content providers and government authorities. For example, there is some evidence that CGNs can cause dropped services in peer-to-peer applications, and lead to low performance of file transfer and video streaming sessions. Despite all this, CGNs offer an immediate relief to the IPv4 address scarcity problem, so it is likely that their popularity will increase over time.

Given the potentially disruptive impact of what seems a likely future scenario, it behooves policymakers, ISPs and Internet users to monitor the extent of CGN deployment in the Internet. But like many aspects of Internet structure, systematic measurement and monitoring of CGN deployment in the wide area is challenging. The MAMI project through Simula Research Laboratory, together with external collaborators at University Carlos III of Madrid and CAIDA/UCSD worked towards addressing this challenge. We built and perfected NAT Revelio, a tool that enables us to actively determine from within residential networks the type of upstream network address translation, namely NAT at the home gateway (customer-grade NAT) or NAT in the ISP (Carrier Grade NAT). Check our talk at PAM 2016 for an overview of how Revelio works.

We deployed Revelio on two large-scale hardware-based measurement platforms – RIPE Atlas in Europe and the FCC Measuring Broadband America (FCC-MBA) in the U.S. – with a total of 5,121 vantage points in over 60 ISPs. The FCC-MBA deployment consisted of 2,477 home routers operated by SamKnows in 21 large residential broadband Internet access service providers in the U.S. We also executed the Revelio tests from 2,644 Atlas probes in 43 ISPs mainly active in Europe. We ran the measurement campaign in two phases (May 2016 and August 2016) on both platforms. Based on the experimental results from the first phase (May 2016), we enhanced the test suite to account for a wide diversity of home network topologies and various access technologies. In the second phase of the measurement campaign (August 2016) we deployed the evolved Revelio suite to investigate the state of CGN deployment in broadband networks.

Our results show that 10% (6 out of 64) of the ISPs we tested have some form of CGN deployment. In particular, one ISP has a large-scale deployment where Revelio detected upstream CGN deployment from all 76 vantage points in that ISP. In the other 5 ISPs we observed evidence of a localized deployment limited to a subset of customers. We verified our results with representatives of the ISPs to validate our positive and negative inferences at the IP level. We confirmed the results for 4 of the 6 positive ISPs by personal communications with ISP representatives. The combination of the FCC-MBA and RIPE Atlas study represents (to the best of our knowledge) the largest active measurement study to date with confirmed CGN deployments in broadband networks at the IP-level granularity.

For a more in-depth analysis of our measurements please visit the openly available technical report.

Posted in Uncategorized | Leave a comment

What happened at IETF98 in Chicago, March 26-31?

Expecting an usual, not particular special IETF meeting, the 98th meeting two weeks ago in Chicago was actually quite exciting, in a positive sense. A lot of work, relevant to the architecture as well as measurement objectives of the MAMI project, made good progress and fostered interesting, still on-going discussions.

The meeting started off with the IP Performanec Metrics (IPPM) session Monday morning. A document on In-situ Operations, Administration, and Maintenance (IOAM) proposing a data model for telemetry and measurement data that can be applied to different (tunneling) protocols was discussed and received positive feedback from the group. This work is highly relevant to MAMI given measurement is one of the basic use cases for the Middlebox Cooperation protocol (MCP).

In the afternoon, there were several talks on middlebox benefits, state management, and privacy in the tsv-area meeting. Dave Dolsen presented draft-dolson-plus-middlebox-benefits and Brian Trammel (remotely) presented draft-trammell-plus-statefulness. Both drafts created quite some interest as well as discussion – any information that is exposed by an endpoint must first demonstrate benefits when consumed by the network.

Tuesday morning the IRTF Measurement and Analysis for Protocols (maprg) had a long slot with a number of presentations on security and privacy relevant topics. maprg usually sends out a Call for Contributions a couple of weeks ahead of the meeting. This time the group had a large number of very interesting submissions and could finally accept only about half of them to fit in the 2.5h slot. The talks presented data on DSCP, ECN, IPv6, Let’sEncrypt, broadcasted hostnames, weak keys in HTTPS, and censorship detected by the OONI project.

In the last session on Tuesdays Tommy Pauly presented Post Sockets in the Transport Services (taps) working group which is common work with the MAMI project updated after an one day workshop held in Feburary in Zürich.

Thursday started with the QUIC meeting were Mirja Kühlewind presented two drafts on Applicability and Managability as well as a proposal by Brian Trammell to add a packet number echo to the public QUIC header for RTT measurements. The draft received positive feedback to address one of the charter milestones and adoption will be confirmed on the mailing list. The proposal to add a packet number echo was lively discussed without reaching consensus yet. There was broad support for this addition but there were also privacy concerns that needs further discussion; similar to all information that will be exposed for network use in the header. QUIC will hold an interim meeting on June 6-8 in Paris.

Later that day, the Multipath TCP working group discussed the use of MPTCP for bandwidth aggregation in network scenarios where one low bandwidth fixed line link (DSL) is bonded with a high speed mobile link (LTE). Several approaches to address this use case have been proposed but there was so far no real consensus to move any of them forward. This time the discussion was led by the chairs attempting to compare the different proposals on a high level allowing the group to move forward. While there were also quite some people who indicated that they think this work should not be taken up by the IETF at all, a larger group of people were interested in this work and there was a clear indication for one of the proposals which is a solution that only requires a small addition to the SYN payload on the bonding link. The MAMI project is also focusing on this use case, using the MCP to signal preferences of the endhost to the bonding boxes in the network.

Thursday afternoon finally also the new MAMI T-shirts arrived (see photo above). Unfortunatly, they came only in to action during the bits-and-bytes and Friday morning that is traditionally rather quite. However, the next event is comig and the next IETF is already on the horizon in July in Prague! See you there! Good bye, Chicago!

Posted in Uncategorized | Leave a comment

Frist Prize at IETF98 Hackathon for “connection identifier” in DTLS group!

On Saturday and Sunday at the IETF there always is a Hackathon. This time at IETF-98 members of the MAMI consoritum were working on the implementation of the “connection identifier” in DTLS, a fairly recent proposal Hannes Tschofenig, Nikos Mavrogiannopoulos, and Thomas Fossati brought to the TLS working group.
The problem this proposal addresses is that an end-to-end DTLS session may silently die because an on-path Network Address Translation (NAT) middlebox dropped state after a (relatively) short period of quiescence. There is known trouble with UDP, as transport used by DTLS, that in-network state for this kind of traffic tends to vanish much more quickly than its TCP counterpart. As an example, the default timeout settings in the latest Linux kernel are 5 days for TCP and only 3 minutes for UDP — that is, three orders of magnitude!

© Stonehouse Photographic/Internet Society
IETF Hackathon, Chicago 25/03/2017

Obviously, there is a good reason for that: since UDP is connectionless, layer 4 devices have no way to possibly track a “connection” other than deep inspecting the flow, which is a pretty expensive activity. So it’s simpler for it to leave the onus of proving that a given UDP 4-tuple has an associated connection to the endpoints, by forcing them to regularly move bytes across. This state of affairs is clearly far from ideal for DTLS because, when a timeout happens in the NAT box, the victim endpoints need to re-negotiate a new crypto session context.

While this is generally annoying, it becomes even more nasty in cases where the client is a resource constrained, battery operated IoT device that woke up from its sleep cycle only to find its session doesn’t work anymore… The second nuisance is that the state that is dropped on the NAT box immediately becomes dead state in the server, consuming precious resources in vain. So, one popular workaround is to create synthetic “keep-alive” traffic, for example using the TLS heartbeat extension. This technique is a) not very robust (choosing the right keep-alive clocking depends on many external factors), and b) certainly, not affordable at smaller device scales, where waking up the “thing” to keep the NAT binding happy has the potential to quickly drain its battery.

© Stonehouse Photographic/Internet Society

Our solution – provide a connection identifier!  We propose to add a 32-bits blob that does one very simple thing: it decouples the DTLS session from the underlying 4-tuple, making it possible for the endpoints to dispatch incoming UDP traffic to the correct crypto session independently of any change in the underlying UDP address.

Sounds simple, right?  Yes, conceptually — apart from the birthday paradox hitting hard at large scales (see https://tools.ietf.org/html/draft-mavrogiannopoulos-tls-cid-00#section-4). What we discovered at the Hackathon is that backporting it to an existing stack (ARM’s mbedTLS) can be more difficult than expected if one needs to maintain API compatibility.

Another practical complication is signaling the wire format change to the receiving endpoint so that it can parse the incoming frame correctly.  This is easy when you can make breaking changes (for example, when transitioning from 1.2 to 1.3).  Not as much if you have to maintain backwards compatibility with existing and deployed versions of the protocol. We have been discussing a couple of possible solutions around this issue — namely: using the Version field, or moving to an extensible record layer format.  We are still undecided on what “the right” approach would be.

Note that there are interesting privacy implications related to using a visible and potentially long-term identifier due to the obvious linkability properties of such a construct. Even if not all the use cases are problematic in this respect, some of them are, and thus we designed (but not yet implemented) a privacy friendly connection identifier based on HMAC-based One-time Password (HOTP) which can be rotated at client’s will at any point in time.

Enough with the babbling! If you read up to this point you will be glad to know the hackathon judges rewarded our herculean effort with the first prize. Yay!

Posted in Uncategorized | Leave a comment

Post Sockets Workshop

MAMI partner ETH Zürich hosted a small workshop last month to discuss a topic central to our vision: how to move network programming beyond the 1980s-era BSD sockets API.  The workshop had two main goals. First, to sync up and share concepts about abstract programming interfaces among researchers and developers working on future networking stacks. Several partners from our H2020 sister project NEAT, which has a more explicit focus on how applications see the network and transport layer, were in attendance.

Our second goal with the workshop was to further develop the concepts in the Post Sockets abstract programming interface. Post Sockets is explicitly asynchronous and message oriented, which is a better match to most network applications and all future transport protocols than the single-streamed SOCK_STREAM service offered by TCP. It breaks the hard relationship between a connection (called a “carrier” in Post, since it carries messages) and the path(s) carrying it, and separates short-term transport layer state (“transients” which hold transport-protocol specific information such as current sequence numbers and flow control windows) from longer-term state (“associations” which hold state such as cryptographic resumption parameters), allowing generalized fast resumption of sessions for commonly-communicating endpoint pairs. Like the NEAT API, it is intended to be transport-protocol independent, and interoperating with NEAT’s policy engine and transport protocol selection machinery is a goal of MAMI’s Post Sockets work.

Our current work on Post Sockets in MAMI focuses on implementation details: how message carriers, associations, and transients bind to underlying transport protocol implementations, specifically in situations with multiple candidate transport and network layer protocols, and where network address translation makes rendezvous non-trivial. Tommy Pauly from Apple will present Post Sockets at the IETF taps session next week in Chicago (checkout remote participation). Also watch this space for future announcements.

We thank the workshop attendees for a very productive conversation, and look forward to working together in the future to drag network APIs into the twenty-first century!

Posted in Uncategorized | Leave a comment

Path Transparency Observatory (PTO) goes live!

One of the core objectives of the MAMI project is to perform measurements and collect measurement data that can help to quantify middlebox impairments on the Internet. While various impairments on different layers of the protocol stack have been detected over the years, the provided measurement data are often not sufficient to assess how much of a problem there actually is. This information is needed to guide development of protocols extension or new protocols such as the Middlebox Cooperation Protocol (MCP) as proposed by MAMI.

Over the last year the MAMI project developed and implemented the Path Transparency Observatory (PTO), an open-source and publicly accessible repository for measurements of path transparency. An Internet path between a vantage point and a target is consider transparent if no impairment has been observed on that path and therefore the packet was successfully received without modification. The goal of the PTO is to collect data from different sources usually in different formats and represent them in a comparable way. To achieve this we apply a pre-processing step that transform the raw data in observations of a network condition c on a path p during a time interval t.

In parallel we also developed PATHspider (see below), an active measurement tool for A/B testing which now also integrates upload facilities for the PTO. We used and are using PATHspider as well as other measurements tools such as tracebox to continuously run measurements to test TCP extensions such as ECN or TFO as well as impairments on protocols on other layers such as the use of the DiffServ codepoint. Currently, only observations for our ECN measurement campaigns of the last couple of month are available over the public PTO interface. However, we have collected more raw data and are currently running the needed processing steps to successively release more observation data to be publicly available in the PTO: https://observatory.mami-project.eu/

For more information check out the ANRW PTO paper and Brian Trammell’s talk at the CAIDA AIMS workshop today! And of course, we will announce on twitter when more data is available!

Posted in Uncategorized | Leave a comment

2 Years of PATHspider Development

The current incarnation of PATHspider‘s first Git commit occurred on the 22nd of February 2015, now just over 2 years ago.

Since then, PATHspider has gained its own target list resolvers and has generalised for performing far more than just the ECN-dependent connectivity failures it was originally created to measure. Through plugins it can now perform measurements relating to other TCP features such as TCP Fast Open, and IP features such as DiffServ.

Active development of PATHspider continues, with the roadmap for PATHspider 2.0 provisionally laid out in the GitHub issue tracker. This includes:

  • seamless MONROE integration,
  • support for workflows integrating with the MAMI Path Transparency Observatory,
  • …and of course, new measurement plugins.

The following video is a visualisation of the last two years of work, featuring commits from the PATHspider authors: myself, Brian Trammell, Elio Gubser, Mirja Kühlewind, Andreas Germann, Piet De Vaere and Ana Custura.

Posted in News | Leave a comment