1. About our transformation

This article provides an overview of our transformation of population and migration statistics to put administrative data at the core of what we do. This includes how our plans have evolved because of the coronavirus (COVID-19) pandemic and our approach for delivering new measures of migration based on administrative data from November 2020 onwards, following the suspension of the International Passenger Survey (IPS) in March 2020.

You can also find an accompanying summary of recent updates on our statistics and research. Your feedback on our latest update and plans would be valuable.

Back to table of contents

2. Our mission – why transform?

It is our mission to provide the best insights on population and migration using a range of new and existing data sources to meet the needs of our users. This is increasingly important in a rapidly changing policy context, where we know our users need better evidence to support decision-making at both national and local levels.

As set out in our previous work on the Administrative Data Census project, the current population system is heavily reliant on the decennial census. While this provides granular data at the lowest levels of geography every 10 years, it delivers less detail throughout the interim years. Additionally, the quality of our population estimates declines as we move further away from the census year.

We have also long acknowledged that the International Passenger Survey (IPS), which underpins our existing international migration estimates, has been stretched beyond its original purpose and that we need to consider all available sources to fully understand international migration.

Enabled by data-sharing powers in the Digital Economy Act 2017 and guided by our data security principles, we are therefore seizing the opportunity to make use of more data to give us a much richer understanding of how our population is changing.

Back to table of contents

3. Why these statistics are so important

Population and migration statistics underpin a wide variety of other statistics (such as unemployment rates), support a vast range of decisions and inform public debate. For example, the ability to forecast pensions, make decisions about local services (such as the number of school places or the provision of health services for an ageing population) and decisions about where to site new businesses.

In a rapidly changing policy and societal context, we also know that our users are highly interested in how migration patterns are changing and what this means for society and the economy. For example, the contribution and impact that migration has on public services – such as education and health care – and on the UK labour market. This includes both the national picture but also what is happening at a more detailed regional and local level.

It is therefore essential that our population and migration statistics are robust and timely and meet this broad range of user needs.

Back to table of contents

4. How are we transforming population and migration statistics?

Working in partnership across the Government Statistical Service (GSS), we are delivering new measures of population and migration based on administrative data sources.

Our progress so far

Using our data-sharing powers through the Digital Economy Act 2017, we have been progressing research into how we can link a range of government data sources to build an integrated system for measuring population and migration.

In January 2019, we published a research engagement report that updated users on our ambition for putting administrative data at the core of our statistics by 2020, dependent on acquiring access to the further data sources needed to fill gaps in coverage.

Since January 2019, we have published a series of updates on our progress towards this ambition. For population statistics, this includes our June 2019 update on developing admin-based population estimates (ABPEs), where our latest rules showed promise for removing patterns of over-coverage seen in previous research. For migration statistics, this includes our improvements based on a preliminary adjustment (for the period to June 2016) using administrative data from the Home Office and the Department for Work and Pensions (DWP) and our move to reclassify these statistics as Experimental Statistics in August 2019 to support ongoing development and innovation.

In November 2019, we updated users on our plan for 2020 onwards, based on our latest progress on data access and the work we needed to do to fill important gaps in coverage for groups such as EU migrants. This set out that our focus for 2020 would be on best addressing user needs in the short term by delivering further improvements to our migration adjustment methodology, in collaboration with the Home Office, the DWP and the wider Government Statistical Service (GSS). We aimed to deliver a completed adjustment methodology by summer 2020, then build on this - alongside the continuing ABPEs development - to deliver improvements to population statistics from 2021 onwards. This in turn would support the delivery of the best quality census outputs in 2022 and the delivery of a fully transformed population and migration statistics system based on integrated administrative data by 2023.

Our transformation journey - where are we now?

Our ambition continues to be putting administrative data at the core of both population and migration statistics, delivering the best quality census outputs in 2022 and a fully transformed system by 2023. However, we have revised our approach for transforming migration statistics this year because of the current situation around the coronavirus (COVID-19). The pandemic presents a significant challenge to the UK, and the Office for National Statistics (ONS) is working to ensure that the UK has the vital information needed to respond to the impact of this pandemic on our economy and society. The overall impact on the ONS, including on how our surveys operate, was outlined in a statement released on 19 March 2020, with further details published in another statement released on 27 March 2020.

What does this mean for ONS migration statistics?

The International Passenger Survey (IPS) - which underpins our existing UK international migration statistics - has been suspended because of the impact of COVID-19. The latest available IPS data cover the vast majority of the year ending March 2020, which we are due to publish in the August 2020 Migration Statistics Quarterly Report (MSQR). However, there will be no new IPS data available for inclusion in the November 2020 MSQR.

Prior to the suspension of the IPS, we had long acknowledged that the survey had been stretched beyond its original purpose and that we needed to use all available data to fully understand migration. Now is the right time to focus more of our efforts on harnessing the power of administrative data for public good, building on the good progress already made through our population and migration statistics transformation programme. We are therefore now planning to move away from the IPS and use administrative data to deliver new measures of migration from 2020 onwards. The August 2020 MSQR will be the last set of migration statistics based on IPS data.

We know that there are challenges we need to overcome to deliver new measures of migration based solely on administrative data, including bringing in the further data we need to replace the IPS and fill gaps in coverage. Through our previous research, we have already identified a range of administrative data sources held across the GSS that can help us to better measure migration - including immigration, income, benefits and education data - and will build on this to design our new measures. This includes continuing our work with the DWP to explore how their Registration and Population Interaction Database (RAPID) could be used, alongside other sources, for the purpose of measuring migration (see Section 5 for more detail).

This means our regular statistical reports will look different from November onwards, as we adapt our methods and go through a period of transitioning to administrative data-based estimates. They will continue to be badged as Experimental Statistics throughout the transition, to support users in understanding the changes. We will also highlight where we are unable to provide certain statistics or breakdowns of data while we build our new migration measures, alongside providing guidance on the quality and coverage to ensure the statistics can be interpreted appropriately.

We are also looking ahead to the implementation of the new immigration system in 2021 and working with the Home Office and across the GSS to review how new administrative data could enhance our estimates of migration in the future. We are also collaborating closely with National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (NISRA) to ensure that we can produce comprehensive UK migration statistics. This includes incorporating the Northern Ireland migration estimates produced separately by NISRA and ensuring alignment with the migration statistics for local areas in Scotland and Northern Ireland, produced by NRS and NISRA respectively.

We will continue to regularly engage with users as we go through this transformation journey and gather feedback to inform our development work. This includes understanding immediate priorities for statistics and evidence on migration. Alongside our regular statistical outputs, we will also continue to explore the use of alternative data that can help provide insights on what has happened to migration since the COVID-19 pandemic began, such as the analysis of travel patterns included in our latest Migration Statistics Quarterly Report.

For ONS UK trade and travel and tourism statistics, today (21 May 2020) we have released a statement on the impact of the IPS suspension and how we are responding to this.

Our plan for population and migration statistics from 2020 onwards

We will continue to ensure that the work to transform UK migration statistics in 2020 aligns with our ongoing research to produce population statistics using administrative data. Migration statistics are an important component of estimating population change and so we will ensure our research is integrated as we move towards a fully transformed system in 2023.

A summary of the revised milestones on this transformation journey are set out in the following table.

This transformation journey - alongside the work to transform surveys across the ONS - also supports the ambition that "censuses after 2021 will be conducted using other sources of data" and importantly that the outputs we publish should be coherent and meet the needs of our users. This is essential to providing evidence to inform the recommendation to the UK government in 2023 about the future of population and housing censuses in England and Wales. The ONS is also working with the devolved administrations on their respective programmes of work, including the NRS and NISRA, which are responsible for the census and producing population statistics in Scotland and Northern Ireland respectively. Ongoing collaboration will take place between the ONS, NRS and NISRA to consider harmonisation and ensure population statistics across the UK remain comparable. Decisions on the census after 2021 in Scotland and Northern Ireland will be made by the Scottish Government ministers and the Northern Ireland Executive respectively.

We have taken the decision not to benchmark the administrative data census outputs with the outputs from Census 2021. Instead, we will ensure that we use the best available sources to produce the best possible outputs from the census - using our understanding of the differences between administrative data and the census and the strengths of all data sources. We will iteratively develop our transformed population and migration statistics system, taking on board feedback from users and making the best use of new data and new methods as they become available. We will rigorously quality assure new methods and share the impact of any changes made, demonstrating the benefits and improvements before implementing them.

Alongside supporting the decision on the future of the census, our transformation journey also plays an important role in supporting our progress in measuring the global Sustainable Development Goal (SDG) indicators in the UK. Migratory status is one of the standard disaggregations required by the UN to measure and report on the global SDGs, and our transformation work will help to improve the evidence base by providing more granular data in the future.

Back to table of contents

5. Update on analysis to explore how the DWP’s RAPID could be used to measure migration

As set out earlier, we are continuing to work with the Department for Work and Pensions (DWP) to explore how further analysis of their Registration and Population Interaction Database (RAPID) could support us in measuring migration. This includes reviewing whether we will make further adjustments to the International Passenger Survey (IPS) data we will report in the August Migration Statistics Quarterly Report (MSQR), alongside how we will use RAPID in our new measures of migration based on administrative data from 2020 onwards.


To produce the current preliminary adjustment for EU migrants, we commissioned the DWP to provide some aggregate analysis of their Lifetime Labour Market Database (L2) dataset. The L2 dataset is a 1% sample of HM Revenue and Customs' (HMRC's) National Insurance and Pay As You Earn (PAYE) system and is supplemented with benefit data from DWP. The L2 dataset allowed the DWP to analyse data on non-UK-national adults who registered for a National Insurance number (NINo) in the UK and had interactions with National Insurance, PAYE, and DWP benefit and local authority benefit systems.

This analysis meant patterns of interaction with the tax and benefits system could be identified, and it allowed the DWP to classify records of people who could, based on prolonged interaction with these systems, be classed as long-term migrants.

The DWP have since developed RAPID. One of the main benefits of RAPID, compared with L2, is that RAPID uses 100% extracts of data, meaning it contains records for everyone with a NINo. An additional benefit of RAPID is that it contains data extracts from a wider range of DWP and HMRC systems than L2, including all benefit, employments, self-employment, pensions and in-work benefit interactions, for example, tax credits and Housing Benefit.

RAPID provides a single coherent view of citizens' interactions with the DWP, HMRC's PAYE and self-assessment systems and local authorities' Housing Benefit systems within a tax year for the UK (between tax year ending 2011 and tax year ending 2019). To do this, RAPID collates 100% extracts of data and summarises the duration of individual interactions with these systems for each tax year.

As well as all people who hold a NINo, RAPID also includes information on migrants registering for a NINo along with their date of registration and self-reported date of arrival in the UK. Information on migrant arrivals and registrations, combined with the data on interactions, allows an assessment of whether the duration of the stay in the UK may classify them as long-term residents.

Our latest progress

The assessment of arrival and interactions data over time has, up to now, supported the development of a methodology for adjusting Long-Term International Migration (LTIM) estimates. In our previous update, we outlined our plan to use RAPID to develop our preliminary adjustments to EU immigration; however, because of the suspension of the IPS, we are extending our analysis to assess whether RAPID could be used to help measure migration for both EU and non-EU nationals. As well as using RAPID to measure arrivals, it may also enable the generation of a resident stock measure. To support this work, we have commissioned the DWP to undertake research to identify if estimates of long-term departures can be made based on the absence of system interactions for 12 months or more. It should be noted that because of the impact of the coronavirus (COVID-19), there may be a reduction in the resource the DWP can provide to support this work.

Our research is being conducted using de-identified aggregate data supplied to the Office for National Statistics (ONS) by the DWP. This uses both DWP and HMRC data. To further understand outputs from RAPID, we will be comparing the number of long-term arrivals and long-term departures in RAPID to estimates from the IPS. Where applicable, we will also look to compare against other available data sources such as the number of visas granted by the Home Office. We will also look to compare the demographic breakdown of both arrivals and departures of both RAPID and the IPS. Understanding both arrivals and departures will enable the ONS to assess the measurement of net migration using RAPID. As we develop our research to measure long-term migration to and from the UK using RAPID, we will provide further updates on our progress.

Back to table of contents

6. What our users have told us

Our users sit at the heart of our framework for transforming these statistics (Figure 1), which describes the important questions we need to answer to meet their needs.

To meet our users' needs, we need to provide coherent statistics on the size (or stock) of the population and how it changes over time (flows, both nationally and locally). We also need to tell a clear story about what is contributing to this change and show how different groups in the population impact on society and the economy, including on our workforce, communities and public services such as the NHS and schools. This needs to be recognised as the story that is being experienced by our users.

Our users have also told us that they want us to deliver these statistics frequently and in a timely manner to be able to make evidence-based decisions. Our statistics also need to be relevant in a rapidly changing society, and we need to be able to report on their quality.

Feedback gathered shows support for how we are reviewing current data sources and looking to maximise the value of administrative data in the future, while challenging us to provide more evidence to ensure we instil confidence in these statistics.

To make sure that our transformation journey continues to be as open and transparent as possible, we will:

  • regularly publish research and methods as we develop them
  • continue to present analysis showing the coherence between different sources of information
  • engage with our users and stakeholders, seeking regular feedback
  • use our research findings, the best available data and methods, and the feedback from users to make decisions about which improvements to make each year
  • implement changes to our statistics when and where appropriate, clearly communicating these changes to users in advance of making them

How to get in touch

We welcome your feedback on this update and on our transformation journey. If you would like to get in touch, please contact us by email at pop.info@ons.gov.uk.

We have recently presented our research at the 2019 Royal Statistical Society Conference, the British Society for Population Studies Conference, the Migration Statistics User Forum, the International Conference on Administrative Data Research and the 2020 International Forum on Migration Statistics in Cairo.

We are also working with local authorities, experts from academia and other government departments through invited engagement groups to help shape and quality assure our work. Additionally, we are engaging with other National Statistical Institutes (NSIs) across the UK and internationally to share experiences and seek feedback on our plans.

Back to table of contents

7. Social statistics transformation

In addition to the work outlined in this article on transforming the population and migration statistics system, a similar programme of work is underway to transform the way we produce social statistics to better meet the needs of our users and to produce the best statistics from all the available data. 

Through our research and collaboration with other government departments, we have been able to produce a range of research outputs on a variety of topics, including:

Transforming the population, migration and social statistics system could also bring additional benefits, and we are looking at combining administrative, survey and big data to produce new outputs that are currently not provided by a 10-yearly census.

Back to table of contents

8. More about our transformation journey

For a comprehensive list of descriptions for the names and terms used in our outputs, see our glossary.

Research Outputs provide more information on using administrative data to produce population statistics.

Previous research using administrative data to produce estimates on the size of the population (previous administrative data-based population estimates (ABPEs)) is available.

August 2017

Our August 2017 report highlights our progress towards developing a better understanding of student migration to and from the UK, alongside the Home Office Exit Checks report also published in August 2017.

May 2018

In May 2018, we published an update on the migration statistics transformation programme. Our report on international migration data sources sets out how we are working with the Home Office in using their administrative data to further our understanding of international migration.

January 2019

Our January 2019 report, Update on our population and migration statistics transformation journey: a research engagement report, updates users on our progress and seeks feedback on our future plans.

May 2019

In May 2019, we published our research into international migration and the education sector, exploring what current data sources tell us about school places, pupil attainment and the number of teaching staff in schools.

June 2019

In June 2019, we published our analysis using activity-based rules and records from single and linked data sources to develop our approach for producing ABPEs. Initial rules have been combined to produce the first ABPEs using this approach. Alongside this, we also published a short note outlining our discussions on the measurement of illegal migration.

August 2019

On 15 August 2019, we published our analysis plans on the contribution and impact of international migration on the health sector. Alongside this, we published a report on international migration and the healthcare workforce and our analysis of migrant labour force within the tourism industry. On 21 August 2019, we published our latest research into understanding different migration data sources, setting out how we can draw on combined strengths of survey and administrative data sources to provide our best assessment yet of international migration. This research was then reflected in the August Migration Statistics Quarterly Report for the first time.

February 2020

On 14 February 2020, we published an overview summarising how we are defining and measuring international migration, including developments of our current methods. Alongside this, we published our latest research investigating international student migrants' travel patterns using linked administrative data and alternative definitions for international migration using Home Office Exit Checks data. We also published a report on applying a statistical quality framework to longitudinally linked administrative data, aiming to identify potential errors within datasets.

Back to table of contents

9. More about our official population and migration statistics

For national and subnational mid-year population estimates for the UK and its constituent countries, see the Population estimates for the UK, England and Wales, Scotland, and Northern Ireland statistical bulletins.

All information and publications on international migration produced by the Office for National Statistics (ONS) Centre for International Migration are available on ourInternational migration page. You can find out more about the ONS Centres and their priorities on our landing page.

All information and publications on the size of the population produced by the ONS are available on our Population estimates page.

You can find more information on the main concepts and definitions used for migration, country of birth and citizenship across the Government Statistical Service (GSS) in the relevant GSS harmonisation guidance.

Back to table of contents

Contact details for this Article

Becca Briggs
Telephone: +44 (0)1329 444661

You might also be interested in: