The redevelopment of private rental prices statistics, intended methodology: December 2023

• We are undertaking an improvement to, and a transformation of, our private rental price statistics; these changes include making better use of existing data sources, improving methods, and developing systems that will reflect changes in the rental market more clearly.
• These new statistics provide a greater level of granularity in rental price statistics, including local authority estimates, and estimates by property type and number of bedrooms.
• The new methodology uses a hedonic regression model, similar to the approach used to calculate the UK House Price Index.

The new Price Index of Private Rents (PIPR) is part of a continuous programme of improvement for consumer price statistics. This article provides an overview of the methods that we have used for our new rental prices estimates. An overview of the project aims and timelines can be found in our Private rental prices development plan article.

Work began at the end of 2019, when we first received access to the data that were necessary to begin redeveloping the private rental prices' statistics.

We researched existing methodologies used for rental prices statistics and began developing our future methodology. During this process we engaged with stakeholders (such as the Advisory Panels for Consumer Prices) and international experts in property price statistics, including members from the Economic Statistics Centre of Excellence (ESCoE).

We have also worked closely with Office for National Statistics (ONS) methodology experts to quality assure multiple aspects of the methodology. This engagement allowed us to decide on our methodology towards the end of 2021.

We have now completed the build of a high-quality reproducible analytical pipeline for our monthly production of private rental prices statistics, which we expect to begin publishing in March 2024.

Currently, the Office for National Statistics (ONS) publishes two private rental prices statistical outputs:

• Index of Private Housing Rental Prices (IPHRP)
• Private rental market summary statistics in England (PRMS)

These outputs use the same source data that will be used for the Price Index of Private Rents (PIPR).

Index of Private Housing Rental Prices

IPHRP measures the change in price of renting residential property from private landlords. A rental price index and its annual percentage change for the UK, its countries and English regions is published.

IPHRP's method is based on a matched-pairs approach, which splits the collected rental transaction data into a sample and a substitution pool. More detail on the methodology can be found in our Index of Private Housing Rental Prices Quality and Methodology Information (QMI). The methodology used to measure the IPHRP was the best available when IPHRP was developed because of the unavailability of microdata to apply more sophisticated methods.

Existing private rental prices' statistics are used to inform the owner occupiers' housing (OOH) costs element of the Consumer Prices Index including OOH (CPIH). This is our lead measure of consumer prices inflation, along with the "actual rentals for housing" aspect of Consumer Prices Index (CPI) and CPIH, and "rent" in the Retail Prices Index (RPI). It is anticipated that the outputs from this development work will eventually be used in the consumer prices statistics. More details about the proposed changes to consumer price inflation statistics can be found in our Correspondence on proposed changes to the Retail Prices Index (RPI).

Private rental market summary statistics in England

PRMS publishes point-in-time arithmetic mean and median rental price estimates for England, English regions and English local authorities.

The current methodology uses all rental transaction data, but limitations prevent compositional changes from being considered. Therefore it is not appropriate to compare PRMS estimates year-on-year to infer trends in the rental market and a price index cannot currently be produced. For more information on the methodology used, please see Section 7: Measuring the data of our Private rental market summary statistics in England bulletin.

The Price Index of Private Rents (PIPR) will combine our two private rental price statistical outputs. PIPR will provide a single monthly publication measuring both the change in price of renting residential property from private landlords and rental price estimates.

Methodologically, PIPR replaces the matched-pairs approach of the Index of Private Housing Rental Prices (IPHRP) with a hedonic regression model. This allows us to make better use of our data and allows PIPR to provide more granular data by producing outputs at a local authority level and broken down by property type and number of bedrooms.

The Price Index of Private Rents (PIPR) measures the change in price of renting residential property from private landlords. We produce a measure that reflects both the newly "agreed" rents and existing rents. Therefore, we aim to reflect the stock of rents and not the "flow" of new rents.

The current Index of Private Housing Rental Prices (IPHRP) is also measured using both newly agreed rents and existing rents. Other measures of private rental prices, such as those published by Homelet, Rightmove and Zoopla, are produced using only newly advertised rentals. For more information about measures of private rental growth, please see our Private rental growth measures, a UK comparison article.

The Johnson Review (2015) pointed to research that suggests a flow measure may be worth considering; that is, only new lets. We investigated the feasibility of measuring the flow of rents, and we concluded that we do not currently have data sources available to us to disaggregate new rents from existing rents.

The new measures of rental prices bring together several administrative data sources. The methodology will use a hedonic regression model, which will allow for mix-adjustment of the monthly price data to control the effect of the changing composition of collected rental properties. This is similar to the approach used to calculate the UK House Price Index. However, the exact detail of each stage in the methodology will be tailored to suit the rental data.

These stages are:

1. Quality assurance checks are completed on the data.

2. Data are cleaned and property records are linked.

3. On an annual basis, a "fixed basket" of properties is created, and missing property characteristics are imputed.

4. On a monthly basis, a monthly sample is created, and missing property characteristics are imputed.

5. The monthly sample is fitted to a hedonic regression model to quantify the relationship between property characteristics and associated rental price for each calendar month.

6. Using the coefficients from the model's output, imputed prices for properties within the fixed basket are calculated.

7. Elementary aggregates are produced at a local authority level by taking the ratio of the geometric means of the predicted prices in the base month and the current month (when calculating the statistics used for the Retail Prices Index (RPI) we use an arithmetic mean instead of a geometric mean).

8. Elementary aggregates are weighted together (Lowe index) and then chain-linked annually to produce a rental price index series over time.

The corresponding average rental price series is derived by applying the index to a base set of rental prices from the reference period (currently January 2023).

For example, if the average rental price in the reference period was £500, and the index in the current month was 110.0, a 10% growth would be applied to the reference period average rental price. Therefore, the average rental price in the current period would be estimated at £550. This ensures the price series is consistent with the published index.

Data

The data sources used fall into two distinct categories: price data and property attributes data. Combining the detailed property attributes data with the price data provides a comprehensive dataset required for use in a hedonic regression model. The Valuation Office Agency (VOA), Scottish Government, Welsh Government and Northern Ireland Housing Executive (NIHE) deploy rental officers to collect information on the prices paid for privately rented properties, along with some characteristics of the properties. Data for Northern Ireland also include data provided by propertynews.com.

Scotland rents data (underlying the IPHRP and PIPR stock measures) are mainly for advertised new lets, while England and Wales include data for achieved rents for both new and existing tenancies.

Annually over 450,000 private rental prices are collected in England, 30,000 in Wales, 25,000 in Scotland and 15,000 in Northern Ireland, which make these sources data rich. Further information on the data sources can be found in Section 6: Methods used to produce the IPHRP data of our Index of Private Housing Rental Prices quality and methodology information methodology.

To strengthen our methods, we are now able to link these rental prices data to other property attributes data (for example, age of the property and floor area), such as from Council Tax data. Properties are linked using a Unique Property Reference Number (UPRN). Separately, we can also link to a geo-demographic segmentation, which will help control for differences in smaller areas.

As with the IPHRP, when a rental price is collected, it will be assumed to be valid either for 14 months from its entry date into the system, or until an update is received. A 14-month validity period will be used as it balances typical contract lengths (which tend to be either 6, 12, 18 or 24 months) against operational practices. Once a property’s price is over 14 months old, and no recollection of the property has been made, the property will be dropped out of the sample. If a recollection of a property is made, the updated information will be used, and the entry date counter reset.

Expenditure weights are updated annually to ensure the estimates are representative of the UK. To calculate expenditure weights, dwelling stock data are multiplied with average rental prices. Dwelling stock data come from the Office for National Statistics, Scottish Government, and Welsh Government.

Dwelling stock estimates are split by the proportion of property types rented privately in Wales, Scotland and the nine regions of England using data from the English Housing Survey, Scottish Housing Conditions Survey and Census (Wales). Dwelling stock estimates are also split by the furnished status of the properties using the split at a national level as identified in the Living Costs and Food Survey. The existing weights from the Index of Private Housing Rental prices have been used for Northern Ireland.

To calculate timely expenditure weights, the most recently available data are used. For a given year, y, the dwelling stock data are based on the period y-3, while average prices are based on the period y-1.

Imputing missing property characteristics

When running the hedonic regression model, some properties may be missing one or more of their price-determining characteristics. For instance, floor area may not be available. These properties are still used in the regression, but the missing property characteristics are imputed. For the monthly sample, a decision tree regressor is used when the affected characteristic is a continuous variable, and set to a missing category when it is categorical.

For the fixed basket, a decision tree regressor is used to impute the price-determining missing property characteristics when the affected characteristic is a continuous variable, and a decision tree classifier when it is categorical.

A type of supervised machine learning method known as decision trees are used for classification and regression problems. They are used to make predictions about unknown quantities based on other information that is available in the data and are used here to impute missing values in price-determining property characteristics using other property information from the monthly sample and fixed basket, respectively.

Valuing a rental property

Each month we receive a new delivery of rental prices data collected by Rent Officers over the month. These will cover properties that were already in the sample, as well as new properties. Therefore, each month, the sample is updated as follows:

1. If the property was not in last month’s sample, that property record is added to the current month sample and the record age count set to one.

2. If the property was in last month’s sample, then the price is updated, and the record age count reset to one.

3. For remaining properties in last month’s sample for which there has not been a price update in the current month, the existing price is carried over to the current month’s sample and the record age count increased by one.

4. Any properties that have not received a price update for over 14 months are removed from the current month sample.

The price of an existing property in the sample is only updated when a match is identified in the newly collected data and the price change is within the acceptable tolerance level.

A regression model is used to estimate the value of each characteristic from the set of properties in the monthly sample. For example, the model might estimate the effect that every additional room and each different location have on the rental price in a certain month.

Then, the rental price of a property in the fixed basket (explained in the following section) is estimated by applying the coefficients estimated from the model to each of its features. This method allows us to estimate the prices of properties with every combination of features (such as number of rooms and local authority), even if that combination was not collected in the period. This creates predicted prices for each property, for each month of the year.

The price-determining characteristics that we use in the regression are:

• number of bedrooms
• floor area (used only for properties in England and Wales)
• property type (detached, semi-detached, terraced, and flat or maisonette)
• furnished status
• geo-demographic segmentation (ACORN)
• local authority district in England and Wales, and Broad Rental Market Area in Scotland
• property age bracket

For Scotland the floor area has been excluded from the regression model because the address information available in the rental data is not suitable for linking.

For our regression model we use a semi-log ordinary least squares (OLS) model, with the mathematical formulation:

where:

• p_i is the rental price of property i

• K is a constant

• βⱼ is the coefficient associated with characteristic j

• x_jⁱ indicates whether property i has the characteristic j (such as detached property); if so, it takes the value 1, otherwise it takes the value 0 (except for floor area where it takes the floor area in square metres)

• e_i is the statistical error term

The logarithm of the rental price paid is used because rental prices tend to be log-normally distributed, meaning the frequency distribution of the log of the rental price is bell-shaped.

Fixed basket

The PIPR methodology is mix-adjusted to control for different types of rental properties being collected in different months. The process of mix-adjustment requires that, in each January, a fixed basket of properties is updated to reflect changes in the composition of rental properties. This basket is then used to produce imputed rental prices for the current year, before the basket is then updated again in the subsequent January.

The fixed basket contains all the collected rental properties in the previous year. If any rental property had been collected more than once in the year, the most recently collected data would be used.

The Ordinary Least Squares model creates coefficients’ estimates, which are used to calculate an imputed rental price for each property within the fixed basket. This imputed rental price reflects what each property in the fixed basket would have been rented for in the current month.

Calculating an index

The Ordinary Least Squares model creates coefficients, which are used to calculate an imputed rental price for each property within the fixed basket. Strata-level (LA by property type by furnished status) average prices are calculated by taking a geometric mean of the imputed rental prices from the fixed basket, which involves multiplying the “n” imputed prices together, and then taking the n^th root. To ensure representativeness when aggregating the data, expenditure weights are used, as described in this section under the Data subheading.

The lower number of collected properties within local authorities can lead to volatility in the series. To attempt to account for this volatility, three-month moving averages are applied to the local authority series. However, some volatility can remain, therefore, local authorities should be analysed in the context of their longer-term trends rather than focusing on monthly movements.

A fuller description of this method and other alternative methods for calculating residential property prices can be found in the Handbook on Residential Property Price Index.

Output

Our measures will be available for the UK, its constituent countries, English regions, and local authorities or broad rental market areas. The new publication will contain:

• an index of private rental prices
• annual rates of change
• average private rental prices
• a breakdown of private rental prices by geography, property type and bedroom category (one-bedroom, two-bedrooms, three-bedrooms and four or more-bedrooms)

Comparisons between data covering different countries

Scotland rents data (underlying both the Index of Private Housing Rental Prices (IPHRP) and Price Index of Private Rents (PIPR) stock measures) are mainly for advertised new lets. Data for England and Wales include achieved rents for both new and existing tenancies.

The Cost of Living (Tenant Protection) Scotland Act capped in-tenancy rental price increases at 0% (and up to 3% in certain circumstances) until 31 March 2023. On 1 April 2023, this rental price increase cap was increased to 3% (and up to 6% in certain circumstances), as reported on the Scottish Government website.

On 20 September 2023, Scottish Government confirmed that this rental price cap would remain in place for up to six months from October 2023. This rental price cap only applies to in-tenancy rent increases, with no restriction on rent increases for new lets.

As Scotland data are predominantly for new lets, only a small proportion of the Scotland data collected are based on existing lets data. This means that collected data will be predominantly based on newly advertised rents, which are not subject to the price cap. Therefore, price changes for existing tenancies are largely estimated for Scotland. This is likely to lead to over-estimation in stock prices and indices for Scotland since late 2022.

Estimates at a local authority level

The newly developed methodology now allows comparisons of rental price changes to be made at more granular levels. However, while our average prices and growth rates at higher geographies are robust, low collection rates in some local authorities can lead to volatility at these levels. While efforts are made to account for this volatility, the change in the price in these local levels can be influenced by the type and number of properties collected in any given period. Lower-level geographic breakdowns should be considered in the context of their longer-term trends rather than focusing on monthly movements.

Estimates for the City of London and the Isles of Scilly are not published because of low collection volumes.

Information on how we researched this methodology can be found in our previous Redevelopment of private rental prices statistics intended methodology article, published on 25 March 2022.

We plan to engage with users to inform our decision to publish these new statistics in March 2024. We are hosting engagement sessions, including a webinar on 1 December 2023, and another in January 2024. You can sign up to these events on Eventbrite.

Northern Ireland

Because of the differences in Northern Ireland data, which have resulted in additional complexities during development, we have made the difficult decision to delay the inclusion of Northern Ireland rental price data in Price Index of Private Rents (PIPR) until 2025. We intend to publish Northern Ireland indices using the current Index of Private Housing Rental Prices (IPHRP) methodology for Northern Ireland until then, including the continued use of Kantar data for consumer prices statistics.

Historic data

This impact analysis provides data from January 2015. We plan to extend the time series back further, but the outcome of this depends on sample sizes and what that means for data quality.