How to Extract and Analyse London Planning Data from the Official Portal

Every week, London’s 33 local planning authorities receive hundreds of applications. The official online registers contain a wealth of structured data – application numbers, statuses, decision dates, development types – but the fields are scattered across different pages and often buried inside PDFs. For community groups, researchers, or developers conducting due diligence, manually copying entries is slow and error-prone. This guide explains how to retrieve, clean and interpret raw planning data from the three key sources: borough portals, the GLA’s strategic referrals database, and the London Datastore. You will learn where each source fits, how to extract data without coding, and what to watch for when merging datasets.

What Data Lives Where – and Why It Matters

Each borough runs its own planning register, usually powered by a system like Idox, Acolaid or Uniform. The search interface lets you filter by date, application type, status or ward. The key fields are the application number, description, status (pending, approved, refused), decision date, and applicant name. However, the export options vary. Some boroughs offer a CSV download; others only display a list on screen. The GLA publishes its stage 1 and stage 2 reports as PDFs on its strategic planning applications page (https://www.london.gov.uk/programmes-strategies/planning/planning-applications-and-decisions/strategic-planning-applications), plus a downloadable Excel spreadsheet of all referred applications. The London Datastore (https://data.london.gov.uk/dataset/planning-applications-boroughs) aggregates planning data from most boroughs in a single CSV, updated monthly, covering all applications from 2015 onward. The key difference: borough portals give you today’s decisions, the GLA gives you strategic referrals, and the Datastore gives you a standardised historical archive.

Borough Portal Data Extraction: Step by Step

For live or recent applications, the borough portal is the most timely source. Start by using the Advanced Search and setting a date range. Many portals allow you to change the results per page to 100 or 200. If no export button exists, you can copy the table rows and paste into a spreadsheet. Watch for truncated fields – descriptions often cut off after 200 characters. To get the full description, click each application link, or use the “View Summary” option. Some portals (e.g. Southwark, Hackney) have an API endpoint accessible via a URL pattern – check the page source for a JSON feed. For example, Southwark’s planning portal uses a RESTful endpoint that returns XML data for each application ID. If you are comfortable with browser developer tools, you can inspect the network requests while clicking through pages to find the raw data feed. A practical tip: bookmark the “Advanced Search” page with your preferred filters pre-filled (e.g., date range and ward) so you can re-run the same search each week without re-entering criteria.

Fields to Capture from Borough Portals

The table below shows the common fields you will see in most borough search results, along with what to watch for when copying data.

When you paste into a spreadsheet, the date column may be treated as text. Convert it to a date format using Excel’s DATEVALUE function or Google Sheets’ DATE function. For the application number, strip out slashes (e.g., 2024/1234 → 20241234) to make it easier to match across datasets.

GLA Strategic Referrals Data: Major Applications Only

Major applications (over 150 units, buildings over 30m, or affecting strategic views) are referred to the Mayor. The GLA publishes a list of all referred applications on its planning webpages, with a downloadable Excel file. The file includes the application number, borough, site address, description, stage 1 and stage 2 decision dates, and the Mayor’s direction. To get the full stage 1 report PDF, use the application number in the GLA’s search tool at https://planning.london.gov.uk. The report contains the officer’s assessment against the London Plan policies, which is invaluable for understanding how the GLA interprets density, design, and transport. One caveat: the GLA dataset is updated weekly, but some fields are only filled after a decision is made. You may need to cross-reference the borough portal for the latest status of a referred application, because the GLA may not reflect a borough committee decision until the Stage 2 report is published.

London Datastore Bulk Download: The Historical Archive

The London Datastore hosts a dataset called “Planning Applications – Boroughs” (https://data.london.gov.uk/dataset/planning-applications-boroughs) that combines records from all 33 LPAs. The CSV contains fields such as application number, site address, description, decision date, development type (residential, retail, mixed-use), and the number of units. Download the latest version and filter by date, borough, or status. The dataset uses a standardised schema, but not all boroughs report every field. For example, the “units” column is often empty for minor applications. The advantage is that you can run year-on-year comparisons across London without scraping each borough separately. A practical use: filter for “outline” applications in your borough over the past five years, and then check the number of reserved matters applications that followed. This reveals how many large sites are still in the pipeline. The dataset is typically updated two to three months behind the current date, so it is best for trend analysis, not for real-time monitoring.

Interpreting the Data: From Raw Fields to Actionable Insights

Once you have the raw data, the real work begins. Look at the description field to classify developments by type – residential, commercial, or mixed. Use the decision date and status to calculate determination times. A typical minor application takes 8 to 13 weeks; anything longer may indicate a contentious case. Cross-reference the applicant name with planning history to see if the same developer consistently submits applications in the same ward. For community groups, tracking the number of pending applications in a ward can alert you to upcoming committee meetings. The London Datastore also includes a field for “outline” vs “full” applications – outline applications require a later reserved matters application, which is a separate event to monitor. You can combine the planning data with the Index of Multiple Deprivation (available on the London Datastore) to see if new developments are concentrated in the most deprived wards. For example, a 2023 study by the London Assembly found that the number of affordable housing units in outline applications was often lower than the final approved numbers – a pattern you can check in your own area.

Common Pitfalls in Data Extraction – and How to Avoid Them

Missing or duplicate records: some boroughs update the portal only once a day, so a decision made in the morning may not appear until the next day. The London Datastore lags by two to three months. For real-time status, the borough portal is the only reliable source. GLA referral data is updated weekly. Another pitfall: application number formats vary. Some boroughs use a year-prefix (2024/1234), others use a running number (24/01234). When merging datasets, normalise the format by stripping slashes and padding with zeros. Also, the London Datastore sometimes contains duplicate entries when a borough resubmits a correction. You can identify duplicates by comparing the application number and decision date – if they match exactly, keep the row with the latest update date. Finally, be aware that some boroughs (e.g., City of London) do not submit all their applications to the London Datastore, so you may need to use their portal separately.

Practical Next Steps for Regular Monitoring

If you plan to monitor a set of wards or a specific developer, set up a recurring task. Export the borough portal data as CSV every Monday morning. Use a simple spreadsheet formula to flag new applications added since the last export. For the London Datastore, subscribe to the dataset page to receive email notifications when the file is updated. For GLA applications, bookmark the referrals page and check every Thursday. Over time, you will build a local database that reveals patterns – which developers are most active, which types of applications are refused most often, and how long decisions take in your borough.

For a deeper dive, combine the planning data with the London Property Data (Council Tax, rateable values) or the Index of Multiple Deprivation to see how development correlates with social need. The official sources are the most authoritative, but always verify the latest status on the borough portal before acting on any dataset. Start this week by downloading the London Datastore CSV and running a simple count of pending applications in your ward – you may be surprised by what you find.