JT: Election night everyone felt the gut punch of the exit polls – where do you think pollsters got it wrong and why did it feel like no one saw it coming?

PK: All eleven polls that came out in the final 24-48 hours of the campaign all had essentially the same error. We all had the Labour-Conservative figure as level-pegging give or take one point, when actually the Conservatives won by 6 percentage points. So all the polls got in one sense the most important thing wrong, but we all had UKIP in third place, the Lib Dems fourth, the Greens and SNP around 4 or 5 per cent. The poll in Scotland did foreshadow the SNP almost clean sweep, and at YouGov we correctly had Labour achieving a bigger swing in London than elsewhere.

The pollsters collectively have commissioned an independent investigation into what happened, but my initial guess is that at the heart of the problem is a small difference between what I call attitude and action. Answering an opinion poll is expressing a point of view, casting a vote is making a decision. And the great majority of time, whether with politics or consumer behaviour or anything else, there’s a close match between attitude and action. But it looks to me as if on this occasion there was a divergence in three people out of every 100 between the attitude they honestly expressed and their action on the day.  

JT: Is this a one-off or do you foresee future problems?

PK: YouGov’s track record is a very good one – we were only one point adrift on the eve of the Scottish Referendum; we were closer than anyone else in last year’s European Elections and we got UKIP spot on; we were within one point of measuring both of Boris’s victories in the London Mayoral elections; in 2010 we got the Tory-Labour balance absolutely spot on; and in 2005 we got every party within one point. So something went wrong this time. And even if we find out for sure what went wrong this time, we then have to make a judgement as to whether the problem is specific to this election, or whether it discloses something fundamental which is has been lurking around but somehow didn’t stop us getting previous elections right.

At this election as well as doing our Daily polls for the Sun and Times, we were also doing separate research for the British Election Study, polling 800-900 people everyday, which adopted the “warm-start approach” – asking a number of questions about issues, leaders and so on before then asking voting intention – as opposed to a “cold-start approach”, which asks voting intention first. We found that on average our data for the Conservatives in the BES poll were 1.8 percentage points higher, and in those last 2 days our figures were 37.5 Conservative 31.5 Labour – almost exactly spot on. But the problem is if we’d all done warm-start surveys and got the right story this time, it doesn’t necessarily mean the “warm-start approach” will work in every election.

JT: So why did warm-starts capture the public’s voting intention more correctly this time as cold-starts had in the past?

PK: I think the problem tends to arise when you’ve got a major party about which many people have conflicting views – if you’ve got heart versus head decision. In an opinion poll people let their heart come through because they’re expressing a cost-free opinion. When they go into a polling station their head says, you’re making a real decision. If something like that’s happening then it says to me a lot of the time it really doesn’t matter whether you ‘cold start’ or ‘warm start’.

I got quite unpopular in certain sections of the Labour Party, not least the Leaders office, for saying month in and month out the fundamentals are for Tories and the fundamentals are against Labour. No party in Opposition has come to government having been behind badly on both the Leader and on the economy. I did predict, consistently and publically, that the Conservatives would gain support towards the end, the problem is our numbers didn’t move. So I looked the figures and my conclusion was the Tory brand toxicity and Labour’s lead on the health service were neutralising leadership and the economy. In retrospect they didn’t neutralise it.

JT: A lot of people will say you can’t really trust polls any more, do you think that political parties should rely so heavily on polls?

PK: Opinion polls perhaps became victims of their own record of reliability, in that everybody exaggerates their level of accuracy. To adapt something Churchill said about democracy, that opinion polls are the worst way to find out what the public thinks except for all the others.

The real function of an opinion poll is to find out what the public thinks. It is part of the democratic conversation between voters and politicians. It’s flawed, it’s imperfect, but it’s the best way we’ve got; and on the whole it works pretty well. But in politics, in a first-past-the-post system, the difference between 34%-34% and 37%-31% is a completely different political outcome. This means that small statistical errors, that in other contexts are neither here nor there, loom large. But if you think that politicians should try to understand what voters think, then if polls are not part of your kit for finding out, then you’re in trouble.

JT: On the Labour Party, there’s a plethora of people weighing in on what they think went wrong – the Beckett commission, Cruddas review – what’s your analysis of the main things that went wrong for the party?

PK: Plainly, voters did not think Ed Miliband was up to the job of Prime Minister and they didn’t think Labour was credible on the economy. Labour never got any form of persuasive story across about the final three years in government between the close of Northern Rock in 2007 and the election in 2010.

I personally think there is something much more fundamental and its to do with the Labour Party’s collective inability to put the Blair era into the correct perspective.  You could divide the Blair era into Blair Mark 1(B1) and Blair Mark 2 (B2). B2 essentially started with Iraq through to his period after PM – there is no appetite within the party to hold B2 up for adulation. But B1 won two huge landslide victories and a third very substantial victory; did win over the business community; did win over the middle classes; and did a lot of radical things in office – minimum wage, SureStart, devolution, Human Rights Act, civil partnerships, NHS spending. The problem for the Labour movement is that because people recoil so completely from B2 they’ve written B1 out of the script and therefore haven’t learnt the right lessons from the B1 era. And my broad view is that until the Party goes back to an updated version of the B1 strategy it can’t win power.

JT: So looking forward, we have a leadership election on, what are your thoughts?

PK: Let me say three things: I think there needs to be a formal political blood letting. One of the features of the Ed Miliband Leadership, which in a way is to his credit, is the way the Party held together, unlike in 1951 and in 1979. The price is that it didn’t have the cathartic process it should have had. I hope the leadership election will be a clarifying contest, out of which will come a clear sense of what Labour is and what Labour isn’t. If it seeks to hold the whole of the Labour coalition together, then I don’t think we can win the next election. I think the party needs to say that form of 20^th century left politics – that rejected what Blair stood for in his prime and want us to go back to scrapping tuition fees, more public spending, higher taxation, more nationalisation, more business regulation – we reject. So if the new leader either embraces that or includes it in an enormous tent, I think we’ll lose.

The second point, after clarifying, is that I think the new leader has to re-sell B1 to the public. We want to learn, we want to develop in a way that is relevant to 2020, but the kind of approach to politics that we had under Blair from the time he became leader in 1994 to some time in the middle of the second term changed everything.

And third, is that I think the new leader must have an explicit and ambitious theory of wealth creation in the private sector. We should be grown up enough in the party not to tolerate capitalism because its too difficult to get rid of it, but to say actually the market mechanism is by far the best way to deliver the great majority of good and services to the public and often quite a lot of the goods and services into the public sector – into hospitals, into schools where the public sector is the final point of delivery.

JT: What issues are important to the electorate and the moment, and will these impact the labour leadership and how labour positions itself in the near future?

PK: The people whose votes are not locked up tend to follow politics less closely, and hold less tribal views and more valence views. So you win an election by appealing to valence voters, rather than positional voters. And so the practical task of any party at an election time is to win the valence war.  Tony Blair was a brilliant valence politician in his prime, Margaret Thatcher was a brilliant valence politician in the 80s, both of whom acquired a reputation in a key art of the electorate for being up to the job, on top of the issues, knowing their mind, and staying strong. So I think one should look at the valence qualities rather than conventional issues when trying to figure out what a party needs to do.

JT: Coming out of the election, a lot people felt dejected about the future of the party. What would your message to young people be about remaining engaged in politics?

PK: We’ve got to learn something from the Scottish Referendum, which absolutely engaged voters of all ages. Broadly speaking, you’ve got to develop a politics where people care about it, and they see its relevance to their own future.

In a more personal sense, I’m the generation born just after the Second World War. My father was a Jew brought up in Vienna who had to leave when the Nazis came in, and ended up fighting for Britain. My mother was Scottish, and my grandmother had been a suffragette. So on both sides of my family the battle to secure democracy was absolutely the core of their being.

The difficulty for you and many of your readers is that you’re two generations further away from it than I am. So it’s harder, all the things for which my father and my grandmother fought, we take for granted. The joy of your generation is that these battles on the whole have been won, not all of them, but the tragedy is that by not living through them you don’t get the inspiration that comes from seeing Nelson Mandela being installed as president of South Africa, the coming down of the Berlin Wall, or reporting on the first free elections in Portugal as I did back in 1975. So I wonder if we can get back to that primary passion for the democratic process. Democracy should never be taken for granted. It doesn’t take a huge amount in different circumstances for it to come crashing to an end.

JT: You do amazing work with organisations like UpRising. What got you into that and how do you see your role as inspiring and encouraging young people in London and the UK?

PK: I’m aware of the luck I’ve had in life. And one of the things I increasingly felt is that this country is held back because far too many highly talented people lack the advantages I had. And UpRising is one part of the fight back, because we mentor talented youngsters from families without the advantages I enjoyed.

One of the other things I’m involved in here at YouGov is a very big project to look at why so few women reach the top in British society.  I do this because I want to live in, and I want my children and grandchildren to grow up in a fair and prosperous society. And at the moment, when you look at how few women get to the top, it is neither as fair as it ought to be nor as prosperous as it could be.

JT: What role do you think young people can play in building a party of the future?

PK: I’m on the whole not a believer in over segmenting electorates. Young groups tend to get patronised by people of my generation, and we pretend to listen, but don’t really. People of my generation are now in senior positions. We try to address the new issues that come up on gender, on race, on climate change, whatever, but the reality is that the challenges and politics of the 2020s and 2030s are not the same as the 1970s and 1980s, when my outlook was largely formed. And what people like you bring to the party is a fresh view, unencumbered by the barnacles of 20^th century thinking. So somehow you’ve got to be the fresh eyes, fresh ideas and fresh voice but without ghettoising it. Good luck!

Jessica Toale is a member of the Young Fabians Executive Committee. She and Alvin Carpio are leading the Young Fabians Future of the Labour Party. Contribute your ideas here: http://www.youngfabians.org.uk/future_of_the_labour_party_1