Policy Papers

‘Troubled Families’: the lessons of history, 1880-2012

John Welshman |

  • RSS Feed Icon

Executive Summary

  • The Government is determined to tackle what it perceives to be 'troubled' families.
  • In December 2011, David Cameron announced that the Government would 'get to grips' with England's 120,000 'most troubled families', while in June 2012 Eric Pickles said that 152 local authorities in England would be offered financial incentives to confront difficult families in their areas.
  • This policy agenda represents the latest in at least eight major reconstructions of the underclass debate in the United Kingdom and the United States between 1880 and the present.
  • There have been both discontinuities and continuities over time and while there have been some periods when no underclass concept has been evident, these have been remarkably brief.
  • The way the concept has been defined in different periods has said as much about broader trends in society and the economy as about the underclass itself.
  • While concerns over the 'social residuum' disappeared during the First World War, the 'problem family' became a preoccupation during the Second World War, and in the 1980s, ideas about an 'underclass' were imported into the UK from the US.
  • While some of these early ideas had little apparent impact on policy-making, over time, debates about behaviour have become more central to public policy.
  • The concept has proved resilient, for a range of reasons, and its history offers important lessons for policymakers, notably that how such families are defined is more important than counting them; and that addressing both behavioural and structural causes of poverty is likely to be the most effective policy.
  • While Government talks about 'history repeating itself' in families and between generations, the history that has really been repeated here is that of a flawed discourse.

The chronology of the 'underclass' debate

The Coalition Government is determined to tackle what it perceives to be 'troubled' families. In December 2011, the Prime Minister announced the Government would 'get to grips' with England's 120,000 'most troubled families'. He argued that a relatively small number of families caused many of the problems in society, and were characterised by any five of seven criteria: worklessness, poor housing, no qualifications, mental health problems, longstanding illness or disability, low income, and being unable to afford food or clothing. Among the areas with the highest concentration of such families were Birmingham (4,500), Manchester (2,500), and Sandwell (1,115). Their problems were deemed to be intergenerational, and they cost the state £9 billion per year (£75,000 per family). According to the Prime Minister, this one social group was seen as living apart from the rest of society, and he promised more targeted support, with the Government pledging £448million to 'turn around' their lives by 2015.

Louise Casey, who formerly headed the previous government's Respect Task Force, was to head up a new Troubled Families Unit in the Department for Communities and Local Government, while family workers would identity these families and work with them to make sure their children got to school on time, and were properly fed. These efforts built on the Family Intervention Projects (FIPs) and Family Nurse Partnership (FNP) that were set up in 2007 by the previous administration. In June 2012, a scheme was announced whereby 152 local authorities in England were offered financial incentives to tackle the troubled families within their areas. Communities Secretary Eric Pickles said that 'these families are ruining their own lives, they're ruining their children's lives and they're ruining the lives of their neighbours'. However, social scientists such as Ruth Levitas, commentators and even some local authorities have questioned the evidence base for current government policy, the existence of such large numbers of troubled families, and their cost to the state.

Historical research offers important insights into government efforts to both define and tackle troubled families, for there is no doubt that there have been a series of similar labels. Ideas of the deserving and undeserving poor were evident in the early modern period, and are arguably timeless. But in the modern period, since c.1880, there have been at least eight major reconstructions. In the 1880s, social investigators such as Charles Booth became concerned about the emergence of a 'social residuum' in London. This was echoed by anxieties about the 'unemployable' in the writings of William Beveridge and the social reformers Beatrice and Sidney Webb in the early 1900s. While this language was less evident in the social surveys published in the first decade of the twentieth century, the 1920s and early 1930s were characterised by the search for a 'social problem group'. In the early post-war period, this metamorphosed into the 'problem family' notion of the 1950s, which cast a powerful spell over volunteers involved in the Family Service Units, public health doctors, and some social workers, and is replicated in the current requirement for local authorities to compile lists of troubled families.

The fourth reconstruction of the concept came through American anthropologist Oscar Lewis's notion of the 'culture of poverty' in the 1960s. Interestingly, it was not the culture of poverty but the British theory of the problem family that was more influential on Sir Keith Joseph as Secretary of State for Social Services, when he referred to the 'cycle of deprivation' in a 1972 speech. But it was again the United States that was the driving force behind the concept of the underclass in the 1980s which at the same time became attractive to observers of economic change and social polarisation in Britain. Finally, from the mid-1990s, the theory of social exclusion, embodied in the Blair government's Social Exclusion Unit, attempted to shed its links with these earlier labels, but nevertheless continued to have echoes with the underclass discourse. Continuities were also evident in specific aspects of policy on child health, most obviously in the way that the phrase 'cycle of deprivation' continued to be used in relation to child poverty and the Sure Start initiative for under-fives. And, as already noted, the continuities are very much alive in current Government initiatives aimed at troubled families.

Who has been doing the defining?

While it is comparatively easy to trace the process of definition and re-definition, through some fascinating historical detective work, it is more difficult to account for the longevity of underclass concepts as a recurring phenomenon. It is important to distinguish underclass theories from related, but more general, ideas about the deserving and undeserving poor, about unemployment and public attitudes towards 'scroungers', and about behaviour more generally. As John Macnicol of the LSE has previously argued, underclass concepts have a number of different strands:

  1. The way that they have been used to signify and denote the alleged behavioural inadequacies of the poor, whether an inability to form attachments to other individuals and agencies, a failure to plan for the future, or a tendency to engage in crime and other forms of antisocial behaviour.
  2. The use of the phrase to denote the ways in which wider structural processes, whether technological and economic change, unemployment, racial and social segregation in cities, or the move to a post-industrial economy, have contributed to a situation in which groups with poor access to education and skills risk being left behind.
  3. The recurring belief in inter-generational continuities, whether of cultural aspirations and habits, or in terms of poverty and teenage pregnancy.
  4. The belief that the underclass exists separately from the working class, as a subset or what has been called the lower class.
  5. The combination of rhetorical symbolism and empirical complexity, where the term 'underclass' has served as a powerful metaphor for social change on the one hand, but where its supporters have also searched - without much success - for empirical proof of its existence.

Historical investigation illustrates marked differences in these concepts and in the individuals or organisations that have used them. In many cases, individual theorists and observers have had a prominent role, such as Charles Booth in the 1880s, Oscar Lewis in the 1960s, and Sir Keith Joseph in the 1970s, though underclass concepts have been more central to the thinking of some than to others. At other times, voluntary organisations and professional groups have been more prominent, such as the Eugenics Society in the 1930s, the Women's Group on Public Welfare in the 1940s, and the Family Service Units of the 1950s. For these individuals and organisations, underclass stereotypes have served both to scapegoat families and to legitimise the functions of professionals. It is only more recently that central government has taken a more active role in sponsoring research. Interestingly the studies sponsored by the Department of Health and Social Security- Social Science Research Council Working Party on Transmitted Deprivation (1974-82) in fact found little evidence to support Keith Joseph's 'cycle of deprivation' hypothesis. The involvement of the Social Exclusion Unit and the Treasury in more recent debates suggests a move away from individuals and voluntary organisations to more centralised policy research processes, illustrating how arguments about behaviour have become more central to public policy.

The American sociologist Herbert Gans has argued that the 'label formation' process includes a number of interested parties: label makers (both alarmists and counters), label users, legitimators, the labelled themselves, and the romanticisers who revive old labels. What support does historical research offer for the framework that Gans has proposed? It certainly is the case, as has been suggested in the American context, that terms follow a trajectory of emergence, popularity, the acquiring of a pejorative character, and then a falling out of favour. The role of the 'alarmists' does appear critical, since discussions of this type are invariably provoked by alarm about the characteristics of the relevant groups and their members. Similarly the role of the legitimators is also important, although academics, professionals, journalists, and politicians can all serve this function. The role of the popular media shows some important continuities, wider changes in its technology and scale notwithstanding. In the 1880s, contemporary periodicals and newspapers were crucial to the propagation of the concept of the social residuum. A century later, in the 1980s, magazines and newspapers were crucial to the rise of interest in the underclass, through articles by the journalist Ken Auletta in the New Yorker, and the Sunday Times sponsorship of the visits by American political scientist, Charles Murray, to Britain. In between, the role of the media was much more muted and the influence of the ideas themselves was limited to a professional rather than a popular arena. The role of the 'romanticisers' seems less evident - once concepts have dropped out of favour and popular usage it is difficult for them to make a comeback.

Has the same idea been invented and re-invented?

Perhaps the most important aspect of this story is the question of whether there is sufficient similarity or linearity between these related concepts to support the argument that there has been a successive reinvention of the underclass concept over the past 132 years. Clearly the economic, political, and social landscape within which these ideas have evolved has changed dramatically. The way that the concept has been defined in different periods has said as much about broader trends as about the underclass itself, in terms of the economy and labour market, the role of women and the emphasis placed on the family, migration and urbanisation, and ideas about behaviour and agency. At various times joblessness, household squalor, mental health (especially mental deficiency or learning disability), long-term poverty, illegitimacy, and crime have been drawn into underclass definitions. Ideas of class formation and biological determinism have played into this, as well as eugenics and a vague and indefinable fear of the other. The social problem group of the 1930s represented a medicalisation of the concept of the residuum, while it was the idea of transmission, more usually associated with infectious disease, which was central to the cycle of deprivation research of the 1970s.

Nevertheless, there are also striking continuities between these ideas, in terms of the alleged physical and mental characteristics of the poor, the stress placed on inter-generational continuities, and the focus on behavioural inadequacies. Arguably most marked has been the emphasis on the cost of these individuals or groups to the taxpayer and the state, through their dependency on charity and social welfare, and disproportionate use of institutions such as prisons, hospitals, asylums, and schools. There has been a clear desire to quantify the size of the 'problem'. In his book In Darkest England and the Way Out (1890), William Booth argued that the 'submerged tenth' comprised around three million people, or one-tenth of the 31 million people in Great Britain at that time. Since then, the underclass has usually been perceived as the 'bottom' ten per cent. The Coalition Government's emphasis on 120,000 troubled families is thus only the most recent example of this phenomenon.

How and why do these concepts emerge?

There are chronological gaps in the history, periods when no underclass concept was available or taken up by social investigators. At these times, a more structural interpretation of poverty and unemployment seemed to be dominant. These include the period from the outbreak of the First World War to the late 1920s, the period from 1937-43, and from 1972 to the early 1980s, when social scientists resisted the imposition of the cycle of deprivation hypothesis. It is striking that these periods are remarkably brief. Above all, the argument that the post-war period was dominated by an emphasis on structural factors, and by economic determinism on the part of social researchers, or by 'knightly' behaviour by professionals in the public sector, seems difficult to sustain given the focus on families deemed to be a problem. Rather it appears that at most times in the period since 1880, there has been a variant of the underclass theory available to researchers, though the scale and influence of such ideas has varied greatly. This, in turn, problematises attempts to identify shifts (as has been suggested of Seebohm Rowntree's 1901 survey of York) between behavioural and structural interpretations.

If we focus on those periods when ideas appear to be in transition, contradictions emerge. Gareth Stedman Jones has proposed, for example, that the idea of the social residuum evaporated during the First World War, when the advent of full employment suggested that those previously deemed unemployable had never really existed. But conversely, it was during the Second World War that the notion of the problem family emerged to replace that of the social problem group. We can date the timing of this with some precision, to the Our Towns report, published by the Women's Group on Public Welfare in March 1943, which argued for an expansion of health and welfare services while at the same time drawing attention to so-called 'problem families'. Similarly the problem family undermines the argument that underclass stereotypes are most likely to emerge in periods of economic dislocation, when attention tends to become focused on the behavioural inadequacies of a 'reserve army of labour'. In fact, the problem family notion, although never central to discussions of social policy, coexisted in 1950s Britain with full employment, economic optimism, and a strong belief in the value of the nuclear family.

The history of the concept of the underclass offers an interesting perspective on debates about policy transfer between the US and the UK. The current literature on policy transfer stresses the need to consider not just what is transferred, but the motivations of those involved. The history of the underclass concept shows the complexities inherent in policy transfer. It would seem at first glance that Britain has been more influenced by the US than vice versa. In the early 1900s, eugenists in Britain were well-aware of the American studies of the Juke and Kallikak families, held to represent inherited criminality and mental deficiency respectively. The 1980s underclass debate is perhaps the best example, with the early debates occurring in the US, and with one mechanism being quite clear - an invitation that the Sunday Times Magazine extended to Charles Murray to visit the UK in 1989. But in other cases, there has been considerable resistance to American ideas. In the late 1960s, for example, British social researchers were resistant to the ideas of Oscar Lewis and it is clear that Sir Keith Joseph's cycle of deprivation owed much more to his earlier interest in the idea of the problem family than to the culture of poverty notion. The 1990s emphasis on social exclusion originally owed more to ideas from across the Channel than from across the Atlantic. It would seem, therefore, that while there are similarities in timing between the invention of underclass concepts in the UK and the US, the form that they took was often very different, reflecting the different histories, ethnic mix, and political cultures of the two countries. Most evident was the much stronger connection with race that was forged with the underclass concept in the US.

What impact have these concepts had on policy-making?

Although it is notoriously difficult to assess, it is also worth asking what practical impact these ideas had on government policymaking. The different concepts were undoubtedly of considerable interest to social commentators, but did they actually influence real policymaking on the ground? The early concepts seem to have had little direct influence. Neither the theory of the social residuum, nor the idea of the unemployable, nor the notion of the social problem group, appears to have influenced policy directly. Some of the ideas were relatively short-lived, and in any case attempts at legislation, whether to segregate mental defectives or to introduce voluntary sterilisation, were unsuccessful. Much later, the cycle of deprivation theory was viewed with hostility by social science researchers, and there was a marked disjunction between the ideas as expressed by Sir Keith Joseph and what researchers actually found. In the 1980s it seems that the underclass concept again had little direct influence on policy - in Britain, for instance, the failure to find empirical support for the existence of an underclass weakened its claims to exert a direct influence on policy.

In other cases there has been a clearer link between specific theories and particular policy initiatives on unemployment and family welfare. In the case of the unemployable, there was a broad link with policies on the administration of unemployment relief in the interwar period, which, it has been argued, were dominated by the search for the scrounger. Moreover very similar debates were evident some fifty years later, in the 1980s, when the renewal of debates about the workshy had a powerful influence on the 1989 Social Security Act. The idea of the problem family was central to the identity of the Family Service Units, and there was a direct link too with local authorities, first through local Health Departments and then in the 1960s through Children's Departments. It has been argued too, that the culture of poverty theory did influence the American 'War on Poverty', and British equivalent projects were apparent in the Educational Priority Areas of the late 1960s and Community Development Projects of the early 1970s. Furthermore, while the underclass notion was arguably less influential, the Charles Murray analysis, on the allegedly detrimental effects of welfare benefits on behaviour, was of considerable interest to policymakers. Finally, the concept of social exclusion had an important influence on Labour policy between 1997 and 2010, and the Sure Start initiative set out to break an alleged cycle of deprivation. Thus as social scientists have become more concerned to unravel the relative influences of agency and structure in the causation of poverty and deprivation, governments have shown increasing interest in ways of influencing behaviour.


It is apparent that the concept of the underclass has been periodically invented and re-invented in the UK and the US over the past 132 years. The persistence of the underclass and related ideas suggests that this process of word substitution is likely to survive and perhaps flourish in the future. There are several reasons for the apparent resilience of the concept. One is the unresolved issue of the relative importance of behavioural and structural factors in the causation of poverty and deprivation. This in part gives the concept much of its ambiguity and flexibility. Another is the relatively late adoption (at least in Britain) of such potentially important data sources as longitudinal and panel data on poverty dynamics and income mobility. The continued likely pace of technological change, globalisation, and economic uncertainty together are likely to continue periodically - or perhaps continually - to raise the spectre, both real and imagined, of groups perceived as 'left behind' or 'cut off' from the mainstream working class. Finally there is the value of the concept as a convenient symbol and metaphor for fears and anxieties whose empirical reality remains unproven. While there has already been some repackaging of terms such as 'troubled families', we can only guess what form these labels will take in the future. Overall, while the Government talks about 'history repeating itself' in families and between generations, the history that has really been repeated here has been that of a flawed discourse.

Summarising the lessons for policy-makers is not so easy. Whereas historians and social scientists approach these issues as being socially constructed, politicians and civil servants feel they are at the sharp end, tasked with framing policies for what they believe are real challenges. Nevertheless history offers some important pointers. First, that it is crucial to examine how such families are defined, what criteria are used to identify them, and whether these are valid and reliable. Second, that behavioural and structural explanations and solutions have too often been juxtaposed as alternatives, and might much more effectively be seen as complementary; targeted policy interventions like the FIPs and FNP (which can only ever serve a small minority of families) must be accompanied by efforts to tackle wider, systemic causes of poverty and unemployment, affecting for instance the working poor as well as the workless. Third, that the Government should focus its energies more on implementing and evaluating specific policies and rather less on counting how many such families allegedly exist, and how much they cost the state, which are little more than rhetorical devices of dubious policy value; as we have seen, this has been a recurring feature of this debate. Fourth, that, partly because of this misplaced energy, we still know surprisingly little about the families, and why they behave as they do. Rather, how these families have been defined over time, and the different attempts made to tackle them, tell us more about the people defining the 'problem', and less about the families themselves. And fifth, that while it is often argued that such problems are intergenerational, in a notional 'cycle of deprivation', these issues are extremely complex, there are also many discontinuities, and the mechanisms for such transmission continue to elude researchers. Overall, while there are very real problems in Britain today, to do with poverty, unemployment, drugs, alcohol, widening social inequality and yes, the behaviour of some people, the Government should focus less on 'problem families', and more on families with problems.

This paper is based on a presentation at a History & Policy seminar for the Department for Education on 24 November 2011, funded by the Arts and Humanities Research Council. Click here for more details.

Further Reading

Deacon, A., '"Levelling the Playing Field, Activating the Players": New Labour and the "Cycle of Disadvantage"', Policy and Politics, 31, 2 (2003), 123-37

Gans, H., The War Against the Poor: The Underclass and Antipoverty Policy (New York, 1995)

Levitas, R., There May be 'Trouble' Ahead: What We Know About Those 120,000 'Troubled' Families, Poverty and Social Exclusion in the UK, Policy Series, Working Paper No. 3 (2012)

Macnicol, J., 'In Pursuit of the Underclass', Journal of Social Policy, 16, 3 (1987), 293-318

Welshman, J., Underclass: A History of the Excluded, 1880-2000 (London, 2006)

Welshman, J., 'From the Cycle of Deprivation to Social Exclusion: Five Continuities', Political Quarterly, 77, 4 (2006), 475-84

Welshman, J., From Transmitted Deprivation to Social Exclusion: Policy, Poverty, and Parenting (Bristol, 2007, paperback edition 2012)

Related Policy Papers


Papers By Author

Papers by Theme

Digital Download

Download and read with you anywhere!


Sign up to receive announcements on events, the latest research and more!

To complete the subscription process, please click the link in the email we just sent you.

We will never send spam and you can unsubscribe any time.

About Us

H&P is based at the Institute of Historical Research, Senate House, University of London.

We are the only project in the UK providing access to an international network of more than 500 historians with a broad range of expertise. H&P offers a range of resources for historians, policy makers and journalists.

Read More