The Australian National Audit Office's latest audit into records management in the federal government contained some fairly shocking statistics, highlighting major black holes in the way records are managed.
Subscribe now for unlimited access.
or signup to continue reading
The document - which focused on the implementation of the National Archives' "building trust in the public record" policy and considered two case studies in its effectiveness (Prime Minister and Cabinet, and the Australian National Maritime Museum) - found compliance to be partly effective in the case of PM&C, with its effectiveness overall lacking at the maritime museum.
In fact, more than half of agencies have never or rarely transferred "retain as national archives" records to the National Archives.
Meanwhile, 93 per cent of records in agencies are now unsentenced - not assigned a retention period or preservation rule - up from 63 per cent in 2019, meaning they will never be identified for transfer to Archives.
Worryingly, the Archives itself does not have any performance measures in place to ensure records are transferred to it (or for other mandatory measures of the policy) and does not include a risk for agencies failing to sentence records in its risk register.
It is a lethal mix compounded by lacklustre record keeping at agencies, inconsistencies and errors in reports to the Archives, and a lack of accountability for information management.
This means that Australian society is at risk of losing a generation's worth of archives.
The impact of this on culture and society will be profound.
Exacerbating this, the audit notes "the lowest maturity scores were amongst agencies with cultural or heritage ... functions" - these manage a high proportion of archival data.
As well as the risk to heritage, failing to sentence records means they may be under-retained (and destroyed while there is still a need for them) or over-retained.
Having less than 10 per cent of records in a department sentenced is an incredible risk to privacy, cyber security, reputation and trust, effectiveness, regulatory compliance, and efficiency.
Under-retained records have affected reparations for Indigenous people and victims of institutional abuse.
Across the globe, they have led to the unlawful deportation of dozens of members of the Windrush generation in the UK, and affected tens of thousands of active cases at the Dallas police.
On the other hand, over-retained records have clogged the systems of policing and immigration agencies, which has led to events such as the Manchester Bombing and the unlawful deportation and detention of Vivian Solon and Cornelia Rau, for example.
They have also magnified major data spills like those faced by the Australian National University, Optus, Medibank - records have been breached in those attacks that should simply not have been in the network anymore.
What does this mean for the rest of government?
Government has never had effective management of its information. It has been a broken promise since the beginning of the digital age. There is just too much volume, variety, and velocity of information.
Traditional models have been proven to be ineffective time and time again.
The ANAO may even have contributed to this by continuing to recommend implementation of traditional models whenever it has encountered information governance failings in its audits.
Agencies fail to sentence records because there is too much impact on users, systems, data and governance.
The process requires staff to know and understand thousands of retention rules and manually apply them, customise productivity systems, and over-handle records time and time again.
These processes became outdated five years ago, and are increasingly laborious, impractical, unsustainable, and untenable considering the volume of digital records held by entities has grown on average by 328 per cent a year, from 51 terabytes in 2013, to over 314,000 terabytes in 2022.
Despite these black holes, some federal government portfolios are bucking the trend through artificial intelligence (AI).
Agencies including Defence, Agriculture, Education, Employment and Workplace Relations, and Treasury are tackling the mass information conundrum by leaning on AI to sentence 100 per cent of classifiable records, with accuracy up to 98 per cent.
And they don't have any of the user, system, data, or governance team impacts preventing organisations like the Maritime Museum from succeeding, who noted in their response to the ANAO audit that funding and limited resources had hampered them from ever implementing effective recordkeeping systems.
MORE OPINION:
Crucially, this is increasingly happening with ethical considerations.
It's clear that decisions about records can impact citizens - like Vivian and Cornelia - so any automated management of them is potentially high-risk.
Fortunately, laws are now coming into force that don't allow use of black-box AI, such as Machine Learning or large language models, for this purpose. Locally, the government is seeking input on responsible AI, and stringent safeguards against inevitable security risks are paramount.
The concept of having full control of records is far less daunting than it appears. With data volumes ever-growing, shifting the burden of its management from manual processes to ethical and responsible automations is poised to untangle the bleak findings of the ANAO's audit.
Yesteryear's manual methods don't cut it in the digital world.
- Rachael Greaves is an analyst, security practitioner, and information management specialist, and currently CEO and co-founder of Australian company Castlepoint.
