AI use for job seekers: Defaulting to a 'computer says no' situation
RNZ / Quin Tauetau
Using AI algorithms to to renew expired Jobseeker benefits automatically could result in 'computer says no' discrimination, experts are warning.
Cabinet has agreed to introduce a suite of new obligations and sanctions for job seekers this year, and halving the time people can be on the benefit from a year to six months is among the proposed changes.
That would effectively double the workload for Ministry of Social Development staff processing re-applications and the ministry said it plans to use automatic decision making to roll these over.
MSD said such AI would not be used to decline Jobseeker Support benefits - that decision would be made by a staff member.
Victoria University of Wellington professor and AI expert Dr Simon McCallum said the potential for bias was too great.
"There are certainly areas to be concerned about," he said.
"Once an automatic decision is made the human won't think fresh about the situation, they'll just use whatever the computer says or default to a 'computer said no' situation."
McCallum said human empathy can be inhibited by such a process.
"When you are the human who just gets a group of people who are potentially problematic your mindset shifts into a 'I'm dealing with people who are problematic' and you get that expectation issue."
He said such automated decisions would need human oversight in a way that could make them less efficient.
AI expert Dr Karaitiana Taiuru specialises in the effect of automated decision making on indigenous and vulnerable communities.
He said it all comes down to the data and algorithms used and if that reflects the nuances of such groups of people.
"The data MSD has could very easily reflect bias against Maori, Pasifika and immigrants so in the ideal world companies and organisations would use those same communities to consult on the algorithms they're using, or how the automated system will work."
Taiuri said that should be the first step.
"MSD really needs to consult properly with all communities that will be impacted by this and I'm sure if they do this will be a cost saver but there has to be that human investment in the project first."
He also said a standard developed for MSD's use of such technology needed an overhaul.
"There is an issue here. MSD are saying they've got some safeguards from 2022 automated tools and AI have significantly changed in the last three years."
MSD said it is reviewing the standard in consultation with the privacy commissioner.
MSD is a co-signatory to The Algorithm Charter for Aotearoa New Zealand, which outlines the ways in which government agencies should demonstrate transparency and accountability in the use of data.
Under the charter, agencies agree to making sure their data is fit for purpose by understanding its limitations and identifying and managing bias - to name a few.
But Taiuru said the charter can not be enforced.
"AI in New Zealand is not being regulated so it is up to individual departments on how they handle automated AI systems and if they choose not to follow guidelines they don't have to."
MSD deputy chief executive for organisational assurance and communication Melissa Gill said it was important to make sure the ministry used automatic decision making carefully and responsibly.
"Our automated decision making process for regranting benefits uses a series of simple checks to check whether a client meets the legal requirements to have their benefit regranted."
She said the process did not include demographic information or profiling about people on benefits.
Gill said the automated process would not decline a benefit regrant, and beneficiaries would still be able to re-apply for their benefits in person.
"The redesigned 26-week reapplication and the use of automatic decision making in this way can make it easier for clients to do many of the basic things they need to do, without having to repeat information they have already given us, or repeat activities already completed. This allows us to focus on the interactions with clients that will help get them into work."
MSD said its Automated Decision Making Standard was introduced in 2022 after consultation with the Privacy Commissioner, the Human Rights Commission and the University of Otago Centre for Artificial Intelligence and Public Policy.
"It provides a framework for good practice and introduces specific automatic decision making safeguards, including the requirement that our clients get their correct entitlement and are not discriminated against," Gill said.
"Where appropriate, we are able to consult with external reference groups to help achieve this. MSD has access to a range of reference groups who we can involve when this be required."
Gill said MSD used the Standard to assess the shortened regrant process, and no specific concerns about bias or discrimination were identified.
"As a result, there was no need to consult with external reference groups."
AUT computer science professor and AI expert Dr Edmund Lai said some degree of bias is unavoidable in both human and computer systems.
He said using automatic decision making to rollover benefits at the expiry date should not be a problem.
"If the tool is used in exactly that way I don't see any problems because you're just saving manpower for screening all the routine cases where reapplication is successful and then a human will look at those other ones and rexamine those cases."
He said it would need human safeguards.
Law changes are needed in order for the government to introduce new obligations and sanctions for job seekers.
A new clause in the Social Services Amendment Bill, which has passed its first reading, vastly expands the decisions that can be made by automated systems to include sanctions.
MSD said it did not plan to use generative AI or automated decision making in that way.
Both the Salvation Army and Law Society have called for the clause to be scrapped, in submissions to the select committee considering the bill.
The Salvation Army's submission stated that automatic decision making "cannot account for the complexities we often see in the individuals we support", such as financial hardship, addictions, mental health issues or unstable living conditions.
The Law Society wants the clause allowing the expanded use of AI to be dropped entirely.
"This raises significant concern about how the use of automated systems will apply where the sanctions provisions involve some form of evaluative judgement, for example those relating to money management and community work."
The AI experts RNZ spoke to said while they supported using technology for good ways to automate decisions, using such AI for sanctions would come with risks.
Lai said it would not be an appropriate use of the technology to cut benefits or decide other monetary sanctions.
All agreed such use of the technology would need human oversight and that is not currently regulated.
Sign up for Ngā Pitopito Kōrero
,
a daily newsletter curated by our editors and delivered straight to your inbox every weekday.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Scoop
6 hours ago
- Scoop
Who's Really To Blame For Our Economic Troubles?
New Zealand's economic downturn is proving hard to shake - but economists say when it comes to identifying the cause, it's not as simple as pointing the finger at government spending cutbacks or the Reserve Bank's interest rate hikes. Although conditions are expected to improve as the year progresses, the economy remains sluggish. The primary sector is providing one of the few bright spots. Last weekend, when issuing an annual update, Kiwibank chief economist Jarrod Kerr blamed the Reserve Bank's decision to thrust the economy into recession to tamp down problem inflation. Some readers contacted RNZ saying that was too simplistic and letting the government off the hook for the impact its spending cutbacks and public sector job cuts were having. "My wife's MPI office in Christchurch reduced headcount by about 10 percent," one wrote. "For each public service worker cut, two to three private sector jobs went with it. The cancellation or freezing of so many infrastructure projects saw my industry (engineering) lay off a quarter to a third of the workforce." So who's to blame? "The overstimulus of the economy from both the government and the Reserve Bank in 2020 and 2021 meant the economy was unsustainably overheated," said chief forecaster at Infometrics Gareth Kiernan. "The government and the Reserve Bank got us into the mess we were in, in the first place, and therefore had to get out again." While normally monetary and fiscal policy would work to smooth the economic cycle, that was not the case this time, which made the upswing hotter and the downturn bleaker. "Normally, if the economy is going hot, they tend to pull things back. And if the economy is slowing down and we're heading to recession, you typically get more government spending and lower interest rates and that sort of thing. That's not what we've had over the last five years because they pumped things up too much and had to cool the jets on the other side. "Both the government and Reserve Bank needed to, they had no choice. "Government spending decisions have exacerbated the recession but they had no choice because they also exacerbated the boom before that." Too much spending He said, in December 2022, government consumption spending was 13.5 percent higher than had been forecast before the pandemic hit. "It's still 7.5 percent larger than we were forecasting now… so they've pulled it back in line but it's still larger than it would otherwise have been expected. "Government investments - infrastructure, schools, hospitals, whatever else - by mid-2024, that's 18 percent larger than we were forecasting." Spending was still 13 percent higher, he said. "That puts some context around the fact that yeah, government has been tight but we still haven't undone or unwound all of the massive government-related Covid boom. "That's also reflective of the fact that fiscal numbers over for the next five years are only just getting back into surplus if you're lucky." Kiernan looked at GDP broken down into household spending, government spending, business investment and exports and imports. He compared the current situation and the experience of the past two years to what normal growth might have been if it had kept going from the 2022 base at a 10-year average rate. "Government investment grew through 2023 and early 2024. It's done its fair share of contribution to how the economy has gone in a positive sense. "If I attribute all of the private sector investment and household spending and the slow growth there to interest rates and in Reserve Bank policy… then you put around about two-thirds of the underperformance of the economy over the last two years down to interest rates, 20 to 25 percent on to the government side of things when you take consumption and investment together and the other 10 percent or so is exports underperforming." Senior economist at ANZ Miles Workman said it was a question of how the Reserve Bank responded to the government's fiscal policies. "The faster the government gets the books back into the black, by restraining spending or increasing taxes, the more likely the RBNZ will need to lower the OCR, which would offset lower demand from fiscal policy settings by shoring up demand from households and businesses. "In other words, it's not as simple as saying government policy has caused X percent of the slowdown and the RBNZ has caused Y percent. The RBNZ's policy stance is a function of fiscal policy settings." 'A con job' Economics professor at the University of Auckland Robert MacCulloch said there had been mishandling of the economy by governments on both sides in recent years. "The narrative the parties are using, that the other party is to blame for our problems… I'm past it now, that all the problems we've got are because of [Jacinda] Ardern and [Grant] Robertson and Labour messed it up, then Labour say the same about National. I think it's a con job on the people of the country. The country hasn't been doing great now for quite a long time under both major political parties." He said the wage subsidy had been a significant source of debt, but while it was introduced by the Labour government, National had put pressure on to extend it for bigger firms, which increased the cost. "Luxon's getting angry saying we inherited a shambles. I mean, what's he talking about? His party voted in favour of the all the fiscal and monetary policy. They were wildly enthusiastic about the wage subsidy scheme." He said the cuts the government were enacting were immaterial in the scheme of the fiscal pressures looming from the ageing population. "The cuts in bureaucracy they've done are just a couple of percent of those increased pressures on health and pensions. They're going to have to grapple with how you deal with those huge outflows with the ageing population… the Nats don't have a plan to rein in the cost of those major programmes." Tough job for Reserve Bank The chief economist at the NZ Initiative, Eric Crampton, said the labour market was "certainly in worse shape" than it had been from mid-2021 to mid-2023. "But employment rates during that period were the result of substantial overheating from debt-fuelled spending and substantial monetary stimulus. They could not be sustained without severe inflationary consequence. It is wrong to blame anyone for labour markets not looking now like they did in 2022." He said monetary policy would always be expected to move last. "That doesn't mean monetary policy is slow. It means that whatever the bank does in monetary policy takes into account what everyone else has done and is expected to do. So if the fiscal authority announces a track - a Budget, Budget policy statement - monetary policy adjusts to take that into account. The Reserve Bank moves last - it makes its move after seeing what everyone else is doing. "Governments can be more and less helpful with all of that. But a bank following an inflation target ought to be taking account of what Parliament is doing. " He said the Reserve Bank had a tough job at present. "While the employment statistics are comparable to where they were around 2016-2017, government estimates still suggest that there is a remaining output gap - that the economy is running a bit below potential. However, there are inflationary pressures both in the non-traded sector and in tradeables. The bank's failure to keep inflation within bounds over the Covid period should have it sensitive to the risks of going above-target again. So I wouldn't want to blame them too much for current unemployment except to the extent that the bank's errors during the Covid period makes things a bit harder now. " He said keeping inflation within its target was the best thing the Reserve Bank could do to ensure long-term stable employment. "Given inflation risks, the bank would likely need to act to offset fiscal stimulus, if Parliament decided to provide further fiscal stimulus - we are still running a substantial deficit as well."
RNZ News
7 hours ago
- RNZ News
Allegations Maori have been removed from Maori roll
Te Pati Maori said they have filed urgent proceedings in the High Court after allegations Maori have been removed or shifted off the Maori roll without their consent. A number of voters have posted online to say their electoral enrolment details have been altered or are not there at all. It comes on the eve of local election enrolment cutoff and ahead of sweeping electoral changes promised by the government. To embed this content on your own webpage, cut and paste the following: See terms of use.
RNZ News
9 hours ago
- RNZ News
Musicians concerned by lack of regulation around fast-evolving AI technology
Jeremy Toy. Photo: RNZ / Evie Richardson New Zealand's musicians are watching on with a mix of horror and wonder as artificial intelligence programmes create increasingly plausible songs, often with just a few clicks of a button. Just weeks ago a band called the 'Velvet Sundown' rose up the Spotify charts before it was revealed all their music had been generated by AI. As the technology continues to evolve at breakneck speed, artists and producers here are concerned about the lack of regulation around the tech. In recent years, a number of easily accessible generative AI music tools have been released, where users can create complete songs by merely inserting a simple prompt. One of these is Suno, a US-based company, which has faced a number of lawsuits from major record labels in the US and Germany over copyright issues. Sophie Burbery, a musician and PHD student studying the topic, said companies like Suno don't disclose what music their AI software is learning from, leaving artists vulnerable. "Suno has admitted that all of its music is trained on anything that it can scrape from the internet under fair use. No decisions have been made yet within those court systems as to whether or not they can actually rely on that as a way of getting away with not paying musicians for their music or licensing it or asking permission because at the moment they've done none of those things. " Experts say the use of AI here in New Zealand is similar to the 'wild west' with no regulation or laws in place. Concern was sparked following the recent release of the government's AI strategy report, which contained no mention of the implications for creative industries. Burbery said if this continues, the consequences could be dire for our music and other creative industries. "It's really up to the government to be doing this work, and to be saying hey look, you want to have your platform up in New Zealand Suno and Udio you need to tell us where and how you've trained your AI, and it has to be labelled as the output and who owns the output of the ai?" Sophie Burbery. Photo: Supplied / Paul Taylor These AI programmes have generated many questions around copyright, an area where New Zealand is unique. Under the current Copyright Act, when a person uses AI to create a piece of art, such as a song, the end product automatically belongs to them, even it's based on a multitude of other people's songs. Clive Elliott KC, a barrister at Shortland Chambers who specialises in Intellectual Property told Checkpoint the current law is not fit for purpose when it comes to protecting creatives. "We can't use old principles that have been around for many years. We've got to say this is a completely different paradigm we face in here and we have to find a way which compensates people who have contributed to the learning process." Elliot said the Copyright Act is simply too out of date to apply to the rapidly evolving technology. "It's theft in a way, but it's theft of a tiny piece of information. And the problem with copyright is you have to show that a substantial part of the work has been copied." "The Copyright Act has been under review for years now. [The government] need to step up and say this is this is urgent." For some in the industry, like producer and artist Jeremy Toy, the risks are worrying. "If it's adopted early on with songwriters that it normalises the process of using AI to create your music. It's completely stripping you of your creative ideas." "I find it offensive as a musician that people think they can train a computer to think independently like a creative." They said there are some things that AI will never be able to replicate. "Connecting on the level that's not verbal, just being in a room with someone and playing music with them, that will never be recreated." Although the buzz around AI has amplified in recent years, some musicians say its been a part of the industry for a while now. Rodi Kirk. Photo: RNZ / Evie Richardson Rodi Kirk, who works in music tech, said AI is commonly used when producing music, particularly in the mixing and mastering stages of production. "One thing that might be surprising is that tools that rely on machine learning are not super new in terms of music production." Kirk remained optimistic about the benefits the technology could bring. "I wouldn't release a song and swap my voice for somebody that was well known, but you might do things that change your voice around for creative purposes. This general suite of tools that will be enabled by AI, I think people will do really creative stuff with them. " With no regulation or protections for artists in sight, Burbery said it is unclear where AI will take the music industry next. "It could offer many great creative possibilities but we don't know what they are because the way it has been developed is so unethical." Sign up for Ngā Pitopito Kōrero , a daily newsletter curated by our editors and delivered straight to your inbox every weekday.
