The rise of Silicon Valley's techno-religion

Straits Times

13 hours ago

Sign up now: Get ST's newsletters delivered to your inbox
The former Rose Garden Inn, now a part of the Lighthaven complex, in Berkeley, California, on May 9, 2025. The Rationalists, a community focused on the risks of artificial intelligence, regularly gather with tech figures and other like-minded people in a complex that covers much of a city block.
BERKELEY, CALIFORNIA - In downtown Berkeley, an old hotel has become a temple to the pursuit of artificial intelligence and the future of humanity. Its name is Lighthaven.
Covering much of a city block, this gated complex includes five buildings and a small park dotted with rose bushes, stone fountains and neoclassical statues.
Stained-glass windows glisten on the top floor of the tallest building, called Bayes House after an 18th century mathematician and philosopher.
Lighthaven is the de facto headquarters of a group who call themselves the Rationalists.
This group has many interests involving mathematics, genetics and philosophy.
One of their overriding beliefs is that artificial intelligence (AI) can deliver a better life if it does not destroy humanity first.
The Rationalists believe it is up to the people building AI to ensure that it is a force for the greater good.
Top stories
Swipe. Select. Stay informed.
World Israel to decide next steps in Gaza after ceasefire talks collapse
Singapore 'I wish I can hear her sing again,' says boyfriend of Yishun fatal crash victim
Singapore Singapore-made bot amble matchmakes strangers virtually - without profile photos
Asia What's it like to deal with brutal US tariffs? Ask Malaysia
Singapore Singapore launches review of economic strategy to stay ahead of global shifts
Singapore A look at the five committees reviewing Singapore's economic strategy
Singapore Conditional warning for ex-manager at Mendaki accused of trying to obtain laptop as bribe
They were talking about AI risks years before OpenAI created ChatGPT, which brought AI into the mainstream and turned Silicon Valley on its head.
Their influence has quietly spread through many tech companies, from industry giants like Google to AI pioneers like OpenAI and Anthropic.
Many of the AI world's biggest names – including Dr Shane Legg, a co-founder of Google's DeepMind; Anthropic's chief executive, Dr Dario Amodei; and Dr Paul Christiano, a former OpenAI researcher who now leads safety work at the US Centre for AI Standards and Innovation – have been influenced by Rationalist philosophy. Mr Elon Musk, who runs his own AI company, said that many of the community's ideas align with his own.
Mr Musk met his former partner, pop star Grimes, after they made the same cheeky reference to a Rationalist belief called Roko's Basilisk.
This elaborate thought experiment argues that when an all-powerful AI arrives, it will punish everyone who has not done everything they can to bring it into existence.
But these tech industry leaders stop short of calling themselves Rationalists, often because that label has over the years invited ridicule.
The Rationalist community is tightly entwined with the Effective Altruism movement, which aims to remake philanthropy by calculating how many people would benefit from each donation. This form of utilitarianism aims to benefit not just people who are alive today, but all the people who will ever live.
Many Effective Altruists (EA) , have decided that the best way to benefit humanity is to protect it from destruction by AI.
Rationalists often identify as EAs. And EAs often adopt Rationalist philosophies. Together, these two movements have pumped hundreds of millions of dollars into companies, research labs and think - tanks that aim to build AI and ensure its safety.
The biggest funders include wealthy tech moguls like Mr Jaan Tallinn, a creator of internet calling service Skype, and M r Dustin Moskovitz, a Facebook co-founder.
'They built a vast, well-funded ecosystem to spread, amplify and validate their ideology,' said Ms Mollie Gleiberman, an anthropologist who has studied the rise of the Rationalists and Effective Altruism.
Whether they are right or wrong in their near-religious concerns about AI, the tech industry is reckoning with their beliefs.
In late 2023, OpenAI's chief executive Sam Altman, was briefly removed from his job because board members with ties to the Rationalist and EA movements said they could not trust him to build AI for the benefit of humanity.
Lighthaven is a physical manifestation of just how much these ideas have suffused Silicon Valley and the San Francisco Bay Area – a modern day temple.
The main building, called Aumann Hall, after Israeli game theorist Robert Aumann, offers seven bedrooms and multiple common areas for parties and weekend conferences.
Eigenspace, named for an esoteric mathematical concept, includes a gym and another communal area large enough for 40 people. Two hundred people can fan out across the synthetic grass-covered park, which also has chairs and electric fire pits.
'It's a place where serendipity can happen. Some people liken it to a college campus or the MIT Media Lab,' said Mr Alex K. Chen, a long-time member of the community, referring to the design lab at Massachusetts Institute of Technology.
Each spring, Lighthaven hosts LessOnline, a conference where bloggers and commenters from Rationalist websites meet in person.
Every Tuesday at 6.30pm, almost like a Bible study, people gather to read and discuss The Sequences, the urtext that gave rise to the movement.
'Religion is text and story and ritual,' said Dr Ilia Delio, a Franciscan sister and professor of theology at Villanova University. 'All of that applies here.'
The Rationalist movement is a lifestyle as much as a set of ideas. The adherents have mixed their focus on AI with advice for how to live your life and manage your career.
The community embraces unconventional ideas, including polyamory and the genetics of intelligence as well as Effective Altruism, which is also a lifestyle. And for aspirational AI developers, Rationalist events have become essential networking opportunities.
Gatherings like the Machine Learning Alignment and Theory Scholars (Mats) program me , hosted at Lighthaven each summer, are a more important way of getting into the field of AI safety than academia, said Ms Sonia Joseph, an AI researcher at McGill University in Montreal and tech giant Meta.
The Rationalists emerged in the late 2000s when an online philosopher named Eliezer Yudkowsky wrote The Sequences, a collection of essays that taught people to re-examine the world through cold and careful thought.
This often involved using statistics and probability to inform their decisions. Part tutorial, part entertainment, part mystic journey, Mr Yudkowsky's essays became a manual for the Rationalist community.
In 2010, Mr Yudkowsky introduced the founders of a British AI company called DeepMind to venture capitalist Peter Thiel, helping to get their company off the ground. Less than four years later, DeepMind was acquired by Google for $650 million (S$837.3 million) . Now its technology and executives are leading the tech giant's AI efforts.
Mr Yudkowsky also ran a nonprofit dedicated to AI safety called the Machine Intelligence Research Institute in Berkeley. Slowly, the movement went global. Rationalist group houses appeared in cities like New York and Boston. Meetings were held in Britain, the Netherlands and Australia.
The first international Effective Altruism summit was held in 2013 at a group house in Oakland, California, that served as the live-in headquarters for Leverage Research, a startup with deep ties to the Rationalist community.
Leading figures in the Rationalist community like Mr Yudkowsky and Mr Tallinn helped guide the EA movement toward their shared concerns about artificial intelligence.
Criticism of the Rationalist and EA movements has been frequent, including claims of sexual harassment in group houses and complaints about the community's interest in eugenics and race science.
The community's reputation was damaged in 2023 after Sam Bankman-Fried, founder of cryptocurrency exchange FTX, who was one of the primary financial backers of the two movements, was convicted of fraud. But the movement continues to prosper.
Bankman-Fried had become a financial trader to benefit the most people through EA causes, including the fight to keep AI safe. In the end, he was found guilty of stealing $8 billion from his customers.
'When you think about the billions at stake and the radical transformation of lives across the world because of the eccentric vision of this group, how much more cult-y does it have to be for this to be a cult? Not much,' said Mr Greg M. Epstein, a Harvard chaplain who saw the rise of the Rationalist and EA communities at the university over the past decade and the author of 'Tech Agnostic', a book that discusses technology as a new religion.
'What do cultish and fundamentalist religions often do?' Mr Epstein added. 'They get people to ignore their common sense about problems in the here and now in order to focus their attention on some fantastical future.'
Each December, hundreds from the community gather in places like the Chabot planetarium in the Oakland Hills and the Freight and Salvage music hall in downtown Berkeley for an annual holiday tradition.
They celebrate the Winter Solstice, the longest night of the year, with songs, stories, humour and questions about the fate of the world.
The most recent celebration opened with a song called Uplift, which praised the power of technology throughout human history. Backed by guitar, violin and keyboards, two singers began in the Stone Age and finished in the future.
'Light to push the sails, read the data, cities glow! Hands type the keys, click the mouse, out we go!' they sang. 'Our voices carry 'round the world and into space! Send us out to colonise another place!'
But Mr Ozy Brennan, a long-time Bay Area Rationalist who served as master of ceremonies that night, warned of clouds ahead.
'We face a number of threats our ancestors couldn't have imagined: nuclear war, bioengineered pandemics, artificial intelligence,' he said.
'If we fail – and there is every chance we might – 100 per cent of the children will die, and so will everyone else.'
Lighthaven's main building, a Tudor-style home with a pink-and-white facade, was built in 1905. In the 1970s, it became a bed-and-breakfast called the Rose Garden Inn and soon joined Berkeley's list of historic landmarks.
About three years ago, the property was purchased for US $16.5 million by a company called Lightcone Rose Garden, according to property records. The company was owned by Lightcone Infrastructure, which runs LessWrong, the primary online home of the Rationalists.
'Light cone' is a physics term the Rationalists and the EAs often used to describe the volume of future events they can influence from the current point in time.
Now, Lightcone runs Lighthaven, too. The staff that oversees the property includes Mr Ray Arnold, who organised the first Secular Solstice.
It was purchased with money from two of the community's biggest funders: Mr Tallinn and Bankman-Fried, according to a legal complaint. The funds from Bankman-Fried, which were used as a deposit, were later returned as part of a court settlement.
Outsiders are not always allowed into Lighthaven. Mr Oliver Habryka, the head of Lightcone, declined a request from The New York Times to tour the facility.
Last year, Ms Joseph, the McGill and Meta researcher, spent the summer at Lighthaven after being accepted into the Mats programme .
Now almost 30, Ms Joseph discovered the Rationalist community as a 14-year-old, when she started reading Harry Potter And The Methods Of Rationality .
In the 660,000-word serialised novel, also written by Yudkowsky, Harry Potter refuses to accept the world of wizardry on blind faith, leaning instead on the laws of philosophy, science and rational thinking. It, too, attracted hundreds of people to the community.
'It attracts outsiders,' Ms Joseph said.
'If you are a gay kid in Kansas who is getting no support from religion, you might discover Harry Potter And The Methods Of Rationality and find a community that is really accepting.'
Like LessWrong, Mats can be an on-ramp to jobs at AI companies. Applicants to the programme are often chosen by top AI researchers at companies like Anthropic. But for Ms Joseph and others, the programme is more than just a career move.
Looking back on her summer at Lighthaven, she remembered the roses and stone cherubs that lined the path to a three- storey building with stained-glass windows.
She remembered the glistening mirror at the bottom of the stairs that evoked Harry Potter And The Methods Of Rationality . She remembered the sign on a locked door that read: 'Eliezer Yudkowsky's Office'.
'All of this feels mythic,' she said. 'Even the non-Rationalist scientists find this compelling – the same way the Manhattan Project was compelling. We want to work on something mythic.' NYTIMES