The order is intended to advance Elon Musk 's work slashing spending with his Department of Government Efficiency.
Musk appeared with Trump in the Oval Office as Trump prepared to sign the order. Musk, the world's richest man, said there are some good people in the federal bureaucracy but they need to be accountable and called it an "unelected" fourth branch.
"The people voted for major government reform and that's what the people are going to get," Musk said. "That's what democracy is all about."
The order said that "agencies will undertake plans for large-scale reductions in force and determine which agency components (or agencies themselves) may be eliminated or combined because their functions aren't required by law."
It also said that agencies should "hire no more than one employee for every four employees that depart from federal service." There are plans for exceptions when it comes to immigration, law enforcement and public safety.
Trump and Musk are pushing federal workers to resign in return for financial incentives, although their plan is currently on hold while a judge reviews its legality. The deferred resignation program, commonly described as a buyout, would allow employees to quit and still get paid until September 30.
Administration officials said more than 65,000 workers have taken the offer.
Hundreds of people gathered for a rally Tuesday across the street from the US Capitol in support of federal workers.
Janet Connelly, a graphic designer with the Department of Energy, said she's fed up with emails from the Office of Personnel Management encouraging people to take the deferred resignation program.
Connelly said she has no plans to take the offer.
She said she thinks of her work as trying to do an important service for the American public.
"It's too easy to vilify us," she said.
Others have said fear and uncertainty have swept through the federal workforce.
"They're worried about their jobs. They're worried about their families. They're also worried about their work and the communities they serve," said Helen Bottcher, a former Environmental Protection Agency employee and current union leader in Seattle.
Bottcher participated in a press conference hosted by Sen. Patty Murray, a Democrat from Washington.
Murray said workers "deserve better than to be threatened, intimidated and pushed out the door by Elon Musk and Donald Trump."
She also said that "we actually need these people to stay in their jobs or things are going to start breaking."