Speaking after a cabinet meeting, he told a news conference that the night-time curfew would now end from June 20, and that face masks would soon no longer be required outdoors.
After restaurants, bars and cafes were allowed last week to reopen indoors for the first time in seven months, Castex said life in France was at last starting to return to normal.
"We're on the right track - let's keep up our efforts," Castex said. "The health situation is improving faster than we had anticipated, everywhere in mainland France."
Across the Channel in Britain, whose vaccination programme is far ahead of France's, a plan to lift most remaining restrictions has been put back a month to July 19 because of the rapid spread of the Delta variant of the virus, which is more infectious, can render vaccines less effective against mild disease, and may put more unvaccinated people in hospital.
The variant accounts for the bulk of new infections in Britain, but French Health Minister Olivier Veran said on Tuesday it only represented 2-4 percent of confirmed cases in France.
He warned that if the Indian variant became dominant in France, it could lead to another wave of infections.
Castex said pressure on French hospitals had decreased significantly. The number of people in intensive care units was down by 116 to 1,952 on Tuesday, according to health ministry data.
France's seven-day moving average of new cases, which was above 40,000 two months ago, stood at 3,500 on Tuesday. French health experts say the pandemic can be considered under control if the rate is below 5,000.