Tamiflu is the main antiviral being used in the UK. The World Health Organization has said healthy patients who catch swine flu do not need to be treated with Tamiflu.
Antiviral drugs should be used in patients who are severely ill or those in high-risk groups including the under fives and pregnant women, it said.
The cost of the drug and the potential for resistance were taken into account by the expert panel.
But the UK government said it was taking a "safety first" approach by offering antivirals to everyone.
A spokesman for the Department of Health said they would keep the policy under review.
The UK has stockpiled enough antivirals - the mainstay of treatment until a vaccine becomes available - to cover half the population.
They are being offered in England to anyone who has the symptoms of swine flu - in many cases through the national pandemic flu helpline.
Minimising the number who take the drug is a good idea
Dr Chris Smith, Cambridge University
Wales and NI flu deaths
In the last week almost 46,000 courses of antivirals were handed out in England.
It is not the first time prescribing of antiviral drugs has come under scrutiny.
Earlier this month research suggested that the drugs should not be used in children because they rarely prevent complications but yet carry side effects.
Infections in England are continuing to fall with an estimated 11,000 cases last week compared with 100,000 a month ago.
There have now been 61 deaths across the UK - 56 of those in England.
Experts expect to see a big surge in infections in autumn when flu viruses traditionally start to circulate more widely.
In updated guidance, the WHO said most patients infected with the pandemic virus continue to experience typical influenza symptoms and fully recover within a week, even without any medical treatment.
So it said healthy patients with uncomplicated illness need not be treated with antivirals, which include Tamiflu and Relenza.
High-risk groups, who should receive treatment include infants and children under five, the over 65s, nursing home residents, pregnant women, and patients with chronic conditions such as asthma or diabetes and those with reduced immunity such as people with HIV.
And patients who initially present with severe flu or whose condition begins to deteriorate should also get the drugs as quickly as possible.
The report also called for more research into the safety of Tamiflu given its widespread use in the UK, Australia and the USA.
Dr Christ Smith a virologist at Cambridge University said when the virus first emerged little was known about how dangerous it would be and how best to treat it.
"We now know it's no more severe than seasonal flu for which we don't make Tamiflu available for healthy members of the general population.
"No drug is without side-effects, and it is expensive.
"Minimising the number who take the drug is a good idea."
But a Department of Health spokesman said: "We believe a safety-first approach of offering antivirals, when required, to everyone remains a sensible and responsible way forward.
"However, we will keep this policy under review as we learn more about the virus and its effects.
"WHO state that 40% of severe cases worldwide have been in previously healthy children and adults and that serious cases should be treated immediately.
"This emphasises the need not to become complacent about the mildness of the illness and the reasoning behind a precautionary policy."