How to remove a field from a PowerPivot chart without removing it from the pivot table ?

I would like to remove a field from my Powerpivot chart, however when I do so it also removes it from the pivot table. How do I removed a field from the chart without removing it from the table? It won't let me drag the field off the chart (like in the old pivot days . . )