Changing faction from Aldor to Scyers

ok.. so i screwed up. I accidentally returned the fel arnament.

I now I'm unfriendly with the Scryers and neutral with Aldor.

I can't pick up the quest from "Adyras". Can you not be neutral to both factions?

is that the problem? It'd be great if some can tell me how I can side with the Scryers