Programming is foremost a structuring problem so you can manage the complexity of it. The problem you thus always solve first is to be as clear as possible about capabilities, responsibilities, and working of the code. Your two solutions are not the same in that respect.
If you make a ManagerForDerivedClass class, then the manager knows about all the guts inside DerivedClass. If you make a Manager<SuperClass> class (or ManagerForSuperClass if there is only one relevant SuperClass), the manager doesn't know about all those details, it deals with the objects at a higher level of abstraction. The latter solution clearly signals it is only responsible for stuff at the higher abstraction level, which reduces the complexity of that code, which is good as you want that as low as possible.
There is a second reason why giving access to the superclass only is a good idea for code that doesn't need the details of the derived class. If the manager class doesn't have access to the details of the objects, he cannot use them by accident. The compiler helps you here. When you try to access stuff that only exists in DerivedClass from the Manager<SuperClass> or ManagerForSuperClass, the compiler barks and refuses to compile it. In the ManagerForDerivedClass you will never ever find such cases.
Having the compiler bark at illegal access attempts enforces that the SuperClass definition is actually correct with respect to how you think about the capabilities and responsibilities of SuperClass. You may now think that SuperClass is enough for the manager, but in my experience it is often the case, that you missed one or two edge cases where you do need some information that is not available in SuperClass. A barking compiler tells you about those cases, and gives you the opportunity to find and fix these bugs in the code structure now, rather than after a few months when you stacked heaps of other code around it.
In my view, clarity of code always wins, period. If you care more for performance than me, clarity of code always wins unless there is hard proof it does not. “hard proof” here means you have profile output that shows that Manager<SuperClass> is a lot slower than ManagerForDerivedClass with an analysis that it is not caused by something else (the point where you create a performance issue is usually different from where it pops up). For upcasting, I don't quite buy you're going to have a sufficient big performance problem. Upcasting is one of the corner stones of OOP, so compilers pulled all tricks to make that fast. If you have a performance problem there, you more likely have a structuring problem or a granularity problem somewhere else, but that would need a careful analysis of a concrete case.