Issue include / exclude filter menus on shop vs category page

Hi

I am trying to use custom filters for certain categories. As a basic example for now I am trying to how only price, category, stores and brands filter on shop page. For that I have created a new filter menu and included it on the shop page. Also I checked to exclude the main filter from the shop page.

So if you navigate from any category page you will get the main filter and if you go to the shop page you have the shop page filter. The thing is that if you go directly to any category you do get the main filter but if you go to the shop page and select a category the filter will get stuck on the shop page filter.

I will share some screenshots to explain.

Best regards,
Horatiu