Pro-rate your fee payments; you undoubtedly pay for pool maintenance, but can only use it X% of the time. Adjust payment accordingly.
In subreddit r/fuckHOA
In a post titled:
HOA conflict of interest? President & Treasurer (married) & run a "Non-Profit" Swim Team sponsored by HOA