This is because apostrophes that show possession never appear inside the word. That rules out "One of my cat's" because you would never say "One of my cat".
The quirk here is I suspect it's also "Two of my cats' nails...". That means two cats. If you wrote "Two of my cat's nails...", that's two nails and one cat.
Great question. The answer is "One of my cats' nails..."